A problem with evaluation on CoNLL dataset

[mczuk]

Hi, thanks for the great work and very good documentation of your work.

I came across the following error:

The number of labels: 51362
Traceback (most recent call last):
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/ha1/czuk/nlp/eclipse/workspace_deepner/luke/luke/examples/cli.py", line 135, in <module>
    cli()
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/ha1/czuk/nlp/eclipse/workspace_deepner/luke/luke/examples/utils/trainer.py", line 32, in wrapper
    return func(*args, **kwargs)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/ha1/czuk/nlp/eclipse/workspace_deepner/luke/luke/examples/ner/main.py", line 90, in run
    results.update({f"dev_{k}": v for k, v in evaluate(args, model, "dev", dev_output_file).items()})
  File "/ha1/czuk/nlp/eclipse/workspace_deepner/luke/luke/examples/ner/main.py", line 155, in evaluate
    print(seqeval.metrics.classification_report(final_labels, final_predictions, digits=4))
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/seqeval/metrics/sequence_labeling.py", line 697, in classification_report
    suffix=suffix
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/seqeval/metrics/sequence_labeling.py", line 139, in precision_recall_fscore_support
    extract_tp_actual_correct=extract_tp_actual_correct
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/seqeval/metrics/v1.py", line 122, in _precision_recall_fscore_support
    check_consistent_length(y_true, y_pred)
  File "/home/czuk/anaconda3/envs/luke/lib/python3.6/site-packages/seqeval/metrics/v1.py", line 97, in check_consistent_length
    raise TypeError('Found input variables without list of list.')
TypeError: Found input variables without list of list.

I guess it might be related to the version of seqeval. I am using seqeval==1.2.2. Is it ok?

[ikuyamada]

Hi,
As also mentioned in the issue comment (#34 (comment)), this error is related to the version of seqeval. We plan to fix this issue, but please use seqeval==0.0.12 for now.

[mczuk]

Thank you, and sorry for the duplicated issue. I missed that ;-)