Error using Named Entity Recognition Tagging

[redfalcoon]

Hi Ludwig Authors,
I'm trying to use ludwig to create a NER extractor from italian text, but I got always an error message that preclude the model generation. Maybe I'm using in a wrong way the ludwig training so I'm sorry to create trouble.
My csv file contains the following data:

frase,mappa
Lorenzo Liparti e' nato a Roma nel 2000,P P O O O C O D
Giancarlo vive a Milano dal 2011,P O O C O D
Nel 2020 si e' diffuso il COVID,O D O O O O M
Il film Titanic e' una vera noia del 2000,O O F O O O O O D
nel libro il signore degli anelli tolkien ha dato il massimo,O O B B B B P O O O O

and the config.yaml file is:

input_features:
-
name: frase
type: text
level: word
encoder: rnn
cell_type: lstm
reduce_output: null
preprocessing:
word_tokenizer: space

output_features:
-
name: mappa
type: sequence
decoder: tagger

when I try to execute the command:

ludwig train --dataset testo.csv --config_file config.yaml

and the system show this log:

2021-05-14 15:13:41.778194: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-14 15:13:41.778237: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/usr/lib/python3.6/site-packages/typeguard/init.py:917: UserWarning: no type annotations present -- not typechecking tensorflow_addons.layers.max_unpooling_2d.MaxUnpooling2D.init
warn('no type annotations present -- not typechecking {}'.format(function_name(func)))
███████████████████████
█ █ █ █ ▜█ █ █ █ █ █
█ █ █ █ █ █ █ █ █ █ ███
█ █ █ █ █ █ █ █ █ ▌ █
█ █████ █ █ █ █ █ █ █ █
█ █ ▟█ █ █ █
███████████████████████
ludwig v0.3.3 - Train

2021-05-14 15:13:43.410379: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-14 15:13:43.410616: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-05-14 15:13:43.410637: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-05-14 15:13:43.410666: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (app01.localdomain): /proc/driver/nvidia/version does not exist
Experiment name: experiment
Model name: run
Output directory: results/experiment_run_4

ludwig_version: '0.3.3'
command: '/usr/bin/ludwig train --dataset testo.csv --config_file config.yaml'
random_seed: 42
dataset: 'testo.csv'
data_format: 'csv'
config: { 'combiner': {'type': 'concat'},
'input_features': [ { 'cell_type': 'lstm',
'column': 'frase',
'encoder': 'rnn',
'level': 'word',
'name': 'frase',
'preprocessing': {'word_tokenizer': 'space'},
'proc_column': 'frase_R4V2yq',
'reduce_output': None,
'tied': None,
'type': 'text'}],
'output_features': [ { 'column': 'mappa',
'decoder': 'tagger',
'dependencies': [],
'loss': { 'class_similarities_temperature': 0,
'class_weights': 1,
'confidence_penalty': 0,
'distortion': 1,
'labels_smoothing': 0,
'negative_samples': 0,
'robust_lambda': 0,
'sampler': None,
'type': 'softmax_cross_entropy',
'unique': False,
'weight': 1},
'name': 'mappa',
'proc_column': 'mappa_mZFLky',
'reduce_dependencies': 'sum',
'reduce_input': None,
'type': 'sequence'}],
'preprocessing': { 'audio': { 'audio_feature': {'type': 'raw'},
'audio_file_length_limit_in_s': 7.5,
'in_memory': True,
'missing_value_strategy': 'backfill',
'norm': None,
'padding_value': 0},
'bag': { 'fill_value': '',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000,
'tokenizer': 'space'},
'binary': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'category': { 'fill_value': '',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'date': { 'datetime_format': None,
'fill_value': '',
'missing_value_strategy': 'fill_with_const'},
'force_split': False,
'h3': { 'fill_value': 576495936675512319,
'missing_value_strategy': 'fill_with_const'},
'image': { 'in_memory': True,
'missing_value_strategy': 'backfill',
'num_processes': 1,
'resize_method': 'interpolate',
'scaling': 'pixel_normalization'},
'numerical': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const',
'normalization': None},
'sequence': { 'fill_value': '',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 20000,
'padding': 'right',
'padding_symbol': '',
'sequence_length_limit': 256,
'tokenizer': 'space',
'unknown_symbol': '',
'vocab_file': None},
'set': { 'fill_value': '',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000,
'tokenizer': 'space'},
'split_probabilities': (0.7, 0.1, 0.2),
'stratify': None,
'text': { 'char_most_common': 70,
'char_sequence_length_limit': 1024,
'char_tokenizer': 'characters',
'char_vocab_file': None,
'fill_value': '',
'lowercase': True,
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_symbol': '',
'pretrained_model_name_or_path': None,
'unknown_symbol': '',
'word_most_common': 20000,
'word_sequence_length_limit': 256,
'word_tokenizer': 'space_punct',
'word_vocab_file': None},
'timeseries': { 'fill_value': '',
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_value': 0,
'timeseries_length_limit': 256,
'tokenizer': 'space'},
'vector': { 'fill_value': '',
'missing_value_strategy': 'fill_with_const'}},
'training': { 'batch_size': 128,
'bucketing_field': None,
'decay': False,
'decay_rate': 0.96,
'decay_steps': 10000,
'early_stop': 5,
'epochs': 100,
'eval_batch_size': 0,
'gradient_clipping': None,
'increase_batch_size_on_plateau': 0,
'increase_batch_size_on_plateau_max': 512,
'increase_batch_size_on_plateau_patience': 5,
'increase_batch_size_on_plateau_rate': 2,
'learning_rate': 0.001,
'learning_rate_warmup_epochs': 1,
'optimizer': { 'beta_1': 0.9,
'beta_2': 0.999,
'epsilon': 1e-08,
'type': 'adam'},
'reduce_learning_rate_on_plateau': 0,
'reduce_learning_rate_on_plateau_patience': 5,
'reduce_learning_rate_on_plateau_rate': 0.5,
'regularization_lambda': 0,
'regularizer': 'l2',
'staircase': False,
'validation_field': 'combined',
'validation_metric': 'loss'}}
tf_version: '2.4.1'

Found hdf5 and meta.json with the same filename of the dataset, but checksum don't match, if saving of processed input is not skipped they will be overridden
Using full raw csv, no hdf5 and json file with the same name have been found
Building dataset (it may take a while)
Writing preprocessed dataset cache
Writing train set metadata
Training set: 3
Validation set: 1
Test set: 1
2021-05-14 15:13:43.478838: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-05-14 15:13:43.481326: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
embedding_size (256) is greater than vocab_size (34). Setting embedding size to be equal to vocab_size.

╒══════════╕
│ TRAINING │
╘══════════╛

Epoch 1
Training: 0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/usr/bin/ludwig", line 10, in
sys.exit(main())
File "/usr/lib/python3.6/site-packages/ludwig/cli.py", line 146, in main
CLI()
File "/usr/lib/python3.6/site-packages/ludwig/cli.py", line 72, in init
getattr(self, args.command)()
File "/usr/lib/python3.6/site-packages/ludwig/cli.py", line 77, in train
train.cli(sys.argv[2:])
File "/usr/lib/python3.6/site-packages/ludwig/train.py", line 412, in cli
train_cli(**vars(args))
File "/usr/lib/python3.6/site-packages/ludwig/train.py", line 197, in train_cli
debug=debug,
File "/usr/lib/python3.6/site-packages/ludwig/api.py", line 487, in train
save_path=model_dir,
File "/usr/lib/python3.6/site-packages/ludwig/models/trainer.py", line 550, in train
self.regularization_lambda
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 828, in call
result = self._call(*args, **kwds)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 871, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 726, in _initialize
*args, **kwds))
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 2969, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3361, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3206, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 990, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 634, in wrapped_fn
out = weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3887, in bound_method_wrapper
return wrapped_fn(*args, **kwargs)
File "/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 977, in wrapper
raise e.ag_error_metadata.to_exception(e)
tensorflow.python.autograph.impl.api.StagingError: in user code:

/usr/lib/python3.6/site-packages/ludwig/models/ecd.py:176 train_step  *
    model_outputs = self((inputs, targets), training=True)
/usr/lib/python3.6/site-packages/ludwig/models/ecd.py:118 call  *
    decoder_outputs = decoder(
/usr/lib/python3.6/site-packages/ludwig/features/base_feature.py:271 call  *
    logits = self.logits(logits_input, target=target, training=training)
/usr/lib/python3.6/site-packages/ludwig/features/sequence_feature.py:243 logits  *
    inputs,
/usr/lib/python3.6/site-packages/ludwig/decoders/sequence_decoders.py:785 _logits_training  *
    return self.call(inputs, training=training, mask=mask)
/usr/lib/python3.6/site-packages/ludwig/decoders/sequence_decoders.py:774 call  *
    LENGTHS: inputs[LENGTHS]

KeyError: 'lengths'

Training: 0%|

[jimthompson5802]

@redfalcoon This seems be related to an error that was fixed in PR #1105. This fix is planned to be incorporated into the next release. Until that occurs, I suggest uninstalling Ludwig 0.3.3 and install from Github the main branch, i.e.,
pip install git+https://github.com/ludwig-ai/ludwig.git

Let me know if this addressed the issue.

[redfalcoon]

@jimthompson5802 I've done the steps that you suggest and after install also the petastorm now the training phase goes well without issue.
Now I will investigate further the solution to check the potentialities.
I wish to thank you for your valuable support and all the team for the great solution

[jimthompson5802]

@redfalcoon Thank you for letting me know that the suggestion works. I'm going to close this issue.