During finetuning, classes_to_id is not correct

[Ahmedn1]

GLiNER/gliner/model.py

Line 100 in e15c22a

all_types_i = list(x['classes_to_id'][i].keys())

This will error out on a KeyError because it is using numeric indices to look up keys that can be strings.
This happens, when entity_types are provided to the create_dataloader

[urchade]

I am not sure to understand

Is it for training or inference?

[Ahmedn1]

@urchade training. Following this notebook with the only difference of providing entity_types to training script. Like so:

train_loader = model.create_dataloader(train_data, batch_size=config.train_batch_size, shuffle=True, entity_types=config.entity_types)

[urchade]

Is entity types in the correct format? It should be a list of string

Actually I do not suggest setting entity types during training for better generalization

[Ahmedn1]

@urchade yes it is a list of strings

[tcourat]

I have the same issue when specifying entity_types in create_dataloader .

The issue is here :

When identity_type is set to none, then class_to_ids is indeed a LIST of dictionnaries (one for each sentence in the batch) :

GLiNER/gliner/modules/base.py

Line 62 in e15c22a

class_to_ids = []

But when providing identity_type, then it is a dictionnary :

GLiNER/gliner/modules/base.py

Line 113 in e15c22a

class_to_ids = {k: v for v, k in enumerate(entity_types, start=1)}

So a quick fix, when using entity_type, is to deal with each case by removing this line :

GLiNER/gliner/model.py

Line 100 in e15c22a

all_types_i = list(x['classes_to_id'][i].keys())

And instead add :

if isinstance(x["classes_to_id"], list) :
    all_types_i = list(x["classes_to_id"][i].keys())
elif isinstance(x["classes_to_id"], dict) :
    all_types_i = list(x["classes_to_id"].keys())

[urchade]

So, you want to fix the label during training, for supervised fine-tuning ?

The solution for this is to add the key "label" to each training samples (i.e in addition to "tokenized_text" and "ner"). You can do it as follows:

for i in range(len(train)):
    train[i]["label"] = labels

@Ahmedn1 @tcourat