Duplicate category value across features crash during training evaluation

[carlogrisetti]

Training a model with multiple output features results in a valueerror in the evaluation phase of the first epoch.
It's not related to the tied category value, even two "simple" categorical outputs gave the error.
Switching to a single categorical value (one or the other) solves the issue and allows for the training to continue.

multiple_output.zip

I don't know if it's only related to this specific combination or it happens with multiple outputs of any type. Will try to investigate further...

[carlogrisetti]

Error has the first shape member set to the batch size. Changing the batch size of course changes the error details but presents the same issue.
Other "main-level" parameters do not affect the behavior and still produce the error.
Preprocessing the input images does not fix the issue (not related to the actual images)
Having a text input does not fix the issue (not related to image input)
Adding a third output does not fix the issue (not related to the numerosity of the output features, in the sense that only 1 output feature works, more than 1 breaks)
Removing the dependency between the two output features does not fix the issue
Using one or another output feature singularly does fix the issue (not related to the specific output feature data)
Not forcing the split and using the split column in the source csv does not fix the issue (not related to preprocessing\split, and it does not even get to validation\test, it breaks on training evaluation alone)

[carlogrisetti]

Both output features to text DOES solve the issue
One output feature category and one output feature text DOES solve the issue

I do not have a dataset and time to test 2 category + 1 text, but it seems almost like the combination of 2 category output features is breaking something. As long as there is only one category output, it works fine

[tgaddair]

Hey @carlogrisetti, is it possible that training and eval sets have a different number of distinct values for the categorical features? Looks like there is an off-by-one error between the output and target, suggesting different vocab sizes.

[carlogrisetti]

Ok, found out how to repro, and can include full sample dataset.
Issue arises where you define two category columns that share the same category names. Ludwig maps them onto a distinct list, hence the shape difference.

Here it is:
multiple-output-same-category.zip

Once you get different category names across different columns, it all works (just tested on the original dataset where I found it, i had the same category name for 2 output features. Once one of them was renamed to be different, it all worked fine)

[w4nderlust]

I figured out what the issue is, working on a fix. Very good catch, thank you, this only show up when you have two output features of the same type.

[w4nderlust]

Should be solved, please confirm if that's true also for your real usecase.

[carlogrisetti]

Confirmed, thanks!