Dynamically set max_new_tokens based on output feature length, GMSL and model window size

[arnavgarg1]

TLDR; This fixes a bug in model.evaluation() that caused underreporting of performance metrics in a majority of cases. By setting max_new_tokens to the largest possible value needed, we ensure that the metric number from model.evaluate() are a good representation of true model performance.

---

This PR updates the config validation setter methods to ensure that the generation.max_new_tokens parameter in the model configuration is set correctly based on the maximum sequence length of the output features. This ensures accurate token generation for LLM model types. It also includes handling for different cases and provides informative logging for configuration changes.

Specifically:

1. If the user sets max_new_tokens in generation to a value != the default value, it is respected as is.
2. Otherwise, we fall back to the max_sequence_length set for the output feature since that is the max number of tokens it learns during fine-tuning.
3. Otherwise, we fall back to the global_max_sequence_length if neither case 1 or case 2 are met. This is an overestimation of the number of required tokens, but it is acceptable for now.
4. Finally, if none of case 1, 2 or 3 are met, set max_new_tokens to the model's window size / 2. This feels like a good tradeoff of inference/evaluation time vs likely covering a majority of the total tokens allocated to the output feature. Can update this value in the future.

[github-actions]

Unit Test Results

  6 files    6 suites   23m 2s ⏱️
12 tests   9 ✔️   3 💤 0 ❌
60 runs  42 ✔️ 18 💤 0 ❌

Results for commit d40b75f.

♻️ This comment has been updated with latest results.

[justinxzhao]

Interesting mechanic for grouping tests - curious if you saw this pattern recommended somewhere?

[arnavgarg1]

@justinxzhao Definitely didn't see it recommended anywhere, but I wanted to find a logical way to group these tests together since they're about the same "topic" but testing different aspects of it, so decided to write a class. Is that fine, or would you like me to just write 4 individual tests?

[arnavgarg1]

The idea in general is that since they're all testing the same function but different scenarios, it makes sense to either put them all in the same dedicated module for clarity or in some sort of container like a class. Typically you can just use parameterization but this one would require a lot of conditionals to be used in the test so I decided to skip it. Alternatively, I could also combine them all into one test. All options are ok - no strong preference.

[justinxzhao]

I can see this being a useful way to organize tests particularly for very large test files. It's a bit more to maintain, but seems sufficiently lightweight.

[arnavgarg1]

Got it, let me split them up!

[justinxzhao]

I can see this being a useful way to organize tests particularly for very large test files. It's a bit more to maintain, but seems sufficiently lightweight.