Pin Transformers to 4.31.0

[arnavgarg1]

It seems like the latest transformers package has a bug. This can be fixed by either using transformers master, or downgrading to 4.31.0 which is stable and works correctly.

This is the error we run into with 4.32.1:

site-packages/ludwig/api.py:1510), in LudwigModel.preprocess(self, dataset, training_set, validation_set, test_set, training_set_metadata, data_format, skip_save_processed_input, random_seed, **kwargs)
   1508     return PreprocessedDataset(proc_training_set, proc_validation_set, proc_test_set, training_set_metadata)
   1509 except Exception as e:
-> 1510     raise RuntimeError(f"Caught exception during model preprocessing: {str(e)}") from e
   1511 finally:
   1512     for callback in self.callbacks:
RuntimeError: Caught exception during model preprocessing: local variable 'tokens' referenced before assignment

Closes: #3568

[tgaddair]

Can you file an issue with HF transformers? It might be an API issue on our end, but either way we probably need them to weigh in.

Also, please add a TODO to the requirements file to revert this once the issue is resolved.

[tgaddair]

Please fix lower bound.

[github-actions]

Unit Test Results

       6 files  ±0         6 suites  ±0   1h 34m 21s ⏱️ + 4m 43s
2 826 tests ±0  2 790 ✔️ +2  12 💤 ±0  24 ❌  - 2 
2 869 runs  ±0  2 824 ✔️ +2  21 💤 ±0  24 ❌  - 2 

For more details on these failures, see this check.

Results for commit 20a532f. ± Comparison against base commit 63f4924.

♻️ This comment has been updated with latest results.

[arnavgarg1]

@tgaddair seems like the issue was flagged and fixed on transformers master: huggingface/transformers#25805

[arnavgarg1]

Associated Ludwig issue: #3571