Write tests for calling CLI arguments downstream to ensure correctly-returned types

[veekaybee]

Issue: Currently we don't assert that passed-in args are the correct type passed to downstream code, causing runtime exceptions. See example of issue here.

There's a couple approaches we can try here, from least effort to most as far as I see it, most manual to most automated:

1. Adding assert statements to all current arg-level actions in cli_evaluate()
2. Writing several unit tests that take in mock parsed args and unit-testing only cli_evaluate()
3. Breaking up cli_evaluate() into likely into two methods, one that starts from the method signature, and one that starts at line 309, where the evaluation logger starts, so we can test the part of the method where we assign variables from the CLI directly, and unit test that,

Happy to discuss any of these and open to other approaches if they've been previously discussed, and also happy to take this as an issue.

[LSinev]

In my opinion, raising proper errors in the main code (not the tests) is preferable to using asserts.
Pros:

• this repo can be used as a submodule or package in a larger pipeline that can try to catch specific bugs.
• described here https://stackoverflow.com/a/40183030

Proper exceptions can also be pytested:
https://stackoverflow.com/questions/23337471/how-do-i-properly-assert-that-an-exception-gets-raised-in-pytest

[veekaybee]

One downside of raising errors in the code ahead of merge to main is that you'd have to know ahead of runtime that the code you're adding to the command line args requires new error handling. But, if all of the arguments are checked in a pre-merge test, you wouldn't be able to run the package to begin with and merge the code.

What we'd like to test is automatically adding new command line args that get parsed correctly downstream:

As a specific case, in this case, this environment variable should be a string rather than a boolean:

   if args.trust_remote_code:
        os.environ["HF_DATASETS_TRUST_REMOTE_CODE"] = (
            args.trust_remote_code if args.trust_remote_code else True
        )
        args.model_args = (
            args.model_args
            + f",trust_remote_code={os.environ['HF_DATASETS_TRUST_REMOTE_CODE']}"
        )

Ultimately this causes an error in cli_evaluate(), so we could perhaps start by adding try/catch exceptions to each of the input CLI args.

[LSinev]

Maybe pydantic can help somehow
pydantic/pydantic-settings#209
https://www.youtube.com/watch?v=7aBRk_JP-qY
https://youtu.be/zN4VCb0LbQI?t=514

this environment variable should be a string rather than a boolean

As defined in documentation
https://docs.python.org/3.8/library/os.html#os.environ
A mapping object representing the string environment.
So, this specific problem should probably be catched by typecheckers at pre-commit time.

[veekaybee]

Closed with #1566