Allow loading models and tasks from external packages

[lorenzomammana]

At the current stage there's a somehow simple way to include new tasks using the --include_external flag, yet there's no way to include external models except from cloning the lmms-eval repository and doing modifications on it, which might not be the ideal scenario in many cases (for example when developing new models internally).

In my scenario I would like to work on an external package (let's say it's an lmms-eval plugin) without the necessity of cloning this repository (using it as a package)

Right now I've done a fork of the repository which allows loading external tasks and models using an environment variable called LMMS_EVAL_PLUGINS to include external repositories (like LMMS_EVAL_PLUGINS=package1,package2), particularly in the __main__.py I'm able to integrate new tasks in this way

if os.environ.get("LMMS_EVAL_PLUGINS", None):
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
        eval_logger.info(f"Including path: {args.include_path}")
        include_path(package_tasks_location)

Assuming that there's a package installed which follows the same structure as lmms-eval, this piece of code retrieves all the content of {package_name}.tasks and includes them as tasks.

In a similar way I've modified the __init__.py of models

if os.environ.get("LMMS_EVAL_PLUGINS", None):
    # Allow specifying other packages to import models from
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        m = importlib.import_module(f"{plugin}.models")
        for model_name, model_class in getattr(m, "AVAILABLE_MODELS").items():
            try:
                exec(f"from {plugin}.models.{model_name} import {model_class}")
            except ImportError:
                pass

In a similar way assuming that there's an external package that follows the same structure as lmms-eval, this piece of code will read the AVAILABLE_MODELS dict from the external package and register new models to be used by lmms-eval.

Does it seem something that you can consider to integrate? I could partially contribute with a PR

[kcz358]

Hi, I think it is a cool feature. We didn't implement it because the original repo lm-eval also does not contain this feature. But feel free to implement and PR one if you have some thoughts on it and we will try to look into it and work on it.