Problem or motivation
currently, if a custom evaluator requires a specific dependency, we don't do anything special about it -- i.e. we'll run the evaluator and if it fails, it fails. this means user will have to install the dependencies in the evaluator manually.
Proposed solution
Haven't thought through what the best approach here would be, but we'd need to figure out a better way to handle the custom evaluator dependencies and install them.
One approach would be to check if the custom evaluator contains the requirements.txt file and then install the dependencies before executing the evaluator. We'd probably need a new virtual environment for the custom evaluator and install/execute it within that.
Human confirmation
Problem or motivation
currently, if a custom evaluator requires a specific dependency, we don't do anything special about it -- i.e. we'll run the evaluator and if it fails, it fails. this means user will have to install the dependencies in the evaluator manually.
Proposed solution
Haven't thought through what the best approach here would be, but we'd need to figure out a better way to handle the custom evaluator dependencies and install them.
One approach would be to check if the custom evaluator contains the
requirements.txtfile and then install the dependencies before executing the evaluator. We'd probably need a new virtual environment for the custom evaluator and install/execute it within that.Human confirmation