Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Clone this wiki locally
Evaluation scripts are Groovy scripts, compiled as subclasses of EvalScript. Using Groovy's meta-programming support for creating domain-specific languages along with builder objects, they are high-level scripts that can rather easily run evaluations.
At the top level, EvalTask classes provide the entry point to configuring a recommender. Commands are registered by name in
META-INF/lenskit-eval/methods/<method>.properties files on the classpath. For example, the train-test evaluator is registered with the following content in
This tells the configurator that, when looking for a method called
trainTest, instantiate the task [TrainTestEvalCommand], configure it using an appropriate constructor, invoke its
execute() method, and return the output. If the last argument to the method is a closure, it invoked to configure the builder prior to calling
Methods can also be registered for builders; in this case, the
build() method is called rather than
execute(). Also, unlike tasks, builders are always run immediately, even when they are used within a target.
Configuring builders and tasks
When a command is configured with a closure, the closure is invoked with appropriate bridge object as its delegate. This delegate is responsible for handing individual directives. The default delegate is DefaultConfigDelegate, which resolves directives against the methods of the command using reflection. For a directive
- If the value is a future that has not yet completed, schedule the directive to actually be called when the future is available.
- If there exists a
setFoomethod, it is used. Arguments are converted using Groovy's standard conversions; in addition, a single
Stringargument can be be converted to a
Fileand classes with public no-arg constructors are automatically instantiated. If no method directly exists, but there is a single-argument
setFoowhose type has an available builder, the builder is instantiated with the non-closure arguments to the directive, configured with the closure (and a builder delegate), and the method is used.
- If there's an
addFoo, it is used just as
setFoo, with one additional rule: if given a single argument which implements
Xis the parameter type to
addFoois called multiple times, once with each element of the list. This is how adding the crossfolder as a dataset works, as it produces multiple data sets.
- Otherwise, look for a named builder or task and instantiate it, looking in
META-INF/lenskit-eval/methodsas described above.
Therefore, the JavaDoc for the various builder classes serves as a complete reference to the configuration directives available for any particular class.
An alternate delegate can be configured for a builder by annotating the builder class with the ConfigDelegate annotation.
If a type is annotated with the BuiltBy annotation, then the configuration engine will attempt to use the builder specified in the annotation to build it when needed (e.g. to fill the argument of a
Builders can also be registered in properties files. The
META-INF/lenskit-eval/builders.properties file maps class names to builder class names. All files with that path are read from the classpath to locate builders.