- Project's slack channel.
Data scientists working on machine learning problems have historically had several issues relating to evaluating their systems: spending time individually developing evaluation frameworks for tasks, comparing results over time, and keeping evaluations consistent among teams. OpenEval is a system designed to address these problems.
In developing this system we set out to build a centralized, easy-to-use platform for groups to evaluate their models. All the user needs to do to evaluate their solver is host it on a thin server, which we provide. Then, on the web interface, they need to select their desired task and dataset to test their solver. After their solver finishes processing the dataset, the user can view the results.
The project contains two main modules.
- The OpenEval core, which contains the main functionalities and the web app.
- A Learner, which acts as a toy system to be evaluated against core.
Quick Guide on Running the Apps
Your will need Java 8 in order to run App. openjdk on Ubuntu seems to have issues.
You will also need sbt
sbt from the parent directory.
projectswill show the names of the existing module names.
project corewill take you inside the core package.
project learnerwill take you inside the examples package.
- Inside each project you can
run the core you can browse to
localhost:9000. To run on specific port simply add the port number after
You also should not have to start it multiple times. You can just save code, and refresh the page.
Note: OpenEval server needs to store its backend information in a SQL database. If you want to run it on your machine
first create a SQL DB, rename
core/conf/application.conf, and add the
DB information (url, username and password).