Prototype leaderboard for evaluations, originally written by Claudia Hauff. What the leaderboard system essentially does is allow people to easily set up an online competition for which participants are expected to submit entries of which the correctness needs to be assessed.
Two directories are included in the repository that contain examples of competitions that were held in real life. Specifically, the
placing leaderboard was used to evaluate the submissions to the MediaEval Placing Task and the
tagcaption leaderboard was used for the Yahoo-Flickr Challenge on Tag and Caption Prediction, both held in 2016. In both
evaluation subdirectories you will find the data that was used to evaluate uploaded submissions, as well as baselines that the organizers of the competitions provided that the participants were expected to outperform. Note that the
tagcaption leaderboard incorporates code from the Microsoft COCO Caption Evaluation repository of Tsung-Yi Lin.
Install Node.JS and NPM
- For OS X this can be simply done using Homebrew by running
brew install node.
- For RHEL6 machines this can be done by running
curl -sL https://rpm.nodesource.com/setup_6.x | sudo -E bash -.
npm installin the directory where the archive was unpacked to install the dependencies the leaderboard needs in order to run.
Create the leaderboard
- Look at the structure and contents of both the
tagcaptiondirectories for inspiration.
- Add, remove or modify the front-end files (the .pug files in the 'views' subdirectory and the corresponding images and stylesheets in the 'public' subdirectory) and the back-end files (the .js files) so it looks and works the way you want.
- Replace the files in the 'evaluation' subdirectory with your own data, and change the email templates in the 'emailer' directory.
Configure the leaderboard
- Update the path in the
app.jsfile to point to the subdirectory where your leaderboard resides.
- Replace the placeholder login/password of an STMP-enabled email account in the
config.jsonfile located in the leaderboard subdirectory. The easiest is to create a new Gmail address, enable two-factor authentication, and then create an app-specific password (this will allow you to hide the real password of the account and to easily create disposable passwords, which is useful in the case of an accidental commit to GitHub).
- Start the application with
npm startfrom the directory where the archive was unpacked.
- Access the leaderboard locally at
http://localhost:8080, or at the port you specified in the config file.
- Consider using Amazon AWS or another cloud computing service to easily host the leaderboard online.