☁️ 🚀 📊 📈 Evaluating state of the art in AI
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Docs: Fix broken hyperlink in Pull Request Template (#1465) Jan 14, 2018
apps Frontend: Bug fix in private leaderboard logic (#2000) Jan 16, 2019
docker CI/CD: Add ssl folder to code directory while building image (#1972) Dec 21, 2018
docs Docs: Update common errors during installation section (#1974) Jan 10, 2019
evalai Docs: Add docs for APIs (#1849) Oct 23, 2018
examples/example1 Update sample challenge evaluation script to support Python3 (#1855) Oct 23, 2018
frontend Frontend: Change max character limit for affiliation (#1901) Jan 10, 2019
middleware Update Python version to 3.6.5 in Dockerfiles & Python3 migration dep… Oct 17, 2018
requirements Fix# 1982: Fix security issue related to PyYAML dependency(#1983) Jan 7, 2019
scripts CI/CD: Minor fix related to travis branch (#1969) Dec 17, 2018
settings Fix #1986 (#1987) Jan 13, 2019
static S3 storage and RDS configurations added Oct 21, 2016
tests Challenges: Fix bug in displaying challenge by pk (#1943) Dec 12, 2018
.coveragerc Settings: Update django project settings (#1610) Mar 30, 2018
.coveralls.yml Fix: Coveralls issue reloved (#163) Nov 28, 2016
.dockerignore Update docker configuration and optimize docker image size(#1860) Oct 24, 2018
.eslintrc Frontend: Allow console debug statement in development (#983) Jun 1, 2017
.gitignore Update .gitignore: Add JETBrain's IDE code editor settings(#1976) Jan 6, 2019
.landscape.yml Coda Quality: Added pylint configuration and modified landscape confi… Jan 11, 2017
.pylintrc Coda Quality: Added pylint configuration and modified landscape confi… Jan 11, 2017
.travis.yml CI/CD: Add AWS CLI in Travis config (#1920) Nov 21, 2018
CODE_OF_CONDUCT.md Docs: Add Code of Conduct (#865) Mar 23, 2017
LICENSE Fix #1022: Change license from GPL to BSD (#1023) May 29, 2017
README.md Docs: Update common errors during installation section (#1974) Jan 10, 2019
bower.json Upgrade(Gulp): Version bump 3.9.1 -> 4.0.0 (#1697) Jun 16, 2018
docker-compose-production.yml CI/CD: Minor fixed in deploy script for production environment (#1966) Dec 17, 2018
docker-compose-staging.yml CI/CD: Add production tooling script (#1916) Nov 20, 2018
docker-compose.yml Frontend: Allow live changes sync in docker container (#1992) Jan 11, 2019
elasticmq-server-0.14.2.jar Backend: Replace RabbitMQ with SQS (#1752) Sep 4, 2018
gulpfile.js Frontend: Move dependencies to work with production env (#1930) Nov 28, 2018
karma.conf.js Frontend: Add unit tests for auth controller (#1468) Feb 10, 2018
manage.py Settings: Update django project settings (#1610) Mar 30, 2018
package.json Frontend: Move dependencies to work with production env (#1930) Nov 28, 2018
pytest.ini Fix: Coveralls issue reloved (#163) Nov 28, 2016
setup.cfg Coda Quality: Added pylint configuration and modified landscape confi… Jan 11, 2017

README.md


Join the chat at https://gitter.im/Cloud-CV/EvalAI Build Status Coverage Status Requirements Status Code Health Code Climate Documentation Status

EvalAI is an open source web application that helps researchers, students and data-scientists to create, collaborate and participate in various AI challenges organized round the globe.

In recent years, it has become increasingly difficult to compare an algorithm solving a given task with other existing approaches. These comparisons suffer from minor differences in algorithm implementation, use of non-standard dataset splits and different evaluation metrics. By providing a central leaderboard and submission interface, we make it easier for researchers to reproduce the results mentioned in the paper and perform reliable & accurate quantitative analysis. By providing swift and robust backends based on map-reduce frameworks that speed up evaluation on the fly, EvalAI aims to make it easier for researchers to reproduce results from technical papers and perform reliable and accurate analyses.

A question we’re often asked is: Doesn’t Kaggle already do this? The central differences are:

  • Custom Evaluation Protocols and Phases: We have designed versatile backend framework that can support user-defined evaluation metrics, various evaluation phases, private and public leaderboard.

  • Faster Evaluation: The backend evaluation pipeline is engineered so that submissions can be evaluated parallelly using multiple cores on multiple machines via mapreduce frameworks offering a significant performance boost over similar web AI-challenge platforms.

  • Portability: Since the platform is open-source, users have the freedom to host challenges on their own private servers rather than having to explicitly depend on Cloud Services such as AWS, Azure, etc.

  • Easy Hosting: Hosting a challenge is streamlined. One can create the challenge on EvalAI using the intuitive UI (work-in-progress) or using zip configuration file.

  • Centralized Leaderboard: Challenge Organizers whether host their challenge on EvalAI or forked version of EvalAI, they can send the results to main EvalAI server. This helps to build a centralized platform to keep track of different challenges.

Goal

Our ultimate goal is to build a centralized platform to host, participate and collaborate in AI challenges organized around the globe and we hope to help in benchmarking progress in AI.

Performance comparison

Some background: Last year, the Visual Question Answering Challenge (VQA) 2016 was hosted on some other platform, and on average evaluation would take ~10 minutes. EvalAI hosted this year's VQA Challenge 2017. This year, the dataset for the VQA Challenge 2017 is twice as large. Despite this, we’ve found that our parallelized backend only takes ~130 seconds to evaluate on the whole test set VQA 2.0 dataset.

Installation Instructions

Setting up EvalAI on your local machine is really easy. You can setup EvalAI using two methods:

Using Docker

You can also use Docker Compose to run all the components of EvalAI together. The steps are:

  1. Get the source code on to your machine via git.

    git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd evalai

    Use your postgres username and password for fields USER and PASSWORD in settings/dev.py file.

  2. Build and run the Docker containers. This might take a while. You should be able to access EvalAI at localhost:8888.

    docker-compose up --build
    
  3. That's it. Open web browser and hit the url http://127.0.0.1:8888.

    SUPERUSER- username: admin password: password
    HOST USER- username: host password: password
    PARTICIPANT USER- username: participant password: password

Using Virtual Environment

  1. Install python 2.7.10 or above, git, postgresql version >= 10.1, have ElasticMQ installed (Amazon SQS is used in production) and virtualenv, in your computer, if you don't have it already. If you are having trouble with postgresql on Windows check this link postgresqlhelp.

  2. Get the source code on your machine via git.

    git clone https://github.com/Cloud-CV/EvalAI.git evalai
  3. Create a python virtual environment and install python dependencies.

    cd evalai
    virtualenv venv
    source venv/bin/activate  # run this command everytime before working on project
    pip install -r requirements/dev.txt
  4. Create an empty postgres database.

    sudo -i -u (username)
    createdb evalai
    
  5. Change Postgresql credentials in settings/dev.py and run migrations

    Use your postgres username and password for fields USER and PASSWORD in dev.py file. After changing credentials, run migrations using the following command:

    python manage.py migrate --settings=settings.dev
    
  6. Seed the database with some fake data to work with.

    python manage.py seed --settings=settings.dev
    

    This command also creates a superuser(admin), a host user and a participant user with following credentials.

    SUPERUSER- username: admin password: password
    HOST USER- username: host password: password
    PARTICIPANT USER- username: participant password: password

  7. That's it. Now you can run development server at http://127.0.0.1:8000 (for serving backend)

    python manage.py runserver --settings=settings.dev
    
  8. Please make sure that node(>=7.x.x), npm(>=5.x.x) and bower(>=1.8.x) are installed globally on your machine.

    Install npm and bower dependencies by running

    npm install
    bower install
    

    If you running npm install behind a proxy server, use

    npm config set proxy http://proxy:port
    
  9. Now to connect to dev server at http://127.0.0.1:8888 (for serving frontend)

    gulp dev:runserver
    
  10. That's it, Open web browser and hit the url http://127.0.0.1:8888.

  11. (Optional) If you want to see the whole game into play, then install the ElasticMQ Queue service and start the worker in a new terminal window using the following command that consumes the submissions done for every challenge:

    python scripts/workers/submission_worker.py
    

If you are facing any issue during installation, please see our common errors during installation page.

The Team

EvalAI is currently maintained by Deshraj Yadav, Akash Jain, Taranjeet Singh, Shiv Baran Singh and Rishabh Jain. A non-exhaustive list of other major contributors includes: Harsh Agarwal, Prithvijit Chattopadhyay, Devi Parikh and Dhruv Batra.

Contribution guidelines

If you are interested in contributing to EvalAI, follow our contribution guidelines.