Skip to content
No description, website, or topics provided.
Jupyter Notebook Python Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Benchmark for platforms comparison


This repository is an application for the comparison of machine learning platforms which we made in Neuromation. From our point of view, for a fair comparison it is necessary to run the same typical task on each participating platform.

In the master branch, you can find the code and commands required to run the task on your local gpu server. For each platform, we created an additional branch so that you can clearly see what commands and changes you need to apply to the project to run the proposed task.

Disclaimer. Please note that we do not use techniques to improve the accuracy of the model (such as data augmentations, learning rate scheduling, etc), since our goal is to compare platforms. As a result, we made the code as simple as possible.

Task definition

As a benchmark, we propose to solve the task of images classification on CIFAR10 dataset. The dataset consists of 60000 32x32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

We have prepared images in .png format arranged in folders here [140 MB]. This dataset is also accessible by writing one line of code in pytorch, but for the purpose of this comparison we work with data as if it was a custom dataset.

While working with each platform we follow a specific scenario and write down all our actions in (please, change a branch to see scenario implementation for each platform).

Comparison results

At all points of comparison, we rated 0, 1, 2, or 3.

Developer Experience ML Environments Data Ingestion AI Starter Kits Collaboration Bring Your Own Cloud Enterprise-ready Total
Neuro Platform 3 3 3 3 3 2 1 18
AWS SageMaker 1 1 3 3 3 1 3 15
Google AI Platform 2 0 0 3 3 1 3 12
Azure Machine Learning 3 3 2 3 3 1 3 18
FloydHub 2 2 3 3 2 3 2 17
Paperspace Gradient 2 3 2 3 3 3 1 17
Determined AI 2 1 2 3 3 3 2 16
You can’t perform that action at this time.