Skip to content
Open solution to the Cdiscount’s Image Classification Challenge
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
CODE_OF_CONDUCT.md Create CODE_OF_CONDUCT.md Apr 26, 2018
CONTRIBUTING.md Create CONTRIBUTING.md Apr 26, 2018
ISSUE_TEMPLATE.md
LICENSE Create LICENSE Apr 26, 2018
README.md Update README.md Oct 5, 2018
__init__.py
data_config.yaml initial Oct 18, 2017
experiment_config.yaml initial Oct 18, 2017
models.py replacing tabs with spaces (#5) Apr 26, 2018
pipelines.py replacing tabs with spaces (#5) Apr 26, 2018
postprocessing.py
preprocessing.py
requirements.txt
run_manager.py Workaround for neptune bug in parsing properties Dec 14, 2017
run_neptune_command.sh renamed main file Oct 21, 2017
utils.py Update utils.py Oct 19, 2017

README.md

What is Cdiscount starter?

This is ready to use, end-to-end sample solution for the currently running Kaggle Cdiscount challenge.

It involves data loading and augmentation, model training (many different architectures), ensembling and submit generator.

More competitions 🎇

Check collection of public projects 🎁, where you can find multiple Kaggle competitions with code, experiments and outputs.

Disclaimer

In this open source solution you will find references to the neptune.ml. It is free platform for community Users, which we use daily to keep track of our experiments. Please note that using neptune.ml is not necessary to proceed with this solution. You may run it as plain Python script 😉.

How to run Cdiscount starter?

Installation

  1. Install the requirements

    pip install -r requirements.txt
  2. Install neptune by simply

    pip install neptune-cli
  3. Finish neptune installation by running

    neptune login
  4. Finally, open neptune and create project cdiscount. Check the project key because you will use it later (most likely it is: CDIS).

Now, you are ready to run the code and train some models...

Run code

remark about the competition data: We have uploaded the data to the neptune platform. It is available in the /public/cdiscount directory. Moreover, we created the meta_data file for large .bson files in the /public/Cdiscount/meta directory. It makes the process way faster.

You can run this end-to-end solution in two ways:

  • If you wish to work on your own machine you can run
    neptune run run_manager.py -- run_pipeline
  • Deploying on cloud via neptune is super easy
    • just go

      source run_neptune_command.sh
    • more advanced option is to run

      neptune send run_manager.py \
      --config experiment_config.yaml \
      --pip-requirements-file requirements.txt \
      --project-key CDIS \
      --environment keras-2.0-gpu-py3 \
      --worker gcp-gpu-medium \
      -- run_pipeline

Collect results and upload to Kaggle

Navigate to /output/project_data/submissions, get your submission file, upload it to Kaggle and check your rank in the competition!

Advanced options

custom data directories

If you do not wish to use default data directories, you can specify custom paths in the data_config.yaml

raw_data_dir: /public/Cdiscount
meta_data_dir: /public/Cdiscount/meta
meta_data_processed_dir: /output/project_data/meta_processed
models_dir: /output/project_data/models
predictions_dir: /output/project_data/predictions
submissions_dir: /output/project_data/submissions

meta data creation

If you want to create meta data locally you should run

python run_manager create_metadata

and your metadata will be stored in the meta_data_dir

data sampling

Since the dataset is very large we suggest that you sample training dataset to a manageable size. Something like 1000 most common categories and 1000 images per category seems reasonable to start with. Nevertheless, You can tweak it however you want in the experiment_config.yaml file

properties:
  - key: top_categories
    value: 100
  - key: images_per_category
    value: 100
  - key: epochs
    value: 10
  - key: pipeline_name
    value: InceptionPipeline

hyperparameter space search

If you like to search the hyperparameter space, neptune can do this for you. Check out hyperparameter optimization.

training without neptune

We give you an option to run this code without neptune. The transition is seamless, just follow these steps:

  1. Download the competition data to some folder your_raw_data_dir

  2. specify data directories in the data_config.yaml

  3. run python code

      python run_manager.py run_pipeline

Final remarks

Please feel free to modify this code in order to improve your score. Add new models, pre- and post-processing routines or ensembling methods.

Have fun competing on this Kaggle challenge!

You can’t perform that action at this time.