👗 Outfit

Outfit is a lightweight library to tidy up your machine learning experiments in a simple way.

The idea of Outfit is to store in your Wardrobe your parameters, output file, scores and features in order to be able to make a request and find out which are your best experimentation according to a given criterion.

How install outfit ?

PyPI:

pip install outfit

Dev version:

git clone https://github.com/AdilZouitine/outfit
cd outfit
pip install -r requirements.txt
pip install -e .

How outfit works ?

• Tutorial 1: Build a simple model management pipeline with outfit for a CNN with Pytorch on MNIST dataset

import datetime

# Here import all the libraries you need for your experiment

from outfit import Wardrobe, getlog

wardrobe = Wardrobe(db_path='foo/bar/mnist.db')

exp = {
    'experiment_name': 'ResNet18',
    'comment': 'Use differential learning rate',
    'date_experiment': datetime.datetime.now()
}

wardrobe.add_experiment(**exp)

param = {
    'dropout': 0.20,
    'kernel_size': '3x3',
    'conv_block_1_lr': 0.01,
    'conv_block_2_lr': 0.001
   }

# Create the instance of your model here with your parameters
wardrobe.add_dict_parameter(param)


# Do your training phase here.
output = {'training log': '/result/training_log_resnet18.txt'}

@getlog(filepath=output['training log'])
def train_model(model, loaders, loss, lr_scheduler, n_epoch):
    ...

output.update({
    'tensorboard': '/result/event.tb',
    'model': 'diff_lr_resnet18.pth'
})

wardrobe.add_dict_output(output)

score = {
    'train acc': 0.96,
    'train loss': 0.430,
    'val acc': 0.94,
    'val loss': 0.460
     }
wardrobe.add_dict_score(score)


wardrobe.tidy() # commit your experiment in database

# If you want to get the best experiments 

for exp in wardrobe.get_best_scores(mode='max',on_score='val acc'):
    '''
    Verbose is true by default and will print on the console 
    at each iteration the parameters, output file, 
    features and scores in a table format.

    Also returns in dictionary the parameters, output file, features and scores.
    '''
    ...

Output:

════════════════════
│ TOP 1 EXPERIMENT │
════════════════════



Table : Experiment 

╒════╤═════════════════╤═══════════════════╤════════════════════════════════╤════════════════════╕
│    │   id_experiment │ experiment_name   │ comment                        │ date_experiement   │
╞════╪═════════════════╪═══════════════════╪════════════════════════════════╪════════════════════╡
│  0 │               1 │ ResNet18          │ Use differential learning rate │                    │
╘════╧═════════════════╧═══════════════════╧════════════════════════════════╧════════════════════╛


Table : Parameter 

╒════╤════════════════╤══════════════════╤═════════════╤══════════════╕
│    │   id_parameter │ parameter_name   │ parameter   │   experiment │
╞════╪════════════════╪══════════════════╪═════════════╪══════════════╡
│  0 │              1 │ dropout          │ 0.2         │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  1 │              2 │ kernel_size      │ 3x3         │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  2 │              3 │ conv_block_1_lr  │ 0.01        │            1 │
├────┼────────────────┼──────────────────┼─────────────┼──────────────┤
│  3 │              4 │ conv_block_2_lr  │ 0.001       │            1 │
╘════╧════════════════╧══════════════════╧═════════════╧══════════════╛


Table : Output 

╒════╤═════════════╤═══════════════╤═══════════════════════════════════╤══════════════╕
│    │   id_output │ type_output   │ path_output                       │   experiment │
╞════╪═════════════╪═══════════════╪═══════════════════════════════════╪══════════════╡
│  0 │           1 │ training log  │ /result/training_log_resnet18.txt │            1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│  1 │           2 │ tensorboard   │ /result/event.tb                  │            1 │
├────┼─────────────┼───────────────┼───────────────────────────────────┼──────────────┤
│  2 │           3 │ model         │ diff_lr_resnet18.pth              │            1 │
╘════╧═════════════╧═══════════════╧═══════════════════════════════════╧══════════════╛


Table : Score 

╒════╤════════════╤══════════════╤═════════╤══════════════╕
│    │   id_score │ type_score   │   score │   experiment │
╞════╪════════════╪══════════════╪═════════╪══════════════╡
│  0 │          1 │ train acc    │    0.96 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  1 │          2 │ train loss   │    0.43 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  2 │          3 │ val acc      │    0.94 │            1 │
├────┼────────────┼──────────────┼─────────┼──────────────┤
│  3 │          4 │ val loss     │    0.46 │            1 │
╘════╧════════════╧══════════════╧═════════╧══════════════╛


Table : Feature 

Other solution:

mlflow & dvc.

These solutions are great, they also offer a user interface and have many more options than my library however for a simple use where you only want to organize your experimentation and make a simple query. Both solutions seem to be overkill.