Monitoring utility for machine learning experiments

Rarog is a monitoring utility for machine learning experiments. You may use it as a light-weight alternative for TensorBoard or Visdom. Rarog stores all records in ClickHouse database using ClickHouse Python Driver.

Features

• log common python data types(bool, int, float, string, iterables)
• log 1d numpy arrays
• distributed experiments monitoring

Setup

Install via pip:

pip install rarog

Start ClickHouse database if required:

docker run -d --name clickhouse --ulimit nofile=262144:262144 -p 9000:9000 yandex/clickhouse-server

Important note: the example above is just the easiest way. For production, you should setup database backups or replicated.

Rarog supports Python 3.4 and newer.

Usage

import random

from rarog import Tracker

tracker = Tracker(name='experiment_name')

# trace values one by one
for step in range(10):
    tracker.trace(
        name='int_value',
        value=random.randint(10, 20),
        step=step)
    tracker.trace(
        name='float_value',
        value=random.random(),
        step=step)
    # provide experiment phase as a string
    tracker.trace(
        name='list_value',
        value=[random.random(), random.random()],
        step=step,
        phase='val')

# trace values by dict
for step in range(10, 20):
    tracker.multy_trace({
        'int_value': random.randint(10, 20),
        'float_value': random.random()
    }, step=step)

# get names of traced metrics
tracker.metrics
# Out: ['time', 'step', 'phase', 'int_value', 'float_value', 'list_value']

If you are going to record more than 100 entries per second, it's better to use sync_step or sync_seconds arguments. Thus writing to the database will be done with some period, which is much faster.

# `exist_ok` flag allow to use the same name for experiment
step_tracker = Tracker(name='experiment_name', sync_step=1000, exist_ok=True)

for step in range(20, 10**4):
    step_tracker.trace(name='bool_value', value=bool(random.randint(0, 1)), step=step)
    step_tracker.multy_trace({
        'int_value': random.randint(10, 20),
        'float_value': random.random()
    }, step=step)

# tracker should be manually synchronized after last entry
step_tracker.sync_accumulated_values()

Experiments can be handled via manager

from rarog import Manager

manager = Manager()
manager.list_experiments()
# Out: ['experiment_name']

manager.remove_experiment('experiment_name')

Retrieving your data

TODO (manually and with visualization)

TODO

• Pytorch tensors support
• Support 2d arrays
• Tensorflow data types support
• Split Aggregation View for summarization and underlying tables
• Store experiments metadata(config, author, etc.)
• Autodocs