Visualizing Results

Warning This tutorial is not complete and there is an open issue about it.

---

Overview

This tutorial introduces tools to visualize and analyse JSON results generated with randopt. It is divided in two parts.

1. We'll introduce roviz and see how to use it to create and share standalone Web reports.
2. Using the randopt's programmatic API, we'll create a visualization tool to monitor the state of a hyperparameter search experiment.

Roviz.py

Web Interface Demo

Generating the Web Interface

Let's expand the simple example from the package overview as follows.

import randopt as ro
from random import random

f = lambda x, y: x**2 + y**2

experiment = ro.Experiment(name='simple_example',
                           params={
                                'x': ro.Normal(),
                                'y': ro.Normal(),
                           })

experiment.sample_all_params()
result = f(experiment.x, experiment.y)
experiment.add_result(result, data={
    'random_convergence': [random() / n for n in range(1, 21)]
})

In this case, we're again finding parameters x and y which minimize a 2-dimensional quadratic, and record their value together with a list of random numbers.

By running this file multiplie times, we record a couple of results as JSON summaries. We can then generate the web interface by calling

roviz.py randopt_results/simple_example

which should open your default browser with a web page similar to this one. To create this file, roviz goes through all saved JSON summaries and concatenates them as a JSON list. It then parses an HTML template and replaces placeholder values by the ones from the JSON summary. This means that all results are available in the HTML file and that a new file must be generated whenever new JSON summaries are created.

Using the Web Interface

The Web interface is divided in two sections: Plots and Table. The Plots sections - initially empty - will contain all the plots we create from the values of the JSON summaries. The Table section contains a large table whose rows are JSON summaries and columns are the parameters / data values for that particular summary. The table can be automatically sorted by clicking on its headers.

Let us now populate the Plots section. To do so, select some results from the table by clicking on their rows or select them all by clicking on the +/- All header. The checkbox in the first row should be checked, and the number of selected results near the top right of the table should be updated. Then click on the green Add Plot button. You should now see a new empty graph in the Plots section.

To interact with the graph, we can change the values of each axes by selecting them in the drop-down menu. For example, one could plot the effect of the x variable on the recorded result.

The content of list values (e.g. random_convergence) can be plotted individually by selecting List Indices on the other axis. Finally, we can also modify the title of a graph (in this case Plot 0) by clicking on its current value. More graphs can be added by following the same procedure, and they can be removed by clicking on the red remove button.

Note The graphs are generated using Plotly.js and are interactive. You can also export them to the plotly website, and directly work on them form there.

Sharing an HTML Report

Once we are satisfied with our Web report, we can export it for future access or sharing. To do so, click on the blue Export Plots. This will open a new browser tab which contains the code of your report, which you can then save as any HTML file. Upon opening this HTML file, you should see all your plots restored as they were in the original window.

The way exportation works is the following. A JavaScript function reads the code of the current page in order to recreate the initial HTML template. Then, the state of the dynamic Web application (AngularJS) is read and overwritten in the HTML template. This new HTML template is then opened in the new browser tab.

As an example, we plotted the influence of X and Y on the results, together with the random convergence values and saved this report. You can access this demonstration here.

Programmatic API

Now that we've seen how to generate HTML reports using roviz, let's have a look at how we can use the programmatic API to build custom visualization. Our task will be to write a live monitoring interface for hyperparameter optimization. The resulting script is available in in the example folder and will look like the following image.

Note The version available in the example folder has some more bells and whistles but relies exactly on the same API.

Since we'll use live plotting and fancy tables, we'll have to first install a few requirements.

pip install matplotlib terminaltables

Our monitoring program will be executed by calling python monitor.py path/to/experiment. Therefore we'll begin by parsing the experiment path and name, and will open the specified experiment.

exp_path = sys.argv[1]
if exp_path[-1] == '/':
    exp_path = exp_path[:-1]
exp_dir, exp_name = os.path.split(exp_path)
exp = ro.Experiment(exp_name, directory=exp_dir)

We'll also instanciate lists for values to be plotted tracked, namely the minimum and maximum results, number of experiments ran, and time elapsed since the monitoring started.

start_time = time.time()
timings = []
minimums = []
maximums = []
counts = []

Conveniently, randopt offers direct access to the minimum/maximum results via the .minimum() and .maximum() methods. Using those, we can write the core of our program.

while True:
    minimums.append(exp.minimum().result)
    maximums.append(exp.maximum().result)
    counts.append(exp.count())
    timings.append(time.time() - start_time)
    plot_statistics(counts, timings, minimums, maximums, exp_name)
    table = table_statistics(
        counts, timings, minimums, maximums, exp_name)
    print(table)
    time.sleep(5)

Namely, we'll fetch the minimum and maximum results every 5 seconds, track them in our instanciated lists, and then update our plot and print the summarizing table. Notice than when fetching the minimums/maximum results, randopt goes through the entire set of saved JSON summaries.

All that remains to be done is to write the functions table_statistics() and plot_statistics(). Since these mostly rely on external programs, we do not provide extensive explanations on their implementation.

Create Table of Statistics

def table_statistics(counts, timings, minimums, maximums, name='Experiment'):
    minimum = "{0:.3f}".format(minimums[-1])
    maximum = "{0:.3f}".format(maximums[-1])
    timing = "{0:.2f}".format(timings[-1])
    data = [
        ['Results Count', 'Minimum Result', 'Maximum Result', 'Time Elapsed'],
        [counts[-1], minimum, maximum, timing],
    ]
    table = SingleTable(data, name)
    table.inner_heading_row_border = True
    table.inner_row_border = True
    table.inner_column_border = True
    table.outer_border = False
    table.justify_columns = {0: 'center', 1: 'center', 2: 'center', 3: 'center'}
    return table.table

Plot Statistics

def plot_statistics(counts, timings, minimums, maximums, name='Experiment'):
    plt.ion()
    plt.clf()

    # Min subplot
    plt.subplot(211)
    plt.title('Experiment ' + name + ' Statistics')
    plt.plot(counts, minimums, label='Minimum')
    plt.legend()
    plt.ylabel('Result')

    # Min subplot
    plt.subplot(212)
    plt.plot(counts, maximums, label='Maximum')
    plt.legend()
    plt.xlabel('Number of experiments')
    plt.ylabel('Result')

    # This renders the figure
    plt.pause(0.05)