Skip to content
A guide for how to use the Comet.ML integration in Beaker with example experiment code.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
mnist
LICENSE
README.md

README.md

Guide to Using Comet.ml with Beaker

Comet.ml is a service providing detailed result tracking and analytics for machine learning experiments. Learn more about Comet.ml and how to use its features by reading the FAQ or Docs.

Terminology in Beaker and Comet.ml have some overlap, and it is important to point out that what Comet.ml names an "Experiment" maps to what Beaker names a "Task."

Your Comet.ml Account

An account at Comet.ml is necessary for using their analytics features. Beaker can create a free-tier account on your behalf, or you can link an existing Comet.ml account with your Beaker account. The free-tier accounts include one private project and unlimited public projects, and paid subscription tiers are also available with additional benefits (free for academic users).

Using an Existing Comet.ml Account

If you already have an account with Comet.ml and would like to link it with Beaker, follow these steps:

  1. Sign in to Beaker and go to your User page by clicking on your name in the top right.
  2. In the section titled "Comet.ml," enter your username on Comet.ml and click "Link Account..."
  3. Accept the confirmation prompt. Please note that linking your Comet.ml and Beaker accounts cannot be undone, and accounts may only be linked on a 1-to-1 basis.
  4. Check your EMail. You will receive a confirmation message from Comet.ml, and need to manually confirm the account link before Beaker can create experiments on your behalf.

Creating a New Account

If you create an experiment on Beaker which uses Comet.ml, but do not have a linked account, Beaker will use your EMail address on file and create an account on your behalf. You will receive an EMail from Comet.ml asking you to reset your password, and then you will be able to log in to your account and view Experiment data. Alternatively, you may sign up on the Comet.ml website and follow the instructions above to manually link your accounts.

Note to Academic Users: The account Beaker creates on Comet.ml uses their free tier, though academic users may request free access to the paid tiers by contacting mail@comet.ml.

Using Comet.ml in Beaker Experiments

In the Code

Beaker will create all the necessary Comet.ml experiments for a Beaker experiment when it is submitted. For Python experiments, it is recommended to use the ExistingExperiment class (see Python docs here). Beaker will provide your API key and Experiment key through environment variables, so the bare minimum code to intialize Comet.ml in your experiment is (Python example):

# Before importing the ML library of your choice
from comet_ml import ExistingExperiment

# Before first sending any metrics, parameters, etc. to Comet.ml
# log_code and log_env_details will instruct Comet to include your code and environment details in the results.
experiment = ExistingExperiment(log_code=True, log_env_details=True)

The experiment object is what you will use to interact with Comet.ml within your task. Use it to log parameters, charts, metrics, and other data to the experiment in Comet.ml.

Runnability Outside Beaker

Because Beaker is handling some of the Comet.ml configuration on your behalf, it is important to note it is up to the user to supply these environment variables when running a Beaker image outside of Beaker.

Variable Purpose
COMET_API_KEY This is your API Key as seen in the Comet.ml UI
COMET_EXPERIMENT_KEY The key of the experiment previously generated in Comet.ml. ExistingExperiment constructor throws an error if this is not present.

Alternatively, the code in the image can be written to determine whether an experiment key is present and use the Experiment class instead if it is not. In Python, for example, this can be accomplished with the following code:

experiment = None
if os.getenv("COMET_EXPERIMENT_KEY") == None:
  experiment = Experiment()
else:
  experiment = ExistingExperiment(log_code=True, log_env_details=True)

In the Spec

The Comet.ml integration is enabled on a per-experiment basis, by adding the following to your experiment spec:

comet:
  enable: true

When enable is set to true, Beaker will create one experiment on Comet.ml for each task defined in the spec. There are optional parameters that may be included, as well:

comet:
  # Required to enable the integration
  enable: true

  # Optional name to give this experiment.
  experiment: my-experiment-name

  # Optional name of the project to place the Comet.ml experiments under.
  project: my-project

  # Optional name of the workspace to place the experiments under. If not given, uses the default workspace created with your account.
  workspace: my-workspace

Links to each Comet.ml experiment will be displayed on the terminal and also on each task's details page in the UI. Please refer to Comet.ml's documentation for more information about Projects and Workspaces.

You can’t perform that action at this time.