Guide to Using Comet.ml with Beaker
Terminology in Beaker and Comet.ml have some overlap, and it is important to point out that what Comet.ml names an "Experiment" maps to what Beaker names a "Task."
Your Comet.ml Account
An account at Comet.ml is necessary for using their analytics features. Beaker can create a free-tier account on your behalf, or you can link an existing Comet.ml account with your Beaker account. The free-tier accounts include one private project and unlimited public projects, and paid subscription tiers are also available with additional benefits (free for academic users).
Using an Existing Comet.ml Account
If you already have an account with Comet.ml and would like to link it with Beaker, follow these steps:
- Sign in to Beaker and go to your User page by clicking on your name in the top right.
- In the section titled "Comet.ml," enter your username on Comet.ml and click "Link Account..."
- Accept the confirmation prompt. Please note that linking your Comet.ml and Beaker accounts cannot be undone, and accounts may only be linked on a 1-to-1 basis.
- Check your EMail. You will receive a confirmation message from Comet.ml, and need to manually confirm the account link before Beaker can create experiments on your behalf.
Creating a New Account
If you create an experiment on Beaker which uses Comet.ml, but do not have a linked account, Beaker will use your EMail address on file and create an account on your behalf. You will receive an EMail from Comet.ml asking you to reset your password, and then you will be able to log in to your account and view Experiment data. Alternatively, you may sign up on the Comet.ml website and follow the instructions above to manually link your accounts.
Note to Academic Users: The account Beaker creates on Comet.ml uses their free tier, though academic users may request free access to the paid tiers by contacting email@example.com.
Using Comet.ml in Beaker Experiments
In the Code
Beaker will create all the necessary Comet.ml experiments for a Beaker experiment when it is submitted. For Python experiments, it is recommended to use the
ExistingExperiment class (see Python docs here). Beaker will provide your API key and Experiment key through environment variables, so the bare minimum code to intialize Comet.ml in your experiment is (Python example):
# Before importing the ML library of your choice from comet_ml import ExistingExperiment # Before first sending any metrics, parameters, etc. to Comet.ml # log_code and log_env_details will instruct Comet to include your code and environment details in the results. experiment = ExistingExperiment(log_code=True, log_env_details=True)
experiment object is what you will use to interact with Comet.ml within your task. Use it to log parameters, charts, metrics, and other data to the experiment in Comet.ml.
Runnability Outside Beaker
Because Beaker is handling some of the Comet.ml configuration on your behalf, it is important to note it is up to the user to supply these environment variables when running a Beaker image outside of Beaker.
|COMET_API_KEY||This is your API Key as seen in the Comet.ml UI|
|COMET_EXPERIMENT_KEY||The key of the experiment previously generated in Comet.ml.
Alternatively, the code in the image can be written to determine whether an experiment key is present and use the
Experiment class instead if it is not. In Python, for example, this can be accomplished with the following code:
experiment = None if os.getenv("COMET_EXPERIMENT_KEY") == None: experiment = Experiment() else: experiment = ExistingExperiment(log_code=True, log_env_details=True)
In the Spec
The Comet.ml integration is enabled on a per-experiment basis, by adding the following to your experiment spec:
comet: enable: true
enable is set to true, Beaker will create one experiment on Comet.ml for each task defined in the spec. There are optional parameters that may be included, as well:
comet: # Required to enable the integration enable: true # Optional name to give this experiment. experiment: my-experiment-name # Optional name of the project to place the Comet.ml experiments under. project: my-project # Optional name of the workspace to place the experiments under. If not given, uses the default workspace created with your account. workspace: my-workspace
Links to each Comet.ml experiment will be displayed on the terminal and also on each task's details page in the UI. Please refer to Comet.ml's documentation for more information about Projects and Workspaces.