Skip to content


Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Powerful Benchmarker

Benchmarking Validation Methods for Unsupervised Domain Adaptation


Clone this repo:

git clone

Then go into the folder and install the required packages:

cd powerful-benchmarker
pip install -r requirements.txt

Set paths in constants.yaml

  • exp_folder: experiments will be saved as sub-folders inside of exp_folder
  • dataset_folder: datasets will be downloaded here. For example, <dataset_folder>/mnistm
  • conda_env: (optional) the conda environment that will be activated for slurm jobs
  • slurm_folder: slurm logs will be saved to <exp_folder>/.../<slurm_folder>
  • gdrive_folder: (optional) the google drive folder to which logs can be uploaded

Folder organization

Visit each folder to view its readme file.

Folder Description
latex Code for creating latex tables from experiment data.
notebooks Jupyter notebooks
powerful_benchmarker Code for hyperparameter searches for training models.
scripts Various bash scripts, including scripts for uploading logs to google drive.
unit_tests Tests to check if there are bugs.
validator_tests Code for evaluating validation methods (validators).

Useful top-level scripts

Delete all slurm logs:

python --delete

Or delete slurm logs for specific experiments groups. For example, delete slurm logs for all experiment groups starting with "officehome":

python --delete --exp_group_prefix officehome

Kill all model training jobs:


Or kill all validator test jobs:

python --validator_tests

Print how many hyperparameter trials are done:


Include a detailed summary of validator test jobs:

python --with_validator_progress

Save to progress.txt instead of printing to screen:

python --save_to_file progress.txt

A simple way to run a program via slurm.

For example, run for all experiment groups starting with "office31", using a separate slurm job for each experiment group:

python --command "python validator_tests/" --slurm_config_folder validator_tests \
--slurm_config a100 --job_name=collect_dfs --cpus-per-task=16 --exp_group_prefix office31

Or run a program without considering experiment groups at all:

python --command "python validator_tests/" --slurm_config_folder validator_tests \
--slurm_config a100 --job_name=zip_dfs --cpus-per-task=16

Upload slurm logs and experiment progress to a google drive folder at regular intervals (the default is every 2 hours):


Set the google drive folder in constants.yaml.

Citing the paper

Coming soon...

Looking for A Metric Learning Reality Check?

Checkout the metric-learning branch.