[ECML 2025] BatMan-CLR: Making Few-shots Meta-Learners Resilient Against Label Noise

This repository contains the work for BatMan-CLR, providing a package for Gradient Based Meta-Learners (GMBL), and their capability of handling different types of label noise. Besides the implementation used for the paper, this repo provides a framework (nmfl) for Noisy Meta-Learning with Few-shot Learning. It provides a composable framework for defining re-usable noise on dataset and task-level.

N.B. Experiments are intended to be run using Slurm for job scheduling and Apptainer as the container runtime (bootstrapped from Docker). We provide the scripting and tooling used to deploy these experiments, but may require adaptation to your own system.

N.B. To run experiments locally (on a single machine), refer to the experiment definitions in slurm, to see the launch configurations for the experiments

Setup

Python Environment

Ensure that you're using python < 3.10.0, more recent versions of python have not been tested.

Docker + Apptainer Setup for Slurm Execution

This project is designed for containerized execution on Slurm clusters via Apptainer, using a Docker image as a base to boostrap from, please refer to Dockerfile

1. Build the Docker image:

   docker build . --tag diff-maml

2. Convert Docker image to a Apptainer image:

   apptainer build diff-maml.sif slurm/image/build.def

3. Submit jobs using Slurm and Apptainer: Use the .sif image within your Slurm batch scripts or job commands. Example:

   apptainer exec --nv diff-maml.sif python3 train.py --config config/default.json

   For concrete experiments, please refer to the experiments to be deployed with sbatch' in [slurm`](./slurm)

Make sure that your Slurm node has GPU support and Apptainer with --nv support (for NVIDIA passthrough). In addition, make sure that the GPU is available during execution, which depends on your cluster configuration.

Data Preparation

Before running the experiments, it is recommended to prepare the data to be used in the experiments. Please contact @JMGaljaard if the data cannot be retrieved using the existing scripts. We depend on learn2learn to create these datasets, as the NoisyFewShotDatasets`, are wrappers around their functionality.

By running an experiment on a single node / locally and rsyncing the data directory to your nodes' storage/NFS ensures that datasets are correctly created. You might need to update the learn2learn dependency / manually collect the data once.

Meta-learning Algorithms

Available Learners

We provide (re-)implementations of a series of meta-learners, as used in our paper.

• GMBL (l2l)
  • MAML (+ first-order MAML) (l2l)
    • First-order MAML
    • Almost No Inner Loop (ANIL)
    • Body Only Inner Loop (BOIL)
• iMAML (CG method)
• REPTILE
• Lazy MAML (HF-approximation)
• EigenReptile

Configure (Single) Experiment

You can run an experiment by activating your virtual environment and using a configuration file:

source venv/bin/activate
python3 train.py --config config/default.json

Example Configuration

{
  "runtimeConfig": {
    "seed": 42,
    "cuda": true,
    "download": true
  },
  "learnerConfig": {
    "algorithm": "reptile",
    "regularization": null
  },
  "trainingConfig": {
    "hyperParamConfig": {
      "innerLR": 0.2,
      "outerLR": 0.003,
      "regularizationCoeff": 0.5,
      "innerLoopSteps": 1,
      "metaBatchSize": 32
    },
    "experimentConfig": {
      "dataset": "omniglot",
      "ways": 5,
      "shots": 5,
      "testSamples": 15,
      "numberBatches": 60000,
      "numberWorkers": 1,
      "testInterval": 5
    },
    "executionConfig": {
      "saveModel": false,
      "saveEpochInterval": 1,
      "saveModelPath": "models",
      "epochSaveStartSuffix": "checkpoint",
      "epochSaveEndSuffix": "final",
      "tensorboard": true,
      "tensorboardDir": "logging"
    }
  }
}

Note: A progress bar is implemented using tqdm. Having a high testInterval value may cause runtime estimates to spike during evaluations.

N.B.: Training and testing share the same hyper-parameters.

Running Locally with Docker

To build and run experiments inside Docker locally:

docker build . --tag diff-maml
docker run --gpus all --rm -i -t diff-maml
...

Running on HPC (Slurm + Apptainer)

If you're running on a cluster with Slurm and Apptainer:

1. Build Docker and Apptainer images as shown above.

2. Create a Slurm batch script, e.g., run_experiment.slurm:

   #!/bin/bash
   #SBATCH --job-name=meta-exp
   #SBATCH --gres=gpu:1
   #SBATCH --time=02:00:00
   #SBATCH --output=slurm-%j.out
   
   apptainer exec --nv diff-maml.sif python3 train.py --config config/default.json

3. Submit the job:

   sbatch run_experiment.slurm