Getting Started

Getting started

Requirements

• Python 3.12 or newer
• uv

Clone the repository and install the locked environment:

git clone https://github.com/KirillHit/spiking_rl_lab.git
cd spiking_rl_lab
uv sync

Run an experiment

The default command trains REINFORCE with an artificial MLP on four parallel CartPole-v1 environments:

uv run spiking-rl-lab

The default agent uses device: cuda. Run the same configuration on CPU with:

uv run spiking-rl-lab agent.params.device=cpu

Use Hydra overrides to change individual values:

uv run spiking-rl-lab \
  agent.params.device=cpu \
  runner.seed=7 \
  trainer.params.timesteps=50000

Use a retained experiment configuration when an exact setup should be reproducible:

uv run spiking-rl-lab experiment=reinforce_cartpole_ann

Command-line overrides have the final say, so a frozen experiment can still be adapted to the current machine:

uv run spiking-rl-lab \
  experiment=reinforce_cartpole_ann \
  agent.params.device=cpu

Inspect the composed configuration without starting training:

uv run spiking-rl-lab --cfg job

Evaluate a checkpoint

Training automatically evaluates checkpoints/best_agent.pt when skrl produced that file. To evaluate a checkpoint explicitly, reuse the configuration that created it:

uv run spiking-rl-lab \
  experiment=reinforce_cartpole_ann \
  runner.mode=evaluate \
  runner.checkpoint_path=/absolute/path/to/best_agent.pt \
  agent.params.device=cpu

Evaluation uses trainer.eval_timesteps and deterministic policy actions: the categorical mode for discrete actions and the Gaussian mean for continuous actions.

Find outputs

Hydra creates one directory per run:

runs/<experiment_name>/<environment_id>/<timestamp>_<agent>/
├── .hydra/config.yaml
├── checkpoints/
├── hardware.json
├── run.log
└── run_metadata.json

Some files appear only when the corresponding stage completes. The composed Hydra config, Git commit and uncommitted diff, hardware information, metrics, logs, and best checkpoint are also sent to MLflow when available.

DagsHub is used by default. Override its repository through:

export DAGSHUB_REPO_OWNER=<owner>
export DAGSHUB_REPO_NAME=<repository>

If DagsHub cannot be reached, tracking falls back to experiments/mlflow.db. Open it with:

uv run mlflow ui --backend-store-uri sqlite:///experiments/mlflow.db

Common configuration mistakes

• env=pendulum_v1 is not a complete switch from the default experiment: Pendulum has continuous actions, while the default policy is categorical and the default network has two outputs. Pair a continuous environment with a Gaussian policy and the correct number of network outputs.
• Rendering and vectorization may be incompatible for some Gymnasium environments. Prefer render=false during parallel training.
• A checkpoint is compatible only with the same agent, policy, network shape, and preprocessing setup used to create it.

See Configuration and experiments for the configuration model and Algorithms for policy compatibility.