Gymnasium RL environments for SuperScript

Custom Gymnasium environments (RLD1 / RLD2 families) used to train and evaluate the hierarchical RL team-selection policies.

---

gym-superscript

Custom Gymnasium reinforcement-learning environments for SuperScript, an agent-based model (ABM) of team formation and allocation in project-based organisations. These environments are used to train and evaluate the hierarchical RL team-selection policies (RLD1 / RLD2) reported in the accompanying article.

This repository is one of three components of the SuperScript optimisation codebase. It is a git submodule of superscript-abm, which is in turn a submodule of OptimizationExperiments (the top-level repository containing the article experiments and reproduction instructions).

What it provides

The active environment families (registered in gym_superscript/__init__.py, classes in gym_superscript/envs/):

Family	Class	Role
RLD1-*	RLD1	Hierarchy level 1: hard-skill allocation of a given team onto a project. Wraps a live SuperScript ABM instance. IDs: RLD1-v0, RLD1-v1.214, RLD1-v3.234, RLD1-v4.454, RLD1-v4.645, RLD1-v4.645-linear, RLD1-v4.645-nonlinear.
RLD2-*	RLD2	Hierarchy level 2: worker/team selection from a 100-worker workforce. Uses a trained RLD1 PPO agent (shipped in gym_superscript/trained_agents/) as its lower-level allocator. IDs: RLD2-v0, RLD2-v2.38, RLD2-v3.106, RLD2-v3.106-linear, RLD2-v3.106-nonlinear.
RLD2_50-*	RLD2_50	50-worker variant of RLD2. IDs: RLD2_50-v1, RLD2_50-v1.37.
RLD2_<N>-*	RLD2_FILTERED	RLD2 on a workforce pre-filtered to N workers (random / index / ML random-forest filter). IDs: RLD2_10-v1, RLD2_15-v1.62-nonlinear, RLD2_20-v1.62-linear, RLD2_20-v1.62-nonlinear, RLD2_30-v1.62-nonlinear, RLD2_40-v1, RLD2_100-v1.

Environment behaviour (worker counts, ABM parameters, linear vs nonlinear success function, evaluation normalisation) is driven by the YAML configs in gym_superscript/envs/*.yaml.

Also present are a number of legacy flat (non-hierarchical) environments (SSEnvMinimal-*, SSEnvAllocateHardSkills*, SSEnvStatic*, SSEnvDynamic*, SSEnvOptimalBaseline*). At this snapshot their entry points are registered but the classes are not exported from gym_superscript.envs, so gym.make() on those IDs will fail; they also depend on an older superscript_model package. They are retained for historical completeness only — use the RLD families.

Known registration quirks at this snapshot (kept as-is because the shipped trained agents and the article results were produced against them):

• RLD1-v3.234 points at rld1_v1.214_config.yaml (not the rld1_v3.234_config.yaml present on disk).
• RLD2_40-v1 registers filtered_worker_count: 50.

Requirements

setup.py declares only gymnasium, but the environments are not standalone. At import/run time they additionally require, in the same Python environment:

• superscript_abm (the superscript-abm package — the envs instantiate and step the ABM in-process);
• rl_zoo3 (from the sibling rl-baselines3-zoo fork submodule) and stable-baselines3 — needed by the RLD2 family to load the trained RLD1 sub-agent;
• numpy, networkx, confuse, PyYAML.

The environment in which the article results were produced is pinned in article_requirements.txt of the top-level OptimizationExperiments repository (Python 3.10, numpy 1.26.4, gymnasium 0.29.1, stable-baselines3 2.3.2; torch 1.13.1 installed separately).

Install

Normally installed as part of the recursive checkout of OptimizationExperiments (see its README for the full sequence). Standalone:

pip install -e .          # after installing superscript-abm and the zoo fork

Usage

import gymnasium as gym
import gym_superscript  # registers the env IDs

env = gym.make('RLD2_20-v1.62-nonlinear')
obs, info = env.reset()

Note that reset() on the RLD family builds and steps a SuperScript ABM instance, so it is not instantaneous. Training for the article was performed through the rl-baselines3-zoo fork (train.py --algo ppo --env <ID>), which imports this package to register the IDs; the tuned hyperparameters per env ID live in that fork's hyperparams/ppo.yml.

Verifying reproduction

The canonical end-to-end verification of the article results (which exercises these environments through the RL optimiser) is the stage-1 golden regression suite in the top-level OptimizationExperiments repository (tests/regression/). See the README there.

License and copyright

Copyright (C) 2025 Michael Christen michael.christen@mobi.ch

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See LICENSE for the full text.