Simulation RL Environments, part 1: PortSimEnv v1. Ideas for v2, v3, post-training and data

[adithya-s-k]

Simulation RL Environments is an experimental, work-in-progress series: take real-world work, rebuild it as a deterministic simulation from its own records, and train agents in it with RL.

Part 1 is PortSimEnv v1: re-plan a week of container-ship dockings at a Port of Barcelona quay, built from the port's 2024 records (1,784 real calls), with gales, crane breakdowns, closures, emergencies and diverted traffic. One graded submit per episode, scored deterministically against a CP-SAT-proven optimum.

• Article: https://huggingface.co/spaces/FineEnvs/simulation-rl-environments
• Environment (OpenEnv Space: play an episode, MCP playground): https://huggingface.co/spaces/FineEnvs/PortSimEnv
• Eval (all 300 rollouts replayed in 3D): https://huggingface.co/spaces/FineEnvs/PortSimEnv-Eval
• Dataset (1,050 train and 50 eval tasks, the source calls, the eval rollouts): https://huggingface.co/datasets/FineEnvs/PortSimEnv
• Bucket (3D twin data, raw eval rollouts): https://huggingface.co/buckets/FineEnvs/PortSimEnv
• Collection: https://huggingface.co/collections/FineEnvs/simulation-rl-envs-6ac4eca150bd4b91790e2540
• Code: https://github.com/adithya-s-k/FineEnvs/tree/main/07-simulation-environments/portsim-v1

Eval on 50 held-out weeks (mean reward): GPT-6.1 Sol 0.89 · Claude Sonnet 5.5 0.78 · GLM-5.3-Flash 0.47 · Qwen3.8-2.4T 0.38 · GLM-5.3 0.31 · Qwen3.8-27B 0.21.

The open question: from seed data we can build realistic environments, but how do we make them as close to the real world as possible? Ideas, pointers and critiques are welcome on:

• v2, a live simulation: delays that arrive over time, control tower and ship communication, re-planning under urgency and partial information
• v3, more of the port: tugs, pilots, the yard, gates and rail; other ports
• Post-training: RL recipes on the 1,050 training tasks, and what to measure
• Data: AIS tracks, weather history, terminal logs, other ports' open data
• Other jobs worth simulating from public records

[earakely-scale]

v2, a live simulation: delays that arrive over time, control tower and ship communication, re-planning under urgency and partial information
v3, more of the port: tugs, pilots, the yard, gates and rail; other ports
Post-training: RL recipes on the 1,050 training tasks, and what to measure
Data: AIS tracks, weather history, terminal logs, other ports' open data
Other jobs worth simulating from public records

This fits into what we support in https://www.agentenvframework.com/ . Would love to collab on this! Happy to do a prototype as a showcase