CARTerpillar Environment - Generalizing CartPole to Multiple Carts

In Summary

Standard RL benchmarks are often fixed, making it difficult to observe how strategies scale for different complexities. CARTerpillar is an extension of the classic CartPole environment to multiple carts, where the number of carts $C$ to be balanced is a parameter that controls the difficulty of the environment. The $C$ carts form a fully-connected system, with all possible pairs of carts $i, j$ linked by a damper (with a constant factor of $k_{i,j} = k$) and a spring (with a constant factor of $b_{i,j} = b$). Adding an $i$-th cart increases the number of connections by $i-1$, making the dynamics more complex, and expands the discrete action space by two actions (push left/right for the new cart). At each timestep, the agent can only move one cart left or right; however, each cart is also subject to a force that is due to all the dampers and springs connecting them.

How To Use

This is a gym-based implementation of the CARTerpillar environment. The env defined in carterpillar.py strictly adheres to the gym standards to help users adopt it as fast as possible.

To experiment with it, you just need to have the gymnasium library installed. Then, you can clone this repo and start using it as follows:

from gymnasium.wrappers import TimeLimit
from CARTerpillar import CARTerpillarEnv

env = CARTerpillar(n_carts=5)
env = TimeLimit(env, max_episode_steps=500)
state, _ = env.reset()
state, reward, done, info = env.step(env.action_space.sample())