Checkpoints for transformer & mlp policy

Transformer Policy

模型参数：

    model = PPO(
        CustomTransformerPolicy,
        env,
        learning_rate=linear_schedule(3e-4),
        n_steps=512,
        batch_size=128,
        n_epochs=10,
        gamma=0.99,
        gae_lambda=0.95,
        clip_range=0.2,
        normalize_advantage=True,
        ent_coef=0.01,
        vf_coef=0.5,
        max_grad_norm=0.5,
        tensorboard_log=tensorboard_log,
        policy_kwargs=dict(
            d_model=64,
            nhead=2,
            num_encoder_layers=1,
            num_decoder_layers=1,
            patch_size=1,
            optimizer_class=torch.optim.Adam,
            optimizer_kwargs=dict(eps=1e-5),
        ),
        verbose=1,
        device=device,
    )

环境参数：

    env = gym.make(
        env_id,
        container_size=(10, 10, 10),
        n_boxes_range=(5, 15),
        use_position_mask=False
    )

训练步数：约 15w 步

MLP Policy

模型参数：

BinPacking3D-v0:
  normalize: true
  n_envs: 128
  n_timesteps: !!float 2e8
  policy: 'MultiInputPolicy'
  batch_size: 2048
  n_steps: 2048
  gamma: 0.99
  learning_rate: !!float 3e-4
  ent_coef: 0.01
  clip_range: 0.2
  n_epochs: 20
  gae_lambda: 0.95
  max_grad_norm: 0.5
  vf_coef: 0.5
  policy_kwargs: "dict(
                    net_arch=dict(pi=[512, 512, 256], vf=[512, 512, 256])
                  )"

env ：

env = gym.make(env_id, container_size=(10, 10, 10), n_boxes_range=(10, 10))