fix: unify logger

[LovelyBuggies]

I have put up with slight inconsistencies in the metric logging parameter names across different algorithms for a long time. Today, I finally decided I’d had enough, so I opened a PR.

[LovelyBuggies]

This pull request aims to improve the consistency of logs across different algorithms. Currently, for all three algorithms, the results are consistent within each panel to within WandB.

TLDR and arXiv

For both, MAGRPO reached 8,000 steps in about 4 hours; MAAC and IAC took about 12 hours to reach 7,000+ steps on B200. The total VRAM usage for all algorithms is under 80 GB. VRAM for MAGRPO and IAC shared is a bit over 40 GB.

CoopHE

MAGRPO reached 10,000 steps in about 5 hours; MAAC and IAC took about 9 hours to reach 8,000+ steps on B200. IAC (not shared-AC) uses the most memory, reaching approximately 130GB of VRAM; MAAC uses the second most, reaching over 110GB of VRAM; MAGRPO and IAC (shared-AC) use the least memory, at about 80GB.

HouseBuild

The performance of Pulse isn't that stable, so you might need to do a few more runs.

MAGRPO reached 4,000 steps in about 8 hours; MAAC and IAC took about 4 hours to reach 1,000+ steps on B200. I only tested IAC with shared actor-critic. The VRAM usage is about 140 GB, which is similar to MAAC and seems to be the limit for the H200. In contrast, MAGRPO uses slightly less VRAM, coming in at under 120 GB.