[Training] Understand new llama3.1 405B RCPs which changed after v5.0

[ShriyaRishab]

In v5.0, llama3.1 405b benchmark was added and RCPs were generated for the same. For v5.1, the llama3.1 405B eval schedule changed by doing more frequent evaluations and skipping a few evaluations in the beginning: mlcommons/training_policies#563. This lead to new RCPs #424.

The aim of this issue is to understand how the new RCPs relate to the old ones and figure out if there is still a way to compare llama3.1 405b results from v5.0 that were scaled to the old RCPs to results from future rounds which would be scaled to the new RCPs.

[ShriyaRishab]

Analyze v5.0 scores with new RCPs and compare to scores with old RCPs -

• Most scores reduced by exactly by 3% although scores for batch sizes from 1216 to 1280 did not change
• Ordering of results does not change

[pgmpablo157321]

Looking at the scaling.json files, we can see that the difference is exclusive due to the scaling factor:

[pgmpablo157321]

Here are the three logs to check the submission and generate the scaling files
log_IBM+CoreWeave+NVIDIA.txt
log_nebius.txt
log_NVIDIA.txt

[pgmpablo157321]

Process to generate the new scores:

• Take main branch
• Replace the llama3.1-405b 5.0.0 RCPs with the llama3.1-405b 5.1.0 RCPs
• Run the 5.0.0 submission checker, this creates the new scaling factors
• Run the 5.0.0 result summarizer and export the results to csv

[ShriyaRishab]

There was a bug in the package checker where old scaling.json files don't get deleted. After manually deleting them we see these new scores

Scores are different by up to 3% with the new RCPs.

To fix the package checker issue, @pgmpablo157321 to help add a check in the RCP checker init() which looks for scaling.json file and deletes it if present in the results dir. After this is done, any new scaling factor is written to a new scaling.json if necessary. This ensures that stable scaling.json files don't get used.

[ShriyaRishab]

WG decision - Given that new RCPs are added in other benchmarks after many rounds and we never go back and run the RCP checker from new rounds to old round results - we can treat this the same way. So nothing to be changed.