fix: MFU incorporates dp_degree now instead of world_size

[le1nux]

What does this PR do?

This PR fixes the MFU and throughput calculations by taking the dp degree into account instead of the world size. When we use parallelization strategies on top of FSDP, then the world size is different from the data parallel degree. This needs to be reflected in throughput and MFU metric calculations, as done by this PR.

Breaking Changes

• Configs need to be adapted to correctly use dp degree rather than world size.

Checklist before submitting final PR

• My PR is minimal and addresses one issue in isolation
• I have merged the latest version of the target branch into this feature branch
• I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
• I have run a sample config for model training
• I have checked that all tests run through (python tests/tests.py)
• I have updated the internal changelog (CHANGELOG_DEV.md)

[therealdavidos]

looks good!

[rrutmann]

Looks good to me. However, I’m wondering if we’re duplicating information between mesh_definition and device_mesh

[le1nux]

Looks good to me. However, I’m wondering if we’re duplicating information between mesh_definition and device_mesh

referencing now the mesh definition.