Integrating planner stats db with ConfigeratorStats #3331
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 5, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
ab7622e to
1104959
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
force-pushed
the
export-D81216987
branch
from
1104959 to
52f17fe
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
force-pushed
the
export-D81216987
branch
from
52f17fe to
eac3ed7
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
pushed a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: Pull Request resolved: meta-pytorch#3331 internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
eac3ed7 to
68b3e4c
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: Pull Request resolved: meta-pytorch#3331 internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
68b3e4c to
0425d29
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
0425d29 to
0262617
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
0262617 to
2aa69bd
Compare
|
This pull request was exported from Phabricator. Differential Revision: D81216987 |
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: Pull Request resolved: meta-pytorch#3331 internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
2aa69bd to
eedcbe5
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 11, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 16, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 16, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
a185637 to
52b2a9f
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 16, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 16, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 16, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
52b2a9f to
933ee24
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
933ee24 to
ce0b8b6
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
ce0b8b6 to
9cb13f1
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 17, 2025
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
…torch#3327) Summary: internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. **How is a sharding plan stored in Configerator?** The Thrift definition of a sharding plan includes two fields: Topology and Dict[int, ShardingOption]. 1. Topology: The Topology field contains the information mentioned in this diff D79142495. 2. Dict[int, ShardingOption]: This field represents a dictionary where each key is a 64-bit hash of a sharding option, and the value is the corresponding Thrift-converted sharding option. The hash is calculated using the storage_hash function within the ShardingOption object, which takes into account factors such as the fqn, sharding type, and compute kernel. **How can a loaded plan be merged with an enumerated search space?** **Background:** When a plan is preserved during the logging stage, a hash is generated to ensure that the same plan can be loaded and validated later. The [hash is calculated](https://www.internalfb.com/code/fbsource/[fdf90ff2be9041f867bc6c9e4aec6ee94862fa11]/fbcode/torchrec/distributed/planner/types.py?lines=1010-1026) using input fields such as topology, batch size, constraints, storage reservation, and storage reservation policy, as well as fields from the sharding options like fqn, sharding type, kernel type, shards, and cache parameters. Once the plan is loaded and validated, we can safely assume that all loaded sharding options are a 1:1 map of enumerated sharded options. During the loading process, we traverse the enumerated search space, calculate the storage hash for each sharding option, look up the corresponding sharding option from the loaded plan, and replace the Shards of the enumerated sharding option with those of the loaded sharding option. This approach enables us to generate precise sharding options that can be seamlessly converted into a sharing plan as done by the planner and this also ensures consistent logging while also facilitating plan replay. Reviewed By: ge0405 Differential Revision: D81185992
Summary: internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner stats db 3. Integration of planner stats db with ConfigeratorStats Reviewed By: ge0405 Differential Revision: D81216987
aliafzal
force-pushed
the
export-D81216987
branch
from
9cb13f1 to
4eac24e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
internal
Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan.
This Diff:
Differential Revision: D81216987