provide basic integration of trec 2d emb to training pipeline #2929
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
Apr 30, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
force-pushed
the
export-D70988786
branch
from
4a38377 to
fbb9472
Compare
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
Apr 30, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
force-pushed
the
export-D70988786
branch
from
fbb9472 to
7695cc8
Compare
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
Apr 30, 2025
…ytorch#2929) Summary: Pull Request resolved: meta-pytorch#2929 * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
force-pushed
the
export-D70988786
branch
from
7695cc8 to
4133ec5
Compare
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
There was a problem hiding this comment.
Pull Request Overview
This PR integrates a basic configuration to support DMP collection syncing via a new configuration value in the training pipelines. Key changes include adding helper functions (get_class_name and assert_instance), updating the sparse train pipeline to optionally sync DMPs every N batches, and adding tests to validate the new sync functionality.
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| torchrec/distributed/utils.py | Added helper functions get_class_name and assert_instance for improved type error messages. |
| torchrec/distributed/types.py | Introduced a stub for prefetch to support future extensions. |
| torchrec/distributed/train_pipeline/utils.py | Updated type checks and stream handling using assert_instance. |
| torchrec/distributed/train_pipeline/train_pipelines.py | Integrated a new sync_embeddings function that optionally syncs DMP collection based on a configurable interval. |
| torchrec/distributed/train_pipeline/tests/test_train_pipelines.py | Added tests to verify proper sync behavior and disable sync when DMP is not used. |
| torchrec/distributed/embedding_types.py | Updated the prefetch method’s parameter type to accept Multistreamable instances. |
|
|
||
|
|
||
| def assert_instance(obj: object, t: Type[_T]) -> _T: | ||
| assert isinstance(obj, t), f"Got {get_class_name(obj)}" |
There was a problem hiding this comment.
[nitpick] Consider including the expected type in the assertion error message (e.g., 'Expected {t}, got {get_class_name(obj)}') to improve debuggability.
Suggested change
| assert isinstance(obj, t), f"Got {get_class_name(obj)}" | |
| assert isinstance(obj, t), f"Expected {t.__name__}, got {get_class_name(obj)}" |
Copilot uses AI. Check for mistakes.
Comment on lines
+119
to
+120
| f"{self.__class__.__name__} does not support context (not expected). " | ||
| "Embedding weight sync is disabled." |
There was a problem hiding this comment.
[nitpick] Consider clarifying or documenting in the warning message when the context is absent so that it’s clear under what conditions embedding sync is disabled.
Suggested change
| f"{self.__class__.__name__} does not support context (not expected). " | |
| "Embedding weight sync is disabled." | |
| f"{self.__class__.__name__}: Embedding weight synchronization requires a valid " | |
| "TrainPipelineContext. No context was provided, so embedding sync is disabled. " | |
| "Ensure that a TrainPipelineContext is passed to enable this feature." |
Copilot uses AI. Check for mistakes.
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: Pull Request resolved: meta-pytorch#2929 * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: Pull Request resolved: meta-pytorch#2929 * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
xunnanxu
added a commit
to xunnanxu/torchrec
that referenced
this pull request
May 1, 2025
…ytorch#2929) Summary: * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
…ytorch#2929) Summary: Pull Request resolved: meta-pytorch#2929 * add `dmp_collection_sync_interval_batches` as a config value to SDD pipeline (and semi sync) * default to 1 (every batch). * disable using `None` * disabled if DMPC not used Differential Revision: D70988786
|
This pull request was exported from Phabricator. Differential Revision: D70988786 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
dmp_collection_sync_interval_batchesas a config value to SDD pipeline (and semi sync)NoneDifferential Revision: D70988786