-
Notifications
You must be signed in to change notification settings - Fork 463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable Global Weight Decay for VBE #2507
Conversation
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
This pull request was exported from Phabricator. Differential Revision: D56200676 |
This pull request was exported from Phabricator. Differential Revision: D56200676 |
9436f4f
to
72f21ee
Compare
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Differential Revision: D56200676
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
1 similar comment
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
Summary: Pull Request resolved: pytorch#2507 Enable Global weight decay for VBE --- **Usage:** set ``` optimizer = OptimType.EXACT_ROWWISE_ADAGRAD weight_decay_mode = WeightDecayMode.DECOUPLE_GLOBAL # for VBE, pass batch_size_per_feature_per_rank # Example: num_features = 2, num_ranks = 4 batch_size_per_feature_per_rank = [ [1, 2, 8, 3] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 0 [6, 10, 3, 5] # batch sizes for [Rank 0, Rank 1, Rank 2, Rank 3] in Feature 1 ] ``` e.g., ``` tbe = SplitTableBatchedEmbeddingBagsCodegen( embedding_specs=[ (E, D, managed_option, ComputeDevice.CUDA) for (E, D) in zip(Es, Ds) ], optimizer=OptimType.EXACT_ROWWISE_ADAGRAD, learning_rate=0.1, eps=0.1, output_dtype=output_dtype, pooling_mode=pooling_mode, weight_decay_mode=WeightDecayMode.DECOUPLE_GLOBAL, ) output = tbe(indices, offsets, batch_size_per_feature_per_rank=batch_size_per_feature_per_rank) ``` Relevant diffs: D53866750 D55660277 D55660762 Reviewed By: sryap Differential Revision: D56200676
This pull request was exported from Phabricator. Differential Revision: D56200676 |
This pull request has been merged in 114bb0d. |
Summary:
Enable Global weight decay for VBE
Usage:
set
e.g.,
Relevant diffs:
D53866750
D55660277
D55660762
Differential Revision: D56200676