Fix param.ds_id not found bug when call 'deepspeed.runtime.swap_tensor.optimizer_utils.OptimizerSwapper._create_param_swap_info #5193
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix param.ds_id not found bug when call 'deepspeed.runtime.swap_tensor.optimizer_utils.OptimizerSwapper._create_param_swap_info' method under condition
![ds_id error](https://private-user-images.githubusercontent.com/6138625/307754791-38b03a56-5b27-4df8-a7c2-67bbbaf94a27.PNG?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwNDEzMTQsIm5iZiI6MTcxOTA0MTAxNCwicGF0aCI6Ii82MTM4NjI1LzMwNzc1NDc5MS0zOGIwM2E1Ni01YjI3LTRkZjgtYTdjMi02N2JiYmFmOTRhMjcuUE5HP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjJUMDcyMzM0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MmIyYWNjYzg0N2QyZjA1ZDU2NmMzOGRmMzM1YjYwZGFmMGQzY2QwZmYxMGQ4ZTBhNGJhZDc5MjkxYTRhN2JkMiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.sCtk5RFCzaNyftOX5oy-0QQhGBOi-NhN4z6lb6sUCWk)
that 'offload_optimizer' and 'offload_param' set to nvme path in zero_config.
In new version of Deepspeed(0.13.3), the deepspeed.runtime.swap_tensor.optimizer_utils.OptimizerSwapper has added 'parameter_id(param)' method to get param id, while in older version like 0.12.4 the origin method is 'id(param)'. This change causes a new bug: when call deepspeed.runtime.zero.stage3.DeepSpeedZeroOptimizer_Stage3._create_fp32_partitions method in init process,the 'self.optimizer_swapper.initialize_paraeters(parameters=[self.fp32_partitioned_groups_flat[i]], src_tensors=[unpinned_fp32_buffer])' method will use unpinned_fp32_buffer without ds_id under optimizer offload condition, which will throw a error : "AttributeError: 'Tensor' object has no attribute 'ds_id'"