[RFC] Unify device check and guard in kernel wrapper #55241
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
💊 CI failures summary and remediationsAs of commit 636d485 (more details on the Dr. CI page):
🕵️ 5 new failures recognized by patternsThe following CI failures do not appear to be due to upstream breakages:
|
This was referenced Apr 2, 2021
For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
…l wrapper" For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 3, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125666279 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
…rd in kernel wrapper" For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 4, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125690005 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
…pper" For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 4, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125691571 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
…pper" For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 4, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125692886 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 5, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125736005 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
wenleix
commented
Apr 5, 2021
For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 5, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparations are required: (1) Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. (2) Acquire the device guard on `common_device`. Today, these two preparations are done seperately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device gurad. There are at least two issues with the current appraoch: 1. Kernel may not impelemnt the device check, or implement in inconsistent way. In fact, there are a few implementations in codebase to do the checks. 2. If the first tensor happnened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile.. ghstack-source-id: 125769869 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/)
wenleix
commented
Apr 6, 2021
| @@ -4347,6 +4350,7 @@ | |||
| SparseCUDA: hspmm_sparse_cuda | |||
|
|
|||
| - func: copy_sparse_to_sparse_(Tensor(a!) self, Tensor src, bool non_blocking=False) -> Tensor(a!) | |||
| device_guard: False | |||
There was a problem hiding this comment.
This calls into Tensor::to :
pytorch/aten/src/ATen/SparseTensorUtils.h
Lines 34 to 39 in bf37bf7
which handles device guard:
pytorch/aten/src/ATen/native/TensorConversions.cpp
Lines 83 to 85 in bf37bf7
There was a problem hiding this comment.
Maybe this should have been a composite haha
For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
…apper" For kernels run on CUDA backend, the following preparations are required: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two preparations are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 20, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 126921579 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
…pper" For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 20, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 126924122 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 20, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 126967104 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 21, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 127011444 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 22, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 127145739 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
|
One issue that cause CUDA memory issue is because of |
…ard in kernel wrapper" For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
|
We found make TI to fetch device guard is a bit tricky:
Given this and in general complication of this PR with device check + TI device guard + ... We will fall back with a more incremental approach by first just adding more device checks. As mentioned in #56570 (comment) . cc @ezyang |
For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) [ghstack-poisoned]
wenleix
pushed a commit
that referenced
this pull request
Apr 23, 2021
Pull Request resolved: #55241 For kernels run on CUDA backend, the following preparation steps are required for most kernels: 1. Make sure all the tensors are on the same device (except for 0-dimension tensor, which is allowed to stay in CPU). We will denote this device as `common_device`. 2. Acquire the device guard on `common_device`. Today, these two steps are done separately. (1) is done in kernels in an adhoc way, while (2) is done in the kernel wrapper (e.g. `RegisterCUDA.cpp`). The kernel wrapper assumes (1) will be done in kernel, so it just pick up the first tensor to acquire device guard. There are at least two issues with the current approach: 1. Kernel may not implement the device check, or implement in an inconsistent way. 2. If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire the wrong device guard. We have seen issues before (db2b273) thus kernel might need to re-acquire device guard in such situation, which is fragile. # Appendix As a concrete example, here are a few implementations in codebase to do the device checks: a. `TensorIteratorBase::compute_types` in `ATen/TensorIterator.cpp`. Used in kernels rely on `TensorIterator` to perform element-wise operations: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorIterator.cpp#L326-L340 b. `checkSameGPU` and `checkAllSameGPU` in `Aten/TensorUtils.cpp`:. Used in a lot of CUDA kernels. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/TensorUtils.cpp#L130-L152 c. `check_attributes` in `ATen/native/RNN.h`. Used in a few RNN kernels: https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/RNN.h#L30-L50 d. `dot_check` in `ATen/native/cuda/LinearAlgebra.cu`. https://github.com/pytorch/pytorch/blob/6d030c14cf4bd2e59b6e4899a9ee4645d2199e68/aten/src/ATen/native/cuda/LinearAlgebra.cu#L378-L383 These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in `TensorList`. ghstack-source-id: 127311961 Differential Revision: [D27540071](https://our.internmc.facebook.com/intern/diff/D27540071/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27540071/)!
|
Command to generate change in |
|
Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
For kernels run on CUDA backend, the following preparation steps are required for most kernels:
Make sure all the tensors are on the same device (except for
0-dimension tensor, which is allowed to stay in CPU). We will denote
this device as
common_device.Acquire the device guard on
common_device.Today, these two steps are done separately. (1) is done in kernels
in an adhoc way, while (2) is done in the kernel wrapper (e.g.
RegisterCUDA.cpp).The kernel wrapper assumes (1) will be done in kernel, so it just pick up
the first tensor to acquire device guard. There are at least two issues with
the current approach:
Kernel may not implement the device check, or implement in an inconsistent way.
If the first tensor happened to be a 0-dimension CPU tensor, (2) will acquire
the wrong device guard. We have seen issues before
(db2b273) thus kernel might need to re-acquire
device guard in such situation, which is fragile.
Appendix
As a concrete example, here are a few implementations in codebase to do the device checks:
a.
TensorIteratorBase::compute_typesinATen/TensorIterator.cpp. Used in kernels rely onTensorIteratorto perform element-wise operations:pytorch/aten/src/ATen/TensorIterator.cpp
Lines 326 to 340 in 6d030c1
b.
checkSameGPUandcheckAllSameGPUinAten/TensorUtils.cpp:. Used in a lot of CUDA kernels.pytorch/aten/src/ATen/TensorUtils.cpp
Lines 130 to 152 in 6d030c1
c.
check_attributesinATen/native/RNN.h. Used in a few RNN kernels:pytorch/aten/src/ATen/native/RNN.h
Lines 30 to 50 in 6d030c1
d.
dot_checkinATen/native/cuda/LinearAlgebra.cu.pytorch/aten/src/ATen/native/cuda/LinearAlgebra.cu
Lines 378 to 383 in 6d030c1
These device checks have slightly different assumption. For example, (a) allows 0-dimension CPU tensors, (b) assumes everything is on GPU device, and (c) also checks all tensors in
TensorList.Differential Revision: D27540071