preserve non-dense or overlapping tensor's layout in *_like functions #46046
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
glaringlee
changed the title
💊 CI failures summary and remediationsAs of commit b844a54 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group. This comment has been revised 67 times. |
This is to solve the non-dense tensor layout problem in #45505 This change will cover all kinds of non-dense tensor. [ghstack-poisoned]
This is to solve the non-dense tensor layout problem in #45505 This change will cover all kinds of non-dense tensor. [ghstack-poisoned]
This is to solve the non-dense tensor layout problem in #45505 TODO: Add test cases Fix all the BC broken test cases This change will cover all kinds of non-dense tensors. [ghstack-poisoned]
added 2 commits
October 12, 2020 07:51
This is to solve the non-dense tensor layout problem in #45505 TODO: Fix all the BC broken test cases This change will cover all kinds of non-dense tensors. [ghstack-poisoned]
This is to solve the non-dense tensor layout problem in #45505 TODO: Fix all the BC broken test cases This change will cover all kinds of non-dense tensors. [ghstack-poisoned]
This is to solve the non-dense tensor layout problem in #45505 TODO: Fix all the BC broken test cases This change will cover all kinds of non-dense tensors. [ghstack-poisoned]
glaringlee
changed the title
…*_like functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The behavior changes are listed below: | code | old | new | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|------------------------------------------------------| | #strided tensors<br>a=torch.randn(2,3,8)[:,:,::2].permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (2, 24, 8) <br>(6, 3, 1) <br>(1, 12, 4) <br>(6, 3, 1) | (2, 24, 8)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | | #memory dense tensors<br>a=torch.randn(3,1,1).as_strided((3,1,1), (1,3,3))<br>print(a.stride(), (a+torch.randn(1)).stride())<br>a=torch.randn(2,3,4).permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (1, 3, 3) (1, 1, 1)<br>(1, 12, 4)<br>(6, 3, 1)<br>(1, 12, 4)<br>(6, 3, 1) | (1, 3, 3) (1, 3, 3)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [ ] Investigate if any fb internal tests are broken This change will cover all kinds of non-dense tensors. [ghstack-poisoned]
…*_like functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The behavior changes are listed below: | code | old | new | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|------------------------------------------------------| | #strided tensors<br>a=torch.randn(2,3,8)[:,:,::2].permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (2, 24, 8) <br>(6, 3, 1) <br>(1, 12, 4) <br>(6, 3, 1) | (2, 24, 8)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | | #memory dense tensors<br>a=torch.randn(3,1,1).as_strided((3,1,1), (1,3,3))<br>print(a.stride(), (a+torch.randn(1)).stride())<br>a=torch.randn(2,3,4).permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (1, 3, 3) (1, 1, 1)<br>(1, 12, 4)<br>(6, 3, 1)<br>(1, 12, 4)<br>(6, 3, 1) | (1, 3, 3) (1, 3, 3)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [ ] Investigate if any fb internal tests are broken This change will cover all kinds of non-dense tensors. Differential Revision: [D24288970](https://our.internmc.facebook.com/intern/diff/D24288970) [ghstack-poisoned]
…*_like functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The behavior changes are listed below: | code | old | new | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|------------------------------------------------------| | #strided tensors<br>a=torch.randn(2,3,8)[:,:,::2].permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (2, 24, 8) <br>(6, 3, 1) <br>(1, 12, 4) <br>(6, 3, 1) | (2, 24, 8)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | | #memory dense tensors<br>a=torch.randn(3,1,1).as_strided((3,1,1), (1,3,3))<br>print(a.stride(), (a+torch.randn(1)).stride())<br>a=torch.randn(2,3,4).permute(2,0,1)<br>print(a.stride())<br>print(a.exp().stride())<br>print((a+a).stride())<br>out = torch.empty(0)<br>torch.add(a,a,out=out)<br>print(out.stride()) | (1, 3, 3) (1, 1, 1)<br>(1, 12, 4)<br>(6, 3, 1)<br>(1, 12, 4)<br>(6, 3, 1) | (1, 3, 3) (1, 3, 3)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4)<br>(1, 12, 4) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [ ] Investigate if any fb internal tests are broken This change will cover all kinds of non-dense tensors. Differential Revision: [D24288970](https://our.internmc.facebook.com/intern/diff/D24288970) [ghstack-poisoned]
…*_like functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The test is based on empty_like. We compare the output tensor's sizes/strides among the empty_like old code, empty_like new code, and TensorIterator based operator exp(). For each sizes/strides we create tensor from both empty_like() and exp() using the following code and record the output tensor's sizes/stride. ``` a = torch.empty_strided([sizes], [strides]) b = torch.empty_like(a) c = torch.exp(a) ``` The output strides changes are listed below: |sizes/strides |old |new |unary op |description | |--------------------|---------------------|--------------------|--------------------|------------------------------------| |(4,3,2)/(10,1,2) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense but overlap | |(4,3,2)/(10,0,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |non-dense but overlap (0 strided) | |(4,3,2)/(12,2,6) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (sliced) | |(4,3,2)/(10,1,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (with gap) | |(4,1,1,2)/(10,0,0,2)|(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |non-dense non-overlap (0 strided) | |(4,1,2)/(10,3,3) |(4,1,2)/(2,2,1) |(4,1,2)/(2,1,1) |(4,1,2)/(2,1,1) |overlap (with equal strides) | |(4,2,2,2)/(2,0,0,6) |(4,2,2,2)/(8,4,2,1) |(4,2,2,2)/(1,8,4,16)|(4,2,2,2)/(1,8,4,16)|overlap (0 strided with equal size) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [ ] Investigate if any fb internal tests are broken This change will cover all kinds of non-dense tensors. Differential Revision: [D24288970](https://our.internmc.facebook.com/intern/diff/D24288970) [ghstack-poisoned]
glaringlee
changed the title
…e functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The test is based on empty_like. We compare the output tensor's sizes/strides among the empty_like old code, empty_like new code, and TensorIterator based operator exp(). For each sizes/strides we create tensor from both empty_like() and exp() using the following code and record the output tensor's sizes/stride. ``` a = torch.empty_strided([sizes], [strides]) b = torch.empty_like(a) c = torch.exp(a) // record the result of b (in both old and new code) and c to create the list below ``` The output strides changes are listed below: |sizes/strides |old |new |unary op |description | |--------------------|---------------------|--------------------|--------------------|------------------------------------| |(4,3,2)/(10,1,2) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense but overlap | |(4,3,2)/(10,0,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |non-dense but overlap (0 strided) | |(4,3,2)/(12,2,6) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (sliced) | |(4,3,2)/(10,1,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (with gap) | |(4,1,1,2)/(10,0,0,2)|(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |non-dense non-overlap (0 strided) | |(4,1,2)/(10,3,3) |(4,1,2)/(2,2,1) |(4,1,2)/(2,1,1) |(4,1,2)/(2,1,1) |overlap (with equal strides) | |(4,2,2,2)/(2,0,0,6) |(4,2,2,2)/(8,4,2,1) |(4,2,2,2)/(1,8,4,16)|(4,2,2,2)/(1,8,4,16)|overlap (0 strided with equal size) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [x] Investigate if any fb internal tests are broken (Update: None so far) This change will cover all kinds of non-dense tensors. Differential Revision: [D24288970](https://our.internmc.facebook.com/intern/diff/D24288970) [ghstack-poisoned]
ngimel
approved these changes
Oct 19, 2020
…e functions" *_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost. This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator. The test is based on empty_like. We compare the output tensor's sizes/strides among the empty_like old code, empty_like new code, and TensorIterator based operator exp(). For each sizes/strides we create tensor from both empty_like() and exp() using the following code and record the output tensor's sizes/stride. ``` a = torch.empty_strided([sizes], [strides]) b = torch.empty_like(a) c = torch.exp(a) // record the result of b (in both old and new code) and c to create the list below ``` The output strides changes are listed below: |sizes/strides |old |new |unary op |description | |--------------------|---------------------|--------------------|--------------------|------------------------------------| |(4,3,2)/(10,1,2) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense but overlap | |(4,3,2)/(10,0,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |(4,3,2)/(6,2,1) |non-dense but overlap (0 strided) | |(4,3,2)/(12,2,6) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (sliced) | |(4,3,2)/(10,1,3) |(4,3,2)/(6,2,1) |(4,3,2)/(6,1,3) |(4,3,2)/(6,1,3) |non-dense non-overlap (with gap) | |(4,1,1,2)/(10,0,0,2)|(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |(4,1,1,2)/(2,2,2,1) |non-dense non-overlap (0 strided) | |(4,1,2)/(10,3,3) |(4,1,2)/(2,2,1) |(4,1,2)/(2,1,1) |(4,1,2)/(2,1,1) |overlap (with equal strides) | |(4,2,2,2)/(2,0,0,6) |(4,2,2,2)/(8,4,2,1) |(4,2,2,2)/(1,8,4,16)|(4,2,2,2)/(1,8,4,16)|overlap (0 strided with equal size) | This is to solve the non-dense tensor layout problem in #45505 TODO: - [x] Fix all the BC broken test cases in pytorch - [x] Investigate if any fb internal tests are broken (Update: None so far) This change will cover all kinds of non-dense tensors. Differential Revision: [D24288970](https://our.internmc.facebook.com/intern/diff/D24288970) [ghstack-poisoned]
|
@glaringlee merged this pull request in a651b87. |
1 similar comment
|
@glaringlee merged this pull request in a651b87. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
*_like functions are used in pytorch to create a new tensor with the same shape of the input tensor. But we don’t always preserve the layout permutation of the tensor. Current behavior is that, for a dense and non-overlapping tensor, its layout permutation is preserved. For eg. passing a channel last contiguous tensor t with ‘shape/stride’ (2, 4, 3, 2)/(24, 1, 8, 4) to empty_like(t) function will create a new tensor with exactly the same ‘shape/stride’ as the input tensor t. However, if the input tensor is non-dense or has overlap, we simply create a contiguous tensor based on input tensor’s shape, so the tensor layout permutation is lost.
This PR preserves the layout permutation for non-dense or overlapping tensor. The strides propagation rule that used in this PR is exactly the same as what is being used in TensorIterator.
The test is based on empty_like. We compare the output tensor's sizes/strides among the empty_like old code, empty_like new code, and TensorIterator based operator exp(). For each sizes/strides we create tensor from both empty_like() and exp() using the following code and record the output tensor's sizes/stride.
The output strides changes are listed below:
This is to solve the non-dense tensor layout problem in #45505
TODO:
This change will cover all kinds of non-dense tensors.
Differential Revision: D24288970