bitwise_and: Port to structured

[Freey0]

Stack from ghstack:

• OpInfo: bitwise_and #61349 OpInfo: bitwise_and
• bitwise_and: Port to structured #60813 bitwise_and: Port to structured
• bitwise_xor: Port to structured #60812 bitwise_xor: Port to structured
• bitwise_or: Port to strucutred #60811 bitwise_or: Port to strucutred
• logit_backward: Port to structured #60817 logit_backward: Port to structured
• tanh_backward: Port to structured #60816 tanh_backward: Port to structured
• sigmoid_backward: Port to structured #60815 sigmoid_backward: Port to structured
• xlogy: Port to structured #60814 xlogy: Port to structured
• fmin has been ported to the structured kernel, removing the old implementation #60810 fmin has been ported to the structured kernel, removing the old implementation
• fmod: Port to structured #60809 fmod: Port to structured

Differential Revision: D29449374

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 8817a0e (more details on the Dr. CI page and at hud.pytorch.org/pr/60813):

---

• 2/2 failures introduced in this PR

---

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_cuda11_1_cudnn8_py3_gcc7_build (1/2)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

Jul 15 13:21:46 rm: cannot remove '/var/lib/jenkins/sccache_error.log': No such file or directory

Jul 15 13:21:46 ++++ extract_trap_cmd
Jul 15 13:21:46 ++++ printf '%s\n' ''
Jul 15 13:21:46 +++ printf '%s\n' cleanup
Jul 15 13:21:46 ++ trap -- '
Jul 15 13:21:46 cleanup' EXIT
Jul 15 13:21:46 ++ [[ pytorch-linux-xenial-cuda11.1-cudnn8-py3-gcc7-build != *pytorch-win-* ]]
Jul 15 13:21:46 ++ which sccache
Jul 15 13:21:46 ++ sccache --stop-server
Jul 15 13:21:46 ++ true
Jul 15 13:21:46 ++ rm /var/lib/jenkins/sccache_error.log
Jul 15 13:21:46 rm: cannot remove '/var/lib/jenkins/sccache_error.log': No such file or directory
Jul 15 13:21:46 ++ true
Jul 15 13:21:46 ++ [[ -n '' ]]
Jul 15 13:21:46 ++ [[ pytorch-linux-xenial-cuda11.1-cudnn8-py3-gcc7-build == *rocm* ]]
Jul 15 13:21:46 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
Jul 15 13:21:46 ++ SCCACHE_IDLE_TIMEOUT=1200
Jul 15 13:21:46 ++ RUST_LOG=sccache::server=error
Jul 15 13:21:46 ++ sccache --start-server
Jul 15 13:21:46 sccache: Starting the server...
Jul 15 13:21:46 ++ sccache --zero-stats
Jul 15 13:21:46 Compile requests                      0

pytorch_xla_linux_bionic_py3_6_clang9_test (2/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jul 15 16:35:54 AssertionError: RuntimeError not raised by

Jul 15 16:35:54 test_random_from_to_bool (__main__.TestRandom) ... ok
Jul 15 16:35:54 test_get_xla_tensor (__main__.TestSelect) ... ok
Jul 15 16:35:54 test (__main__.TestToXlaTensorArena) ... ok
Jul 15 16:35:54 
Jul 15 16:35:54 ======================================================================
Jul 15 16:35:54 FAIL: test_pred_type (__main__.TestAtenXlaTensor)
Jul 15 16:35:54 ----------------------------------------------------------------------
Jul 15 16:35:54 Traceback (most recent call last):
Jul 15 16:35:54   File "/var/lib/jenkins/workspace/xla/test/test_operations.py", line 1208, in test_pred_type
Jul 15 16:35:54     self.assertRaises(RuntimeError, lambda: c & c.byte())
Jul 15 16:35:54 AssertionError: RuntimeError not raised by <lambda>
Jul 15 16:35:54 
Jul 15 16:35:54 ----------------------------------------------------------------------
Jul 15 16:35:54 Ran 124 tests in 59.011s
Jul 15 16:35:54 
Jul 15 16:35:54 FAILED (failures=1)
Jul 15 16:35:54 + cleanup
Jul 15 16:35:54 + retcode=1
Jul 15 16:35:54 + set +x
Jul 15 16:35:54 =================== sccache compilation log ===================
Jul 15 16:35:54 =========== If your build fails, please take a look at the log above for possible reasons ===========

---

1 job timed out:

• pytorch_linux_xenial_cuda11_1_cudnn8_py3_gcc7_build

---

Preview docs built from this PR

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[ezyang]

@ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[ezyang]

This one gonna need XLA update

[JackCaoG]

@ezyang This pr seems to remove the constrain of

    # PRED cannot be automatically promoted to other dtypes in bitwise ops.

for bitwise and, which xla happens to have a test on in https://github.com/pytorch/xla/blob/master/test/test_operations.py#L1208. Is this intentional? Should pt/xla follow the same guidance and remove the constrain of this kind of type promotion?

[ezyang]

No, I'm guessing this is unintentional, and not exercised in any of PyTorch's native tests. Thanks for identifying the issue.

[ezyang]

Oof but I don't understand how this PR could cause the behavior change :(

[ezyang]

OK actually this is pretty weird.

>>> c = torch.zeros(2) >= 1
>>> c.byte() | c
tensor([0, 0], dtype=torch.uint8)
>>> c | c.byte()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: result type Byte can't be cast to the desired output type Bool

This seems wrong. The reason this happens is because of how the output used to be allocated in the functional case:

Tensor bitwise_and(const Tensor& self, const Tensor& other) {
  Tensor result = at::empty({0}, self.options());
  at::bitwise_and_out(result, self, other);
  return result;
}

aka bitwise_and always takes on the type of its first argument, but then casting is not disabled.

numpy is willing to promote both ways

>>> c | c.astype('byte')
array([[1, 1]], dtype=int8)
>>> c.astype('byte') | c
array([[1, 1]], dtype=int8)
>>> c

so actually seems like the new behavior is correct. @Freey0 as a follow up we should add tests for this bug.

[Freey0]

Ok, I will add a test for this bug in the new PR

[ezyang]

@JackCaoG what do you think of this analysis?

[JackCaoG]

@ezyang I think it is fair. Original comment on xla test(by Ailing) also mentioned that this behavior is not aligned with numpy and might change in the future. I guess now is the future 😄 . I will update xla to remove this constrain.

[JackCaoG]

@ezyang I think I fixed the XLAOP casting part to allow this type promotion but I had an check that I borrowed from pytorch that failed

void CheckBinaryOpTypePromotion(const at::Tensor& out, const at::Tensor& self,
                                const at::Tensor& other) {
  at::ScalarType resultType = at::result_type(self, other);
  XLA_CHECK(at::canCast(/*from=*/resultType, /*to=*/out.scalar_type()));
}

when I do xla_c & xla_c.byte()

I think this is part of the functional case you mentioned because out.scalar_type() changed as I swap between xla_c.byte() & xla_c (result type: Byte out type Byte) and xla_c & xla_c.byte()(result type: Byte out type Bool).

How does structured binary op performs type checking for input and output? Does pt/xla need to care about the type checking in this case(not sure if it is always done by pytorch structuredop )?

[ezyang]

Does pt/xla need to care about the type checking in this case(not sure if it is always done by pytorch structuredop )?

For now, still yes, because you are writing regular XLA kernels and not structured XLA kernels (that was along the lines of what @bdhirsh was working on)

How does structured binary op performs type checking for input and output?

It seems like the meta function seems to imply it's normal type promotion:

TORCH_META_FUNC2(bitwise_and, Tensor) (const Tensor& self, const Tensor& other) {
  build_borrowing_binary_op(maybe_get_output(), self, other);
}

If you implement this the same way, e.g., add, is implemented, does that give you the correct behavior?

[ezyang]

@ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[ezyang]

@ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[ezyang]

@ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@ezyang merged this pull request in a0c9d70.