Skip to content

Conversation

@will-cromar
Copy link
Collaborator

These 4 tests fail with error messages like these on both XRT and PJRT:

test_baddmm_integer_types (__main__.TestAtenXlaTensor) ... 2023-02-14 18:19:56.358333: E tensorflow/core/tpu/kernels/tpu_compilation_cache_external.cc:113] While rewriting computation to not contain X64 element types, XLA encountered an HLO for which this rewriting is not implemented: %dot.6 = s64[10,3,5]{2,1,0} dot(s64[10,3,4]{0,2,1} %p1.4, s64[10,4,5]{0,2,1} %p0.3), lhs_batch_dims={0}, lhs_contracting_dims={2}, rhs_batch_dims={0}, rhs_contracting_dims={1}, operand_precision={highest,highest}
2023-02-14 18:19:56.358430: F tensorflow/core/tpu/kernels/tpu_program_group.cc:86] Check failed: xla_tpu_programs.size() > 0 (0 vs. 0)

Confirmed with these commands

XRT_TPU_CONFIG="localservice;0;localhost:51011" TPU_NUM_DEVICES=4 python pytorch/xla/test/test_operations.py -v
PJRT_DEVICE=TPU_LEGACY python pytorch/xla/test/test_operations.py -v
PJRT_DEVICE=TPU python pytorch/xla/test/test_operations.py -v

@will-cromar will-cromar requested a review from JackCaoG February 14, 2023 19:11
@will-cromar
Copy link
Collaborator Author

Adding to TPU CI and triggering build

@will-cromar will-cromar force-pushed the wcromar/disable-py-operations-tpu branch from f310e83 to 9d1b351 Compare February 14, 2023 20:23
@will-cromar will-cromar merged commit 83f425a into master Feb 14, 2023
JackCaoG added a commit that referenced this pull request Feb 16, 2023
* Fix HLO dumping (#4619)

* Update TF pin to 2/13 (#4615)

* Update TF pin to 2/13

* Fix pinned commit

* Add patch to revert TF 3e24055

* Add comment to new patch

* Fix patch command in TPU CI (#4623)

* Skip execution for extract_compiled_graph (#4612)

* Only warm up cache for dynamo extract_graph step

* Add missing config

* Make sure warm up run does not cause place holder to be created

* Fix tests

* Disable failing `test_operations.py` tests on TPU (#4622)

* Disable `test_operations.py` tests failing on TPU

* Add to TPU CI

* Bazel (#4528)

* Replace tensorflow with a bazel external repository

* Basic migration to bazel for xla_client.

* Revert to blob

* Add vscode config.

* Update newlines

* Merge with pjrt client test build changes.

* Migrate tests to new build

* Format test and plugin

* Order imports

* Conditionally apply tf patches; apply pt patches always.

* Format python

* configure formatters

* Mirror TF pin update an fixes in bazel.

* Support local and sandboxed build based on flags

* Add cloud cache URLs for llvm.

* Merge with upstream

* Update TF pin

* Fix patching regression

* Revert "Bazel (#4528)" (#4631)

This reverts commit 3a90f5a.

---------

Co-authored-by: JackCaoG <59073027+JackCaoG@users.noreply.github.com>
Co-authored-by: Will Cromar <wcromar@google.com>
Co-authored-by: stgpetrovic <stgpetrovic@gmail.com>
JackCaoG pushed a commit that referenced this pull request Feb 16, 2023
* Disable `test_operations.py` tests failing on TPU

* Add to TPU CI
chandrasekhard2 pushed a commit that referenced this pull request Feb 22, 2023
* Disable `test_operations.py` tests failing on TPU

* Add to TPU CI
chandrasekhard2 pushed a commit that referenced this pull request Feb 22, 2023
* Disable `test_operations.py` tests failing on TPU

* Add to TPU CI
mateuszlewko pushed a commit that referenced this pull request Mar 15, 2023
* Disable `test_operations.py` tests failing on TPU

* Add to TPU CI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants