Skip to content

[do not review]debug gpu ci with new gpu type #5517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

[do not review]debug gpu ci with new gpu type #5517

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a0158ab
use manfei's change
vanbasten23 Aug 29, 2023
6513122
use a5g gpu
vanbasten23 Aug 29, 2023
55ceff7
add the change in the xrt branch
vanbasten23 Aug 29, 2023
a4da921
comment out test_dynamo.py as if fails
vanbasten23 Aug 29, 2023
087a7bc
print the value
vanbasten23 Aug 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .circleci/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@ function run_torch_xla_tests() {
# PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --use_nested_fsdp --use_small_fake_sample --num_epochs=1
# PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --auto_wrap_policy type_based --use_small_fake_sample --num_epochs=1
# XLA_DISABLE_FUNCTIONALIZATION=1 PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --use_nested_fsdp --use_small_fake_sample --num_epochs=1
PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --use_nested_fsdp --use_small_fake_sample --num_epochs=1
PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --auto_wrap_policy type_based --use_small_fake_sample --num_epochs=1
XLA_DISABLE_FUNCTIONALIZATION=1 PJRT_DEVICE=GPU python test/test_train_mp_imagenet_fsdp.py --fake_data --use_nested_fsdp --use_small_fake_sample --num_epochs=1
# Syncfree SGD optimizer tests
if [ -d ./torch_xla/amp/syncfree ]; then
echo "Running Syncfree Optimizer Test"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
needs: build
with:
docker-image: ${{ needs.build.outputs.docker-image }}
runner: linux.8xlarge.nvidia.gpu
runner: linux.g5.12xlarge.nvidia.gpu
timeout-minutes: 240
disable-xrt: 1
secrets:
Expand All @@ -69,7 +69,7 @@ jobs:
needs: build
with:
docker-image: ${{ needs.build.outputs.docker-image }}
runner: linux.8xlarge.nvidia.gpu
runner: linux.g5.12xlarge.nvidia.gpu
timeout-minutes: 210
collect-coverage: true
disable-xrt: 1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test_xrt.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
needs: build
with:
docker-image: ${{ needs.build.outputs.docker-image }}
runner: linux.8xlarge.nvidia.gpu
runner: linux.g5.12xlarge.nvidia.gpu
timeout-minutes: 180
disable-xrt: 0
secrets:
Expand Down
1 change: 1 addition & 0 deletions WORKSPACE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ http_archive(
"//openxla_patches:f16_abi_clang.diff",
"//openxla_patches:gpu_race_condition.diff",
"//openxla_patches:constexpr_return.diff",
"//openxla_patches:service.diff",
],
strip_prefix = "xla-cd2cf5c34931e4fc1cacf83bfc480a5b93f05f6d",
urls = [
Expand Down
41,741 changes: 41,741 additions & 0 deletions infra/ansible/out.txt
View file
Open in desktop

Large diffs are not rendered by default.

14 changes: 14 additions & 0 deletions openxla_patches/service.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
delete this patch after openxla pin updated to date after or equal to 08/09
diff --git a/xla/service/layout_assignment.cc b/xla/service/layout_assignment.cc
index f69224f42..43feda2a0 100644
--- a/xla/service/layout_assignment.cc
+++ b/xla/service/layout_assignment.cc
@@ -154,7 +154,7 @@ OperandLayoutConstraint::OperandLayoutConstraint(
instruction_(instruction),
operand_no_(operand_no) {
CHECK(shape_layout.LayoutIsSet());
- CHECK(ShapeUtil::CompatibleIgnoringElementType(
+ CHECK(ShapeUtil::CompatibleKind(
shape_layout.shape(), instruction->operand(operand_no)->shape()))
<< shape_layout.shape() << " is not compatible with "
<< instruction->operand(operand_no)->shape() << " (for operand "
1 change: 0 additions & 1 deletion test/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,6 @@ function run_xla_op_tests {
run_test "$CDIR/test_metrics.py"
run_test "$CDIR/test_zero1.py"
run_test "$CDIR/dynamo/test_dynamo_integrations_util.py"
run_test "$CDIR/dynamo/test_dynamo.py"
run_test "$CDIR/dynamo/test_bridge.py"
run_test "$CDIR/dynamo/test_num_output.py"
run_save_tensor_ir "$CDIR/dynamo/test_dynamo_graph_dump.py"
Expand Down
4 changes: 4 additions & 0 deletions test/stablehlo/test_stablehlo_compile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ def test_resnet18_stablehlo_compile(self):
xla_resnet18.eval()
xla_input = torch_input.to(device)
xla_output = xla_resnet18(xla_input)
print('xw32 cpu_output=', cpu_output)
print('xw32 xla_output.cpu()=', xla_output.cpu())
self.assertTrue(
torch.allclose(cpu_output, xla_output.cpu(), rtol=1e-03, atol=1e-03))
self.assertTrue(
torch.allclose(cpu_output, xla_output.cpu(), rtol=1e-05, atol=1e-05))
self.assertEqual(met.counter_value('StableHloCompile'), 1)
Expand Down