S390x update docker image #153619

AlekseiNikiforovIBM · 2025-05-15T13:32:12Z

Add ninja-build for pytorch tests.
Switch to gcc 14 due to fix for precompiled headers and s390x vectorization interaction.
Disable -Werror when building onnxruntime.
Pin onnx version.

It is required for multiple tests like TestUtilityFuns_opset_9::test_custom_op_fallthrough from onnx/test_utility_funs.py.

There is a bugfix for precompiled headers and s390x vectorization interaction in gcc >= 14. This switch fixes all tests in test/inductor/test_cpu_cpp_wrapper.py on s390x.

New gcc leads to new warnings. Don't turn those warnings into errors.

pytorch-bot · 2025-05-15T13:32:16Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153619

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

⏳ 1 Pending, 1 Unrelated Failure

As of commit 2fdae5d with merge base cd119dd ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / unstable-linux-focal-cuda12.6-py3.10-gcc11-sm89-xfail / build (gh)
ninja: build stopped: subcommand failed

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cyyever · 2025-05-15T14:40:06Z

What failures for ONNX 1.18?

AlekseiNikiforovIBM · 2025-05-16T07:26:41Z

I did not investigate build failure yet, but here's example of log of build failure:
https://github.com/pytorch/pytorch/actions/runs/15060014173/job/42333617598

Traceback (most recent call last):
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 435, in build_wheel
          return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 423, in _build
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 318, in <module>
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 115, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 1102, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "<string>", line 257, in run
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 96, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-gm6mhyl8/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "<string>", line 293, in build_extensions
      AssertionError: Bug: No generated python stubs found
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
ERROR: Could not build wheels for onnx, which is required to install pyproject.toml-based projects
Failed to build onnx

Newer onnx version doesn't build with old protobuf from manylinux. Drop system protobuf and let python pull protobuf from pypi when it's needed.

huydhn · 2025-05-22T09:32:40Z

@pytorchbot merge -f 'Docker build job has passed, no need to run trunk job'

pytorchmergebot · 2025-05-22T09:34:33Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

AlekseiNikiforovIBM added 3 commits May 15, 2025 15:29

s390x: add ninja-build for tests

cdfe4b8

It is required for multiple tests like TestUtilityFuns_opset_9::test_custom_op_fallthrough from onnx/test_utility_funs.py.

Switch manylinuxs390x-builder to gcc 14

61de77a

There is a bugfix for precompiled headers and s390x vectorization interaction in gcc >= 14. This switch fixes all tests in test/inductor/test_cpu_cpp_wrapper.py on s390x.

s390x: disable -Werror when building onnxruntime

c4e195b

New gcc leads to new warnings. Don't turn those warnings into errors.

AlekseiNikiforovIBM requested a review from jeffdaily as a code owner May 15, 2025 13:32

pytorch-bot bot added the topic: not user facing topic category label May 15, 2025

pytorchbot added the open source label May 15, 2025

jerryzh168 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 16, 2025

AlekseiNikiforovIBM requested review from huydhn, malfet and seemethere May 19, 2025 09:12

AlekseiNikiforovIBM mentioned this pull request May 20, 2025

S390x: update more tests #150116

Closed

Fix building s390x docker image

2fdae5d

Newer onnx version doesn't build with old protobuf from manylinux. Drop system protobuf and let python pull protobuf from pypi when it's needed.

AlekseiNikiforovIBM force-pushed the s390x_update_docker_image branch from 0010f39 to 2fdae5d Compare May 21, 2025 14:49

AlekseiNikiforovIBM mentioned this pull request May 22, 2025

UNSTABLE Build manywheel docker images for s390x / build-docker-cpu-s390x #154074

Open

huydhn approved these changes May 22, 2025

View reviewed changes

pytorchmergebot added the merging label May 22, 2025

pytorchmergebot closed this in 2f57ee5 May 22, 2025

pytorchmergebot added Merged and removed merging labels May 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

S390x update docker image #153619

S390x update docker image #153619

Uh oh!

AlekseiNikiforovIBM commented May 15, 2025

Uh oh!

pytorch-bot bot commented May 15, 2025 •

edited

Loading

Uh oh!

cyyever commented May 15, 2025

Uh oh!

AlekseiNikiforovIBM commented May 16, 2025

Uh oh!

huydhn commented May 22, 2025

Uh oh!

pytorchmergebot commented May 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

S390x update docker image #153619

S390x update docker image #153619

Uh oh!

Conversation

AlekseiNikiforovIBM commented May 15, 2025

Uh oh!

pytorch-bot bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153619

⏳ 1 Pending, 1 Unrelated Failure

Uh oh!

cyyever commented May 15, 2025

Uh oh!

AlekseiNikiforovIBM commented May 16, 2025

Uh oh!

huydhn commented May 22, 2025

Uh oh!

pytorchmergebot commented May 22, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pytorch-bot bot commented May 15, 2025 •

edited

Loading