Some CI jobs are failing on Apple

[shoumikhin]

🐛 Describe the bug

See https://hud.pytorch.org/hud/pytorch/executorch/main
The failures started yesterday afternoon

By looking at the logs I can say that those previous logs that succeeded mentioned macos-14-arm64 runners, and the failed ones are using macos-15-arm64.

There's a common error fetching a prebuilt torch wheel:

Example of a failing job:
https://github.com/pytorch/executorch/actions/runs/17279168142/job/49043504059

Versions

Current CI

[atalman]

Looks like issue is related to torchaudio:

MD -MT src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o -MF src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o.d -o src/libtorchaudio/sox/CMakeFiles/libtorchaudio_sox.dir/types.cpp.o -c /private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/src/libtorchaudio/sox/types.cpp
      ninja: build stopped: subcommand failed.
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/setup.py", line 144, in <module>
          _main()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/setup.py", line 99, in _main
          setup(
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/tools/setup_helpers/extension.py", line 99, in run
          super().run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-x_jy67c1/tools/setup_helpers/extension.py", line 187, in build_extension
          subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchaudio
  Running setup.py clean for torchaudio
Failed to build torchaudio
ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchaudio)
ERROR conda.cli.main_run:execute(125): `conda run .ci/scripts/setup-macos.sh --build-tool cmake` failed. (See above for error)

[atalman]

Another error I see:

/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/src/libtorio/ffmpeg/pybind/pybind.cpp
      ninja: build stopped: subcommand failed.
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/setup.py", line 144, in <module>
          _main()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/setup.py", line 99, in _main
          setup(
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/tools/setup_helpers/extension.py", line 99, in run
          super().run()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/x7/ch5v91h56_zbvbd1y2f600dm0000gn/T/pip-req-build-485m887y/tools/setup_helpers/extension.py", line 187, in build_extension
          subprocess.check_call(["cmake", "--build", "."] + build_args, cwd=self.build_temp)
        File "/Users/runner/work/_temp/conda_environment_17279168142/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchaudio
  Running setup.py clean for torchaudio
Failed to build torchaudio
ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchaudio)
ERROR conda.cli.main_run:execute(125): `conda run .ci/scripts/setup-macos.sh --build-tool cmake` failed. (See above for error)
Traceback (most recent call last):
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 102, in <module>
    main()
    ~~~~^^
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 61, in main
    run_cmd_or_die(f"bash { os.environ.get('RUNNER_TEMP', '') }/exec_script")
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/executorch/executorch/test-infra/.github/scripts/run_with_env_secrets.py", line 39, in run_cmd_or_die
    raise RuntimeError(f"Command {cmd} failed with exit code {exit_code}")
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1