cpp_extension doesn't support include paths with spaces

[kmatzen]

🐛 Describe the bug

It seems like cpp_extension has a bug where it doesn't properly escape paths on the command line if they are used for compiler arguments. This is an issue if the path has a space in it.

Here's one example:

pytorch/torch/utils/cpp_extension.py

Lines 2190 to 2191 in 5e0440e

	common_cflags += [f'-I{include}' for include in user_includes]
	common_cflags += [f'-isystem {include}' for include in system_includes]

These paths are unescaped.

import torch.utils.cpp_extension

torch.utils.cpp_extension.load(
    name="foo",
    sources=["foo.cpp"],
    extra_include_paths=["foo/this is/a path/with spaces/"],
    verbose=True,
)

Using /Users/matzen/Library/Caches/torch_extensions/py310_cpu as PyTorch extensions root...
Emitting ninja build file /Users/matzen/Library/Caches/torch_extensions/py310_cpu/foo/build.ninja...
Building extension module foo...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF foo.o.d -DTORCH_EXTENSION_NAME=foo -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_clang\" -DPYBIND11_STDLIB=\"_libcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1002\" -I/Users/matzen/foo/this is/a path/with spaces -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/TH -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/THC -isystem /Users/matzen/mambaforge/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /Users/matzen/foo.cpp -o foo.o
FAILED: foo.o
c++ -MMD -MF foo.o.d -DTORCH_EXTENSION_NAME=foo -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_clang\" -DPYBIND11_STDLIB=\"_libcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1002\" -I/Users/matzen/foo/this is/a path/with spaces -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/TH -isystem /Users/matzen/mambaforge/lib/python3.10/site-packages/torch/include/THC -isystem /Users/matzen/mambaforge/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /Users/matzen/foo.cpp -o foo.o
clang: error: no such file or directory: 'is/a'
clang: error: no such file or directory: 'path/with'
clang: error: no such file or directory: 'spaces'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/Users/matzen/mambaforge/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1884, in _run_ninja_build
    subprocess.run(
  File "/Users/matzen/mambaforge/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/matzen/xyz.py", line 4, in <module>
    torch.utils.cpp_extension.load(
  File "/Users/matzen/mambaforge/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1268, in load
    return _jit_compile(
  File "/Users/matzen/mambaforge/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1492, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/Users/matzen/mambaforge/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1607, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/Users/matzen/mambaforge/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'foo'

Versions

Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.0 (x86_64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.1.0.2.5)
CMake version: version 3.27.4
Libc version: N/A

Python version: 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:27:35) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.0-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] mypy==1.1.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.1
[pip3] pytorch-lightning==1.5.0
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchmetrics==0.11.1
[pip3] torchvision==0.14.1a0
[conda] mkl                       2022.2.1         h44ed08c_16952    conda-forge
[conda] numpy                     1.26.1                   pypi_0    pypi
[conda] pytorch                   1.13.1          cpu_py310h2bbf33f_1    conda-forge
[conda] pytorch-lightning         1.5.0                    pypi_0    pypi
[conda] torchaudio                0.13.1                py310_cpu    pytorch
[conda] torchmetrics              0.11.1             pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.14.1          cpu_py310hd5ee960_0    conda-forge

cc @malfet @zou3519

[malfet]

I remember fixing it in the past, but only for Windows. Here is the fix for this particular problem, but I agree it needs a more comprehensive redesign

diff --git a/torch/utils/cpp_extension.py b/torch/utils/cpp_extension.py
index d3d264d6172..caff8827de5 100644
--- a/torch/utils/cpp_extension.py
+++ b/torch/utils/cpp_extension.py
@@ -2187,8 +2187,8 @@ def _write_ninja_file_to_build_library(path,
 
     common_cflags += [f"{x}" for x in _get_pybind11_abi_build_flags()]
 
-    common_cflags += [f'-I{include}' for include in user_includes]
-    common_cflags += [f'-isystem {include}' for include in system_includes]
+    common_cflags += [f'-I{shlex.quote(include)}' for include in user_includes]
+    common_cflags += [f'-isystem {shlex.quote(include)}' for include in system_includes]
 
     common_cflags += [f"{x}" for x in _get_glibcxx_abi_build_flags()]