[REQUEST] Use python sysconfig to generate CFLAGs

[williamtwomey]

Is your feature request related to a problem? Please describe.

In conda/venv, DeepSpeed fails to build pt_binding.o as it thinks Python.h is missing. Looking at the c++ command it's trying to use, it's not including the correct paths (/usr/include/python3.11, ~/venv/python3.11/include, etc.)

Describe the solution you'd like

Using sysconfig we can generate this, similar to what python-config.py does. Something like

        flags = ['-I' + sysconfig.get_path('include'),
                 '-I' + sysconfig.get_path('platinclude')]

Describe alternatives you've considered
Looking at builder.py we should be able to override this with the CFLAGS, but this isn't always an option (and is an extra hoop we don't need if we let python tell us what paths to include)

Additional context
N/A

[williamtwomey]

FWIW, this appears to be recently introduced - 0.13.1 does not have this issue, but 0.14.1+ do. (This on Debian Bookworm, using python 3.11.2 as well as conda 3.11.9)

python3.11-dev is installed

[1/11] c++ -MMD -MF pt_binding.o.d -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/TH -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DBF16_AVAILABLE -c /home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -o pt_binding.o
FAILED: pt_binding.o
c++ -MMD -MF pt_binding.o.d -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/TH -isystem /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -DBF16_AVAILABLE -c /home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -o pt_binding.o
In file included from /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/csrc/Device.h:4,
                 from /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
                 from /home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/extension.h:9,
                 from /home/user/python-3.11-venv/lib/python3.11/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp:7:
/home/user/python-3.11-venv/lib/python3.11/site-packages/torch/include/torch/csrc/python_headers.h:12:10: fatal error: Python.h: No such file or directory
   12 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.