Reland "[MLIR][Python] add Python wheel build demo/test" (#160481) #160488

makslevental · 2025-09-24T10:16:35Z

Reland standalone wheel build. The fix is to gate the test behind BUILD_SHARED_LIBS=OFF (because bundling all libs in the wheel requires valid rpaths which is not the case under BUILD_SHARED_LIBS=ON).

llvmbot · 2025-09-24T10:17:09Z

@llvm/pr-subscribers-mlir

Author: Maksim Levental (makslevental)

Changes

Reland standalone wheel build. The fix is to gate the test behind BUILD_SHARED_LIBS=OFF (because bundling all libs in the wheel requires valid rpaths which is not the case under BUILD_SHARED_LIBS=ON).

Full diff: https://github.com/llvm/llvm-project/pull/160488.diff

6 Files Affected:

(modified) mlir/examples/standalone/CMakeLists.txt (+6-2)
(added) mlir/examples/standalone/pyproject.toml (+65)
(modified) mlir/test/CMakeLists.txt (+1)
(modified) mlir/test/Examples/standalone/lit.local.cfg (+6)
(added) mlir/test/Examples/standalone/test.wheel.toy (+32)
(modified) mlir/test/lit.site.cfg.py.in (+3)

diff --git a/mlir/examples/standalone/CMakeLists.txt b/mlir/examples/standalone/CMakeLists.txt
index e2bcda7fa6f0b..c6c49fde12d2e 100644
--- a/mlir/examples/standalone/CMakeLists.txt
+++ b/mlir/examples/standalone/CMakeLists.txt
@@ -63,8 +63,12 @@ if(MLIR_ENABLE_BINDINGS_PYTHON)
   include(MLIRDetectPythonEnv)
   mlir_configure_python_dev_packages()
   # Note: for EXTERNAL_PROJECT_BUILD this must be set from the command line.
-  set(MLIR_PYTHON_PACKAGE_PREFIX "mlir_standalone" CACHE STRING "" FORCE)
-  set(MLIR_BINDINGS_PYTHON_INSTALL_PREFIX "python_packages/standalone/${MLIR_PYTHON_PACKAGE_PREFIX}" CACHE STRING "" FORCE)
+  if(NOT MLIR_PYTHON_PACKAGE_PREFIX)
+    set(MLIR_PYTHON_PACKAGE_PREFIX "mlir_standalone" CACHE STRING "" FORCE)
+  endif()
+  if(NOT MLIR_BINDINGS_PYTHON_INSTALL_PREFIX)
+    set(MLIR_BINDINGS_PYTHON_INSTALL_PREFIX "python_packages/standalone/${MLIR_PYTHON_PACKAGE_PREFIX}" CACHE STRING "" FORCE)
+  endif()
   add_subdirectory(python)
 endif()
 add_subdirectory(test)
diff --git a/mlir/examples/standalone/pyproject.toml b/mlir/examples/standalone/pyproject.toml
new file mode 100644
index 0000000000000..5a1e6e86513c3
--- /dev/null
+++ b/mlir/examples/standalone/pyproject.toml
@@ -0,0 +1,65 @@
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# Copyright (c) 2025.
+
+[project]
+name = "standalone-python-bindings"
+dynamic = ["version"]
+requires-python = ">=3.8,<=3.14"
+dependencies = [
+    "numpy>=1.19.5, <=2.1.2",
+    "PyYAML>=5.4.0, <=6.0.1",
+    "ml_dtypes>=0.1.0, <=0.6.0; python_version<'3.13'",
+    "ml_dtypes>=0.5.0, <=0.6.0; python_version>='3.13'",
+]
+
+[project.urls]
+Homepage = "https://github.com/llvm/llvm-project"
+Discussions = "https://discourse.llvm.org/"
+"Issue Tracker" = "https://github.com/llvm/llvm-project/issues?q=is%3Aissue%20state%3Aopen%20label%3Amlir%3Apython%20"
+"Source Code" = "https://github.com/llvm/llvm-project/tree/main/mlir/python"
+
+[build-system]
+requires = [
+    "scikit-build-core>=0.10.7",
+    "typing_extensions>=4.12.2",
+    "nanobind>=2.9, <3.0",
+    "pybind11>=2.10.0, <=2.13.6",
+]
+build-backend = "scikit_build_core.build"
+
+[tool.scikit-build]
+# This is the minimum version of scikit-build-core.
+minimum-version = "0.10.7"
+# This pyproject.toml must be adjacent to the root CMakeLists.txt (wherever project(...) is specified).
+cmake.source-dir = "."
+# This is for installing/distributing the python bindings target and only the python bindings target.
+build.targets = ["StandalonePythonModules"]
+install.components = ["StandalonePythonModules"]
+
+[tool.scikit-build.cmake.define]
+# Optional
+CMAKE_C_COMPILER = { env = "CMAKE_C_COMPILER", default = "" }
+CMAKE_CXX_COMPILER = { env = "CMAKE_CXX_COMPILER", default = "" }
+CMAKE_C_COMPILER_LAUNCHER = { env = "CMAKE_C_COMPILER_LAUNCHER", default = "" }
+CMAKE_CXX_COMPILER_LAUNCHER = { env = "CMAKE_CXX_COMPILER_LAUNCHER", default = "" }
+CMAKE_GENERATOR = { env = "CMAKE_GENERATOR", default = "Ninja" }
+LLVM_USE_LINKER = { env = "LLVM_USE_LINKER", default = "" }
+# Optional but highly recommended (this makes the bindings compatible with other bindings packages
+# by preventing symbol collisions).
+CMAKE_VISIBILITY_INLINES_HIDDEN = "ON"
+CMAKE_C_VISIBILITY_PRESET = "hidden"
+CMAKE_CXX_VISIBILITY_PRESET = "hidden"
+
+# Non-optional (alternatively you could use CMAKE_PREFIX_PATH here).
+MLIR_DIR = { env = "MLIR_DIR", default = "" }
+# Non-optional
+CMAKE_BUILD_TYPE = { env = "CMAKE_BUILD_TYPE", default = "Release" }
+MLIR_ENABLE_BINDINGS_PYTHON = "ON"
+# Effectively non-optional (any downstream project should specify this).
+MLIR_PYTHON_PACKAGE_PREFIX = "mlir_standalone"
+# This specifies the directory in the install directory (i.e., /tmp/pip-wheel/platlib) where _mlir_libs, dialects, etc.
+# are installed. Thus, this will be the package location (and the name of the package) that pip assumes is
+# the root package.
+MLIR_BINDINGS_PYTHON_INSTALL_PREFIX = "mlir_standalone"
diff --git a/mlir/test/CMakeLists.txt b/mlir/test/CMakeLists.txt
index 628adcfb6e285..98b7b71b660ec 100644
--- a/mlir/test/CMakeLists.txt
+++ b/mlir/test/CMakeLists.txt
@@ -84,6 +84,7 @@ llvm_canonicalize_cmake_booleans(
   MLIR_RUN_CUDA_SM80_TESTS
   MLIR_RUN_CUDA_SM80_LT_TESTS
   MLIR_RUN_CUDA_SM90_TESTS
+  BUILD_SHARED_LIBS
   )
 
 configure_lit_site_cfg(
diff --git a/mlir/test/Examples/standalone/lit.local.cfg b/mlir/test/Examples/standalone/lit.local.cfg
index 3b12dcbd99e83..22c4546b49363 100644
--- a/mlir/test/Examples/standalone/lit.local.cfg
+++ b/mlir/test/Examples/standalone/lit.local.cfg
@@ -7,7 +7,13 @@ config.substitutions.append(("%cmake_exe", config.host_cmake))
 config.substitutions.append(("%cmake_generator", config.host_cmake_generator))
 config.substitutions.append(("%host_cxx", config.host_cxx))
 config.substitutions.append(("%host_cc", config.host_cc))
+config.substitutions.append(("%hostc_compiler_launcher", config.host_c_compiler_launcher))
+config.substitutions.append(("%hostcxx_compiler_launcher", config.host_cxx_compiler_launcher))
 config.substitutions.append(("%enable_libcxx", config.enable_libcxx))
 config.substitutions.append(("%mlir_cmake_dir", config.mlir_cmake_dir))
+config.substitutions.append(("%mlir_obj_root", config.mlir_obj_root))
 config.substitutions.append(("%llvm_use_linker", config.llvm_use_linker))
 config.substitutions.append(("%cmake_build_type", config.cmake_build_type))
+
+if config.llvm_shared_libs_build == "0":
+    config.available_features.add("non-shared-libs-build")
diff --git a/mlir/test/Examples/standalone/test.wheel.toy b/mlir/test/Examples/standalone/test.wheel.toy
new file mode 100644
index 0000000000000..17d8cb5b246c9
--- /dev/null
+++ b/mlir/test/Examples/standalone/test.wheel.toy
@@ -0,0 +1,32 @@
+# There's no real issue with windows here, it's just that some CMake generated paths for targets end up being longer
+# than 255 chars when combined with the fact that pip wants to install into a tmp directory buried under
+# C/Users/ContainerAdministrator/AppData/Local/Temp.
+# UNSUPPORTED: target={{.*(windows).*}}
+# REQUIRES: non-shared-libs-build
+
+# RUN: export CMAKE_BUILD_TYPE=%cmake_build_type
+# RUN: export CMAKE_CXX_COMPILER=%host_cxx
+# RUN: export CMAKE_CXX_COMPILER_LAUNCHER=%hostcxx_compiler_launcher
+# RUN: export CMAKE_C_COMPILER=%host_cc
+# RUN: export CMAKE_C_COMPILER_LAUNCHER=%hostc_compiler_launcher
+# RUN: export CMAKE_GENERATOR=%cmake_generator
+# RUN: export LLVM_USE_LINKER=%llvm_use_linker
+# RUN: export MLIR_DIR="%mlir_cmake_dir"
+
+# RUN: %python -m pip wheel "%mlir_src_root/examples/standalone" -w "%mlir_obj_root/wheelhouse" -v | tee %t
+
+# RUN: rm -rf "%mlir_obj_root/standalone-python-bindings-install"
+# RUN: %python -m pip install standalone_python_bindings -f "%mlir_obj_root/wheelhouse" --target "%mlir_obj_root/standalone-python-bindings-install" -v | tee -a %t
+
+# RUN: export PYTHONPATH="%mlir_obj_root/standalone-python-bindings-install"
+# RUN: %python "%mlir_src_root/examples/standalone/test/python/smoketest.py" nanobind | tee -a %t
+
+# RUN: FileCheck --input-file=%t %s
+
+# CHECK: Successfully built standalone-python-bindings
+
+# CHECK: module {
+# CHECK:   %[[C2:.*]] = arith.constant 2 : i32
+# CHECK:   %[[V0:.*]] = standalone.foo %[[C2]] : i32
+# CHECK: }
+
diff --git a/mlir/test/lit.site.cfg.py.in b/mlir/test/lit.site.cfg.py.in
index 2fc595dfabbf5..c678c03b9bc49 100644
--- a/mlir/test/lit.site.cfg.py.in
+++ b/mlir/test/lit.site.cfg.py.in
@@ -8,6 +8,7 @@ config.llvm_tools_dir = lit_config.substitute("@LLVM_TOOLS_DIR@")
 config.lit_tools_dir = "@LLVM_LIT_TOOLS_DIR@"
 config.spirv_tools_tests = @LLVM_INCLUDE_SPIRV_TOOLS_TESTS@
 config.llvm_shlib_ext = "@SHLIBEXT@"
+config.llvm_shared_libs_build = "@BUILD_SHARED_LIBS@"
 config.llvm_shlib_dir = lit_config.substitute(path(r"@SHLIBDIR@"))
 config.python_executable = "@Python3_EXECUTABLE@"
 config.enable_assertions = @ENABLE_ASSERTIONS@
@@ -15,6 +16,8 @@ config.native_target = "@LLVM_NATIVE_ARCH@"
 config.host_os = "@HOST_OS@"
 config.host_cc = "@HOST_CC@"
 config.host_cxx = "@HOST_CXX@"
+config.host_c_compiler_launcher = "@CMAKE_C_COMPILER_LAUNCHER@"
+config.host_cxx_compiler_launcher = "@CMAKE_CXX_COMPILER_LAUNCHER@"
 config.enable_libcxx = "@LLVM_ENABLE_LIBCXX@"
 config.host_cmake = "@CMAKE_COMMAND@"
 config.host_cmake_generator = "@CMAKE_GENERATOR@"

makslevental · 2025-09-24T10:21:54Z

Here's the mlir-nvidia builder on this PR https://lab.llvm.org/buildbot/#/buildrequests/5649957?redirect_to_build=true

joker-eph · 2025-09-30T18:30:39Z

I have some errors somehow with the mlir/test/Examples/standalone/test.wheel.toy test right now, here is a sample of the output:

# |   -- Generating done (0.0s)
# |   CMake Warning:
# |     Manually-specified variables were not used by the project:
# | 
# |       MLIR_BINDINGS_PYTHON_INSTALL_PREFIX
# |       MLIR_PYTHON_PACKAGE_PREFIX
# | 
# | 
# |   -- Build files have been written to: /tmp/lit-tmp-4u27dlpw/tmpfnnlna_l/build
# |   *** Building project with Ninja...
# |   ninja: error: unknown target 'StandalonePythonModules'
# | 
# |   *** CMake build failed
# |   error: subprocess-exited-with-error
# |   
# |   × Building wheel for standalone-python-bindings (pyproject.toml) did not run successfully.
# |   │ exit code: 1
# |   ╰─> See above for output.
#

Does it ring a bell?

makslevental · 2025-09-30T19:35:58Z

I have some errors somehow with the mlir/test/Examples/standalone/test.wheel.toy test right now, here is a sample of the output:

# |   -- Generating done (0.0s)
# |   CMake Warning:
# |     Manually-specified variables were not used by the project:
# | 
# |       MLIR_BINDINGS_PYTHON_INSTALL_PREFIX
# |       MLIR_PYTHON_PACKAGE_PREFIX
# | 
# | 
# |   -- Build files have been written to: /tmp/lit-tmp-4u27dlpw/tmpfnnlna_l/build
# |   *** Building project with Ninja...
# |   ninja: error: unknown target 'StandalonePythonModules'
# | 
# |   *** CMake build failed
# |   error: subprocess-exited-with-error
# |   
# |   × Building wheel for standalone-python-bindings (pyproject.toml) did not run successfully.
# |   │ exit code: 1
# |   ╰─> See above for output.
#

Does it ring a bell?

I haven't seen this but if I get it it's because you don't have MLIR_ENABLE_BINDINGS_PYTHON=ON and because I must've forgotten to gate that test. Will double check.

makslevental · 2025-09-30T19:52:05Z

Fixed (i.e., gated) here #161427

llvm#160488) Reland standalone wheel build. The fix is to gate the test behind `BUILD_SHARED_LIBS=OFF` (because bundling all libs in the wheel requires valid rpaths which is not the case under `BUILD_SHARED_LIBS=ON`).

makslevental requested a review from jpienaar September 24, 2025 10:16

llvmbot added the mlir label Sep 24, 2025

Reland "[MLIR][Python] add Python wheel build demo/test" (#160481)

9c93028

makslevental force-pushed the users/makslevental/reland-standalone-wheel branch from 9f8ce40 to 9c93028 Compare September 24, 2025 10:43

makslevental merged commit 59e74a0 into main Sep 24, 2025
10 checks passed

makslevental deleted the users/makslevental/reland-standalone-wheel branch September 24, 2025 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reland "[MLIR][Python] add Python wheel build demo/test" (#160481) #160488

Reland "[MLIR][Python] add Python wheel build demo/test" (#160481) #160488

Uh oh!

makslevental commented Sep 24, 2025

Uh oh!

llvmbot commented Sep 24, 2025

Uh oh!

makslevental commented Sep 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

joker-eph commented Sep 30, 2025

Uh oh!

makslevental commented Sep 30, 2025

Uh oh!

makslevental commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Reland "[MLIR][Python] add Python wheel build demo/test" (#160481) #160488

Reland "[MLIR][Python] add Python wheel build demo/test" (#160481) #160488

Uh oh!

Conversation

makslevental commented Sep 24, 2025

Uh oh!

llvmbot commented Sep 24, 2025

Uh oh!

makslevental commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

joker-eph commented Sep 30, 2025

Uh oh!

makslevental commented Sep 30, 2025

Uh oh!

makslevental commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

makslevental commented Sep 24, 2025 •

edited

Loading