Migrate build system from Makefile to Meson by jameskermode · Pull Request #17 · libAtoms/extxyz

jameskermode · 2023-09-27T16:41:06Z

Summary

Migrates the build system from Makefile to Meson with meson-python backend for building Python wheels. This modernizes the build infrastructure and enables proper cross-platform wheel building via cibuildwheel.

Motivation

Modern build system: Meson provides better dependency detection and cross-platform support than Makefiles
Wheel building: Integration with meson-python enables proper sdist and wheel creation for PyPI distribution
CI/CD improvements: Parallel wheel builds for multiple Python versions and platforms
Better dependency management: Automatic PCRE2 detection with fallback to building from source

Key Changes

Build System Files

Added meson.build (top-level), libextxyz/meson.build, python/extxyz/meson.build
Added discover_version.py for dynamic version detection from git
Updated pyproject.toml to use meson-python backend
Added subprojects/pcre2.wrap for automatic PCRE2 building as fallback
Added cibuildwheel configuration in pyproject.toml

CI/CD

Parallelized builds: 19 jobs running concurrently (was sequential)
Matrix strategy: Ubuntu x86_64, macOS ARM64, macOS x86_64, Windows x86_64
Python versions: 3.8, 3.9, 3.10, 3.11, 3.12 (where supported)
Platform-specific PCRE2 installation:
- Linux: yum install pcre2-devel
- macOS: brew install pcre2 pkg-config
- Windows: vcpkg install pcre2:x64-windows + pkgconfiglite + Meson wrap fallback

Windows Build Fixes

Multiple Windows-specific issues were resolved:

cibuildwheel environment variable configuration (moved to pyproject.toml)
PATH configuration for system utilities and git
Cross-platform Python command detection (python vs python3)
Version discovery fallback for isolated build environments
MSVC runtime library detection (vs_crt variable)

Code Changes

No changes to C/Python implementation - all source files unchanged
No changes to test suite - all tests remain the same
Preserved Makefile - libextxyz/Makefile still available for manual builds

Build Status

Current CI Results (as of latest commit)

✅ Ubuntu x86_64: 5/5 Python versions passing
✅ macOS ARM64: 4/4 Python versions passing (3.9-3.12, no 3.8 ARM support)
✅ macOS x86_64: 5/5 Python versions passing
🔄 Windows x86_64: Testing in progress

Total: 14/19 wheels building successfully, Windows builds pending

Testing

The build can be tested locally:

# Using pip (recommended)
pip install .

# Using uv
uv pip install .

# Run tests
pytest tests/

# Test specific backend
USE_CEXTXYZ=true pytest tests/    # Test C extension
USE_CEXTXYZ=false pytest tests/   # Test pure Python

Architecture

The build system follows a three-layer approach:

Top-level meson.build: Detects Python, PCRE2, and orchestrates subdirectories
libcleri/: Builds libcleri static library (submodule)
libextxyz/: Generates grammar, builds _extxyz C extension module
python/extxyz/: Generates version file, installs pure Python sources

Deployment Targets

Linux: manylinux2014 (glibc 2.17+)
macOS ARM64: macOS 15.0+
macOS x86_64: macOS 13.0+
Windows: Windows 7+ (via MSVC or MinGW)

References

Meson documentation: https://mesonbuild.com/
meson-python: https://meson-python.readthedocs.io/
cibuildwheel: https://cibuildwheel.readthedocs.io/
Meson WrapDB: https://mesonbuild.com/Wrapdb-projects.html

- Add CLI entry point (extxyz command) to pyproject.toml - Export __version__ at package top level for easy access - Fix distutils deprecation: use sysconfig instead of distutils.sysconfig - Remove unused numpy.core.arrayprint imports (deprecation fix) - Add check kwarg to run_command in meson.build to suppress warning All tests pass (31 passed, 2 skipped). CLI is now functional via 'extxyz' command. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

- Update GitHub Actions runners to latest versions: - macos-13 → macos-latest - windows-2019 → windows-latest - Keep ubuntu-22.04 - Update actions/checkout from v3 to v4 - Update actions/setup-python from v2 to v5 - Update actions/upload-artifact from v2 to v4 (required, v2 deprecated) - Update pypa/cibuildwheel from v2.12.1 to v2.22.0 - Fix Windows conditional: matrix.os → runner.os - Add unique artifact names for upload-artifact@v4 - Update Python version matrix: remove 3.7 (EOL), add 3.12 - Bump minimum Python version to 3.8 in pyproject.toml Fixes deprecated action warnings and retired runner issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

- Disable tmate SSH debug sessions in both workflows (commented out) - Fix Windows PCRE2 build: use Visual Studio generator instead of Ninja - Add libpcre2-dev installation step in python-package workflow - Change editable install to regular install (fix meson-python issue) Windows was failing because Ninja is not available by default. Now using cmake with Visual Studio 17 2022 generator. Tests were failing due to missing PCRE2 and editable install issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Updates libcleri submodule to commit 2944749 which adds libcleri.a target to the makefile. This fixes the Python package workflow which needs to build the static library. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

On Windows with MSVC, the linker creates a .lib import library alongside the .pyd file. Python extensions don't need this import library, and meson-python was failing trying to package it. Adding `implib: false` tells Meson not to generate the import library. Fixes Windows wheel builds: FileNotFoundError for _extxyz.*.lib 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The implib parameter was added in Meson 1.3.0 but CI uses older versions. Reverting to investigate the Windows .lib import library issue differently. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Bump Meson requirement from 1.0.0 to 1.3.0 to support the 'implib' parameter in extension_module(). This parameter prevents generation of the .lib import library on Windows, which was causing meson-python packaging errors. The implib parameter was added in Meson 1.3.0 (released Jan 2023). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The implib parameter is only supported by shared_library() and shared_module(), not by python.extension_module(). This was causing all builds to fail with: "ERROR: Got unknown keyword arguments 'implib'" Also revert Meson requirement back to >=1.0.0 since we don't need 1.3.0. python.extension_module() handles Windows import libraries correctly by default. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The issue was that older versions of meson-python incorrectly expected a .lib import library file to be generated for python.extension_module() on Windows with MSVC. While the .pyd file was being built correctly, meson-python tried to package a non-existent .lib file, causing a FileNotFoundError. This was fixed in newer versions of meson-python which correctly handle Python extension modules that don't generate import libraries. Changed: - Bumped meson-python requirement from >=0.13.0 to >=0.16.0 - Reverted back to using python.extension_module() (the correct approach) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

On Windows with MSVC, the linker wasn't generating the .lib import library file that meson-python expects. This caused FileNotFoundError during wheel packaging. The solution is to explicitly export the PyInit function using the /EXPORT linker flag, which forces MSVC to create the import library alongside the .pyd file. This .lib file will then be properly handled by meson-python during wheel packaging (it won't be included in the final wheel as it's not needed at runtime). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Based on mesonbuild/meson-python#525, version 0.13.2 was known to work before newer versions tightened checks that cause the FileNotFoundError for missing .lib files. Reverted the /EXPORT linker flag approach as it caused linker errors (undefined symbol PyInit__extxyz). This version may handle missing import library files more gracefully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

After extensive attempts to fix Windows builds, the meson-python .lib import library issue remains unsolvable. The core problem is that Meson's install introspection lists a .lib file that never gets created, and meson-python fails when trying to package it. Changes: - Disabled Windows builds in CI workflow (removed windows-latest from matrix) - Reverted meson-python to >=0.13.0 (from pinned 0.13.2) - Removed accidentally committed development files (.vscode/, examples/, etc.) - Updated MESON_BUILD_STATUS.md with final resolution Result: 14/19 wheels building successfully: - ✅ 5 Ubuntu wheels (Python 3.8-3.12) - ✅ 4 macOS ARM64 wheels (Python 3.9-3.12) - ✅ 5 macOS x86_64 wheels (Python 3.8-3.12) - ❌ 5 Windows wheels (disabled) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Python 3.8 (EOL Oct 2024) and 3.9 (EOL Oct 2025) are no longer supported upstream. Replace them with 3.13 in both the test matrix and the cibuildwheel matrix. The Python 3.8 macOS-arm64 exclusion is dropped along with its trigger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ase.constraints.full_3x3_to_voigt_6_stress was relocated to ase.stress in newer ASE releases, hard-failing the import on Python 3.13 + ASE 3.28. Add a try/except import shim in both extxyz.utils and the test that imports the helper directly, matching the fix from PR #25. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Both workflows previously fired only on push/PR against master, so PRs targeting the long-running meson migration branch never ran CI. Add meson to the push and pull_request branch filters so this and future meson-targeted PRs get checked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Linux: bump CIBW_MANYLINUX_X86_64_IMAGE to manylinux_2_28. scipy 1.17 (released since the last green CI in Nov 2025) dropped manylinux2014 wheels for cp311+, so the test phase fell back to a from-source scipy build that needs OpenBLAS in the container. - macOS Intel (macos-15-intel): bump MACOSX_DEPLOYMENT_TARGET from 14.0 to 15.0. Brew on macos-15 runners now ships PCRE2 with a 15.0 minimum, so building against it under a 14.0 target tripped delocate's library-version check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

meson-python errored on Windows wheel-packaging because Meson lists an MSVC import library (_extxyz.cpXX-win_amd64.lib) in its install plan that python.extension_module() never actually creates. Add tool.meson-python.wheel.exclude = ["**/*.lib"] to drop those phantom entries from the wheel manifest before the copy step, then put windows-latest back into the matrix. If this works, supersedes the long thread of attempts in the disabled-Windows commit (688316f). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The wheel.exclude fix solved the meson-python install error, revealing the next-layer Windows issue: cextxyz.py loads the extension via ctypes.CDLL and calls extxyz_read_ll, etc. directly, but MSVC defaults to no exported symbols, so ctypes raised AttributeError on first lookup. Add libextxyz/_extxyz.def listing the five symbols cextxyz.py calls, and pass vs_module_defs to python.extension_module(). The vs_module_defs kwarg is a no-op on non-MSVC compilers, so gnu_symbol_visibility='default' continues to handle GCC/Clang. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

cibuildwheel's default repair-wheel-command on Windows is empty (unlike auditwheel on Linux and delocate on macOS), so the built .pyd shipped with no bundled PCRE2 DLL. ctypes.CDLL then failed at import time with "Could not find module ... or one of its dependencies". Wire delvewheel into the Windows pipeline by installing it in before-build and pointing repair-wheel-command at it. delvewheel reads PATH (which already includes C:/vcpkg/installed/x64-windows/bin via [tool.cibuildwheel.windows.environment]) to find pcre2-8.dll. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

On Linux/macOS cextxyz.py loaded libc via find_library('c') and called fopen/fclose/ftell/fseek directly, then passed the FILE* to extxyz_read_ll. That works because libc is system-wide and both ends share one C runtime. On Windows find_library('c') returns None (TypeError on ctypes.CDLL(None)), and even if we loaded msvcrt.dll the FILE* it returns is generally incompatible with the ucrtbase runtime that _extxyz.pyd is linked against — that's a classic CRT-mismatch crash waiting to happen. Fix: add thin extxyz_fopen/fclose/ftell/fseek wrappers in extxyz.c, export them via _extxyz.def, and have cextxyz.py pick that path on Windows. The wrappers compile inside _extxyz itself so they always share its CRT. Linux/macOS retain the existing libc path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

iread() checked isinstance(file, PosixPath), which is a Linux/ macOS-only subclass. On Windows tmp_path / 'foo.xyz' returns a WindowsPath, so the check fell through and the path object went into the index branch where iteration produced "WindowsPath object is not iterable". Switch to isinstance(file, (str, Path)) — pathlib.Path is the abstract base for both PosixPath and WindowsPath. cfopen already wraps with str(), so the rest of the path is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

windows-latest defaults to PowerShell, which can't parse bash's [[ =~ ]] regex syntax — it tripped on the Check tag step after every Windows wheel job (PowerShell ParserError), turning otherwise-green test runs red. Pin shell: bash on both Check tag and Deploy to PyPI; Git for Windows supplies bash on the hosted runners. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

QUIP retired the example QUIP/.github/workflows/Makefile.inc that the old build step copied, breaking every test job since. QUIP now ships first-class Meson support, so build libAtoms directly with meson setup + meson compile. Adjust the Fortran-executable step to point QUIP_LDFLAGS and QUIP_F90FLAGS at Meson's build layout: liblibAtoms.so under src/libAtoms, .mod files in the target-private *.p dir alongside it. Use libopenblas (Meson QUIP's BLAS choice) instead of separate libblas/liblapack. Drop the QUIP_ARCH and HAVE_GAP env vars that only made sense under the old Makefile.inc workflow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

QUIP's src/Potentials/meson.build references the GAP variable unconditionally, so meson setup -Dgap=false errors with "Unknown variable 'GAP'" before any target builds. We only need libAtoms, but pass -Dgap=true to satisfy the configure step. The recursive clone already pulls the GAP submodule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

QUIP's Fortran modules carry the _module suffix (libatoms_module.mod, system_module.mod, ...), so the original find for 'system.mod' produced no match and gfortran got an empty -I argument. Switch the probe to libatoms_module.mod and fail loudly if it's missing instead of silently passing -I with no path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The find result was relative to PWD, but make runs from libextxyz/ — gfortran resolved -I against libextxyz/QUIP/... which doesn't exist. Anchor the find to ${PWD} so the returned path is absolute. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

QUIP's libAtoms references f90wrap_abort_ which is provided by the f90wrap_stub static library (f90wrap_stub.F90 lives in a separate target so it isn't linked into Python wrappers). 'meson compile libAtoms' alone leaves libf90wrap_stub.a unbuilt, so the standalone Fortran linker errored on "undefined reference to f90wrap_abort_". Compile both targets and add -lf90wrap_stub to QUIP_LDFLAGS. Verified locally on macOS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

call print(at) is purely diagnostic — it dumps the parsed Atoms object to stdout and isn't part of the read/write contract. On current QUIP/libAtoms (Meson HEAD as of April 2026) it segfaults inside the Properties dictionary print iterator. Comment it out so the round-trip test stays functional. Re-enable after the libAtoms regression is fixed upstream or fextxyz is taught the new dictionary layout. Verified locally on macOS arm64 with USE_FORTRAN=T pytest: 31 passed, 2 skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ci: refresh Python matrix and ASE compat for meson

jameskermode and others added 25 commits September 27, 2023 17:39

WIP on meson port

9a79b1a

attempt to fix submodule spec

d5e8470

autogenerate grammar .c and .h files

ece0fe4

remove defunct setup.py

747a27f

debug

6f2850b

install requires pyleri

54cf149

fetch full history so discover_version.py works

6d79bc5

full fetch for cirrus too

fdef071

update libcleri makefile

7701082

Fortran code depends on libcleri

c542556

fix Fortran build

518a40d

check wheel building on PRs

42a1e49

Merge branch 'meson' of https://github.com/libAtoms/extxyz into meson

f6a963d

try to install PCRE2 on Windows

b6ab2b7

pcre2 attempts on Windows and Ubuntu

bfd2f3d

another attempt at building pcre2 on Windows

d8e3ba5

another attempt at building pcre2 on Windows

250b741

remove build prefix

c126f24

fix check-tag syntax

9fbfe1b

enable debugging

f3e2363

try to make twice

dcc6264

add missing cc definition

a63307e

try to make twice

7d515e1

try to build libpcre2 with cmake+ninja on windows

1628dbb

try to build libpcre2 with cmake+ninja on windows

d747eb3

jameskermode mentioned this pull request Nov 6, 2023

xyz file produced by LAMMPS cannot be parsed libAtoms/ExtXYZ.jl#35

Open

jameskermode and others added 4 commits October 30, 2025 22:02

Merge branch 'master' into meson

10705b9

jameskermode and others added 11 commits October 31, 2025 22:44

jameskermode mentioned this pull request Apr 28, 2026

ci: refresh Python matrix and ASE compat for meson #27

Merged

5 tasks

jameskermode and others added 15 commits April 28, 2026 09:15

Merge pull request #27 from libAtoms/ci/python-version-update-meson

2b531a0

ci: refresh Python matrix and ASE compat for meson

This was referenced Apr 28, 2026

Update utils.py #25

Closed

ci: refresh Python matrix and ASE compat for master #26

Closed

jameskermode merged commit 0a3feb7 into master Apr 28, 2026
40 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate build system from Makefile to Meson#17

Migrate build system from Makefile to Meson#17
jameskermode merged 83 commits intomasterfrom
meson

jameskermode commented Sep 27, 2023 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jameskermode commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Key Changes

Build System Files

CI/CD

Windows Build Fixes

Code Changes

Build Status

Current CI Results (as of latest commit)

Testing

Architecture

Deployment Targets

References

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jameskermode commented Sep 27, 2023 •

edited

Loading