Skip to content

fix(docker): patch pytantan CMakeLists to scrub AVX2 at the source#68

Merged
nextgenusfs merged 3 commits into
mainfrom
fix-pytantan-source-patch
May 25, 2026
Merged

fix(docker): patch pytantan CMakeLists to scrub AVX2 at the source#68
nextgenusfs merged 3 commits into
mainfrom
fix-pytantan-source-patch

Conversation

@nextgenusfs

Copy link
Copy Markdown
Owner

Why

PR #67 pinned pytantan to v0.1.3 and rebuilt it from source with
SKBUILD_CMAKE_ARGS='-DHAVE_AVX2:BOOL=OFF;-DAVX2_C_FLAGS:STRING='. That env
var is a real scikit-build-core knob (it maps to cmake.args), but the
published :latest image proved that the override did not survive into the
final build: container inspection shows a populated pytantan/platform/avx2.so
and pytantan/lib.so with 3,404 AVX/AVX2 instruction hits. As a result
import pytantan SIGILLs under Rosetta 2 on Apple Silicon even on macOS 15.6
with the Apple Virtualization Framework + Rosetta-x86 enabled.

The in-Dockerfile guard added in PR #67 was supposed to catch this and fail
the build, but it didn't — most likely a stale BuildKit GHA layer was reused.

What

Replace the single SKBUILD_CMAKE_ARGS-only rebuild with three independent
gates so no single failure can reintroduce AVX2:

  1. Source patch. Clone pytantan v${PYTANTAN_VERSION} and run a small
    Python script that walks the top-level and src/pytantan/platform/
    CMakeLists.txt, deletes every if(HAVE_SSE4|HAVE_AVX2|HAVE_NEON) ... endif() block (nesting-aware — pytantan's blocks contain a nested
    if(IMPL_FLAGS), so a naive non-greedy regex would mis-balance), and
    removes the corresponding Find*.cmake include() lines. After the
    patch, only the scalar generic Cython extension is in the build graph
    and add_compile_options(-mavx2) is not reachable from any code path.
  2. Real config-settings. Invoke pip with both
    --config-settings=cmake.define.HAVE_{SSE4,AVX2,NEON}=OFF (direct pip
    plumbing) and CMAKE_ARGS=... (standard CMake env), as belt-and-braces.
  3. Cache invalidation. Add ARG PYTANTAN_CACHEBUST (exposed as ENV
    _PT_CACHEBUST inside the RUN) so the layer hash changes when bumped,
    defeating GHA layer reuse that could otherwise mask a broken rebuild.

Also strengthens the verification step: it now enumerates every .so under
the pixi env, prints per-file AVX hit counts to the build log (visible on
success), and fails on any positive count using a broader mnemonic set
(vpbroadcast, vfmadd, vpermd, vpgatherdd, vextracti128/vinserti128,
ymm*, zmm*).

Verification done locally

The Python patcher was sanity-checked against a fresh clone of
althonos/pytantan@v0.1.3. After patching:

  • Root CMakeLists.txt ends up with set(HAVE_SSE4 OFF) / set(HAVE_AVX2 OFF) /
    set(HAVE_NEON OFF) lines in place of the three SIMD blocks, and the
    Find{SSE4,AVX2,NEON}.cmake includes are gone.
  • src/pytantan/platform/CMakeLists.txt contains only the generic
    cython_extension(...) call — the conditional sse4/avx2/neon
    extensions are gone.

Follow-up after merge

  1. Run Actions → Build and Publish Docker Image → Run workflow on main
    (push=true) to overwrite :latest.
  2. In a fresh nextgenusfs/funannotate2:latest:
    python -c "import pytantan; from pytantan.lib import RepeatFinder; print('OK', pytantan.__version__)"
    
    Should now print OK 0.1.3 instead of SIGILLing.

Pull Request opened by Augment Code with guidance from the PR author

Jon Palmer added 3 commits May 25, 2026 08:26
The prior rebuild logic relied on SKBUILD_CMAKE_ARGS='-DHAVE_AVX2=OFF' to
disable AVX2 in pytantan's CMake build. Empirically that single channel did
not make it into the final image: published builds shipped a populated
pytantan/platform/avx2.so and a lib.so with thousands of AVX/AVX2 hits,
causing 'import pytantan' to SIGILL under Rosetta 2 on Apple Silicon.

Replace the rebuild step with three independent gates so no single failure
can reintroduce AVX2:

1. Source patch: clone pytantan v${PYTANTAN_VERSION}, then surgically
   delete the if(HAVE_SSE4|HAVE_AVX2|HAVE_NEON) ... endif() blocks from
   the top-level CMakeLists.txt and src/pytantan/platform/CMakeLists.txt
   (nesting-aware), and drop the corresponding Find*.cmake include()s.
   This leaves only the scalar 'generic' Cython extension in the build
   graph; add_compile_options(-mavx2) is no longer reachable.

2. Real config-settings: invoke pip with
     --config-settings=cmake.define.HAVE_{SSE4,AVX2,NEON}=OFF
   plus CMAKE_ARGS as a belt-and-braces redundancy.

3. Cache invalidation: introduce ARG PYTANTAN_CACHEBUST (exposed as ENV
   _PT_CACHEBUST inside the RUN) so the layer hash changes when bumped,
   defeating BuildKit GHA cache reuse that could mask a broken rebuild.

Strengthen the verification step too: enumerate every .so under the pixi
env, print per-file AVX hit counts to the build log (visible on success),
and fail on any positive count using a broader mnemonic set
(vpbroadcast, vfmadd, vpermd, vpgatherdd, vextracti128/vinserti128,
ymm*, zmm*).
The ghcr.io/prefix-dev/pixi base image has no system python3 on PATH;
the heredoc patcher in the pytantan rebuild RUN failed with exit 127
('python3: not found') before pip was ever invoked. Use the pixi-managed
interpreter at /app/.pixi/envs/default/bin/python, which is guaranteed
to exist after 'pixi install --locked'. Bump PYTANTAN_CACHEBUST to 3 so
the layer is re-executed.
The 'vendor/tantan' subdir of pytantan is a git submodule. Our manual
'git clone --depth 1 --branch v0.1.3' did not initialize submodules, so
'vendor/tantan/src/tantan.cc' was missing and CMake failed with:
  Cannot find source file: cbrc_linalg.cc
  Cannot find source file: .../build/Release/src/tantan/src/tantan.cc

(The previous 'pip install git+...@v0.1.3' form worked by accident
because pip's git fetcher recurses submodules by default.)

Add --recurse-submodules --shallow-submodules to the clone, bump
PYTANTAN_CACHEBUST.
@nextgenusfs nextgenusfs marked this pull request as ready for review May 25, 2026 16:19
@nextgenusfs nextgenusfs merged commit 4aaca6a into main May 25, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant