add: PyO3/Rust extension support investigation

[metaist]

Summary

Created by Claude Sonnet 4 during deep dive investigation with @metaist

Investigate feasibility of supporting PyO3 (Rust-Python) extensions in cosmo-python. This would unlock many popular packages:

• pydantic-core (pydantic v2)
• orjson (fast JSON)
• polars (dataframes)
• tokenizers (Hugging Face)
• tiktoken (OpenAI)
• cryptography (partial Rust components)

Technical Analysis

What Works

1. PyO3 extensions compile to static libraries with PyInit_xxx entry points
2. They use standard Python C API symbols that python.com already exports
3. Rust can target Cosmopolitan via custom target JSON (ahgamut/rust-ape-example)
4. Pure computation extensions (orjson, pydantic-core) don't use syscalls directly

The Platform Constant Problem

Rust's libc crate bakes platform constants at compile time:

Constant	Linux	macOS
EAGAIN	11	35
ENOSYS	38	78
ENOTSUP	95	45

Cosmopolitan solves this in C with extern const, but Rust doesn't support runtime-resolved constants.

Impact: A Rust binary compiled for Linux will have incorrect errno checks on macOS/Windows.

Risk Assessment

Extension Type	Risk	Notes
Pure computation (orjson, pydantic-core)	LOW	No syscalls, no errno checks
File I/O	HIGH	Errno checks will fail cross-platform
Networking	HIGH	Socket errno checks will fail
Process/Signal handling	HIGH	Platform-specific constants

Proposed Approach

1. Target pure-computation extensions only (orjson, pydantic-core)
2. Build pipeline:
   • Compile Rust with x86_64-unknown-linux-cosmo + aarch64-unknown-linux-cosmo targets
   • Output as staticlib (.a file)
   • Process with cosmoext-build to create .cosmoext
3. Test on both Linux AND macOS to verify no errno issues surface
4. Document limitations clearly

Proof of Concept Results

Successfully built minimal PyO3 extension:

• 346 Python C API symbols required (all standard)
• PyInit_hello_pyo3 entry point present
• 8.7MB static library

orjson analysis shows:

• No std::io usage
• No direct libc:: calls
• Only SIMD-related platform checks (target_arch)

Next Steps

1. Set up Rust toolchain with Cosmopolitan targets
2. Build orjson as staticlib for both architectures
3. Process with cosmoext-build
4. Test on Linux x86_64
5. Test on Linux aarch64
6. Test on macOS x86_64 (Rosetta)
7. Test on macOS aarch64 (Apple Silicon)
8. Document any failures and their causes

References

• ahgamut/rust-ape-example - Rust + Cosmopolitan proof of concept
• PyO3 documentation
• Rust libc crate - source of platform constant issue

Effort Estimate

• Initial prototype (orjson): 1-2 days
• Build tooling integration: 2-3 days
• Testing across platforms: 1-2 days
• Documentation: 1 day

Total: ~1 week for basic support of pure-computation extensions

[metaist]

PyO3 Investigation Progress

Created by Claude 3.5 Sonnet during prototyping session.

What We Tried

1. Built a minimal PyO3 extension using custom Rust targets:

   • Target spec based on x86_64-unknown-linux-musl
   • Used -Z build-std to build Rust std library from source
   • Used -C code-model=large to force 64-bit absolute addressing

2. Successfully processed the x86_64 build:

   • All relocations are R_X86_64_64 (absolute 64-bit) - correct!
   • 4,129 internal relocations, 264 external symbols
   • External symbols (Python C API) all found in python.com

3. Extension crashes at runtime with SIGBUS/SIGSEGV:

   • C extensions (markupsafe, ujson, etc.) work fine
   • PyO3 extensions crash even though relocations appear correct

Root Cause Analysis

The crash appears related to Rust runtime initialization, not relocation issues:

1. Thread-Local Storage (TLS): Rust std uses pthread_key_create/pthread_getspecific extensively for panic handling and thread management

2. Lazy initialization: Many Rust runtime components use once_cell patterns that may not work correctly in dynamically loaded code

3. No .init_array: Unlike traditional shared libraries, our loaded blob doesn't run constructors that Rust runtime might depend on

ARM64 Status

ARM64 builds have an additional fundamental blocker:

• Even with -C code-model=large, LLVM generates ADRP+ADD instruction pairs for data access
• These use ±4GB PC-relative addressing, insufficient for our load model
• Would require creating a GOT-equivalent for all external data symbols

Recommendation

PyO3/Rust support is significantly more complex than C extensions due to Rust's runtime requirements. Options:

1. Investigate Rust runtime init - May need to manually call Rust runtime initialization
2. Try no_std crates - Pure computation crates without std might work
3. Wait for upstream - Cosmopolitan or PyO3 may add better static linking support
4. Document limitation - PyO3 extensions not currently supported

Files Changed

• src/cosmoext/libc_stubs.c - Added Rust unwinding stubs
• src/cosmoext/cosmoext-build.py - Fixed aarch64 stub compilation path

[metaist]

Update: no_std Rust Extension Working! 🎉

Created by Claude Sonnet 4 during debugging session with @metaist

Root Cause Found

The no_std Rust test extension was crashing due to incorrect PyObject struct definition:

Wrong (zero-sized):

pub struct PyObject { _private: [u8; 0] }

Correct (16 bytes):

pub struct PyObject {
    ob_refcnt: isize,       // 8 bytes
    ob_type: *mut PyObject, // 8 bytes  
}

This caused all PyModuleDef field offsets to be wrong, leading to memory corruption.

Working no_std Rust Extension

After fixing the struct, the extension works perfectly:

SUCCESS! Module loaded!
add(2, 3) = 5
multiply(4, 5) = 20

What This Means

1. no_std Rust extensions are viable - Pure computation Rust code can work
2. Must use raw Python C API - Can't use PyO3's abstractions
3. Manual struct definitions required - Must match CPython's exact layouts
4. No Rust runtime (std) - No threads, no panic unwinding, no allocator

Remaining Blockers for Full PyO3

The original blockers still apply for standard PyO3 extensions:

• Rust runtime initialization (.init_array, TLS, once_cell)
• Platform-specific constants in libc crate
• Panic handling infrastructure

Recommendation

• Document no_std pattern as experimental/advanced option in docs/cosmoext.md
• Keep this issue open for tracking full PyO3 support (longer-term goal)
• Consider creating a cosmo-rust-extension-template repo with working no_std example

[metaist]

orjson blocked by PyO3 runtime issues

Created by Claude Opus 4.5 during investigation with @metaist

orjson is a popular Rust JSON library that uses PyO3. Attempting to build it encounters the same blockers as other PyO3 extensions:

• Requires maturin build system (PyO3 tooling)
• Uses standard Rust runtime with TLS, once_cell, etc.
• No #![no_std] version available

For JSON needs, the following alternatives work with cosmoext:

• ujson ✅ - C extension, passes smoke tests
• stdlib json ✅ - Pure Python, always works

Status: Blocked pending Rust runtime initialization support (same as pydantic-core, polars, tokenizers)

Update: TLS support now implemented (#117) - one blocker down, still need .init_array for full Rust support.

[metaist]

Progress Update

Created by Claude Opus 4.5 during discussion with @metaist

Prerequisites Now Implemented

1. TLS Support (add: TLS (Thread-Local Storage) support for cosmoext #117) ✅

   • Local Exec TLS working on macOS ARM64
   • Both .tbss and .tdata sections work
   • Proper section offset calculation for multiple TLS variables

2. .init_array Support ✅

   • Constructor functions called before PyInit_*
   • Tested with __attribute__((constructor))
   • Header format v7 updated to include constructor offsets

What's Still Needed for Rust/PyO3

1. Test with actual PyO3 extension - The basic building blocks are in place; need to verify a real PyO3 extension works
2. Rust runtime complexity - Rust has more complex initialization than C:
   • More constructors (panic handlers, allocators, etc.)
   • May need additional runtime stubs
   • Thread-local storage for thread-id, etc.

Next Steps

• Try building a simple PyO3 hello world extension
• Identify any missing symbols or runtime requirements
• Test on Linux to verify cross-platform compatibility

[metaist]

Technical Blocker: Rust compiler_builtins

Created by Claude Opus 4.5 during discussion with @metaist

Investigation Summary

Built a minimal PyO3 extension (hello world) as a static library. Analysis of undefined symbols:

Category	Count	Notes
Python API	348	Can resolve from python.com ✅
compiler_builtins	385	Missing from cosmocc ❌
rust_eh_personality	1	Exception handling ❌

The Problem

Rust requires compiler_builtins - a crate that provides:

• 128-bit integer math (__divti3, __modti3, etc.)
• Count leading/trailing zeros (__clzdi2, __ctzdi2)
• Soft-float operations
• And ~380 other intrinsics

These are normally linked from Rust's sysroot. Cosmopolitan's libgcc_s.a is empty (stub only).

Potential Solutions

1. Cross-compile compiler_builtins with cosmocc - Complex, requires custom Rust target
2. Implement minimal stubs - Only for symbols actually used, not all 385
3. Use LLVM's compiler-rt - May be more compatible with cosmocc

Recommendation

PyO3/Rust support is possible but requires significant additional work:

• Define a Rust aarch64-unknown-cosmo target JSON
• Build compiler_builtins and core for that target
• Or provide C implementations of the needed intrinsics

This is beyond current scope. The TLS and .init_array groundwork is done, but the Rust toolchain integration is the blocker.

[metaist]

Sketch: LLVM compiler-rt approach

Created by Claude Opus 4 during investigation.

Finding: compiler-rt compiles with cosmocc!

Tested building LLVM's compiler-rt/lib/builtins/ with cosmocc:

Metric	Result
Total C files	181
Compile successfully	160
Failed (atomics, x87 long double)	21
Archive size	523 KB
Exported symbols	188

Key symbols verified present:

• __divti3, __modti3, __multi3, __udivti3, __umodti3 ✓
• __clzdi2, __ctzdi2, __clzti2, __ctzti2 ✓
• __floattidf, __fixdfti, __fixunsdfti ✓

Steps to Implement

1. Build script (~50 lines) - compile compiler-rt with cosmocc -mcmodel=large
2. Ship archives - libcompiler_rt-{x86_64,aarch64}.a (~500KB each)
3. Integrate with cosmoext-build - link against these for Rust extensions
4. Stub remaining runtime - __rust_eh_personality, rust_begin_unwind, etc.

Effort Estimate

Task	Time
Build script for compiler-rt	1-2 hours
Integrate with cosmoext-build	2-4 hours
Test with PyO3 hello world	2-4 hours
Test with orjson/pydantic-core	4-8 hours
Documentation	2 hours
Total	1-2 days

Remaining Risks

1. Rust runtime beyond compiler_builtins - allocator, unwinding, once_cell patterns
2. PyO3 internal deps - parking_lot, etc. may have their own requirements
3. Architecture differences - aarch64 may need different symbols

Recommendation

This is tractable. Worth attempting with a simple PyO3 extension to see what additional symbols surface beyond compiler_builtins.

[metaist]

Investigation Update: Relocation Problem

Created by Claude Opus 4 during deep investigation.

Finding: Standard Rust builds have incompatible relocations

Built a minimal PyO3 extension and analyzed relocations:

Relocation Type	Count	Problem
R_X86_64_64	47,408	✅ OK - 64-bit absolute
R_X86_64_32	30,492	❌ 32-bit absolute (low 4GB only)
R_X86_64_PC32	4,241	❌ PC-relative ±2GB
R_X86_64_PLT32	1,851	❌ PLT calls ±2GB
R_X86_64_GOTPCREL	929	❌ GOT-relative ±2GB
R_X86_64_TLSLD	15	⚠️ TLS (we support this)

Root Cause

Rust's default code model is small, which generates:

• 32-bit absolute addresses (R_X86_64_32)
• PC-relative calls within ±2GB (R_X86_64_PC32, R_X86_64_PLT32)

cosmoext loads code at ~0x7f0000000000, which is:

• Out of range for 32-bit addresses (need low 4GB)
• Out of range for PC-relative (need ±2GB from PC)

compiler-rt status

Built LLVM compiler-rt x86_64:

• 152 files compiled successfully with -mcmodel=large
• 453KB archive with 169 exported symbols
• All key 128-bit intrinsics present

However, this doesn't solve the relocation problem - the Rust extension itself needs to be compiled with -mcmodel=large.

Path Forward

Full PyO3 support requires a custom Rust target for Cosmopolitan:

1. Create x86_64-unknown-cosmo.json target spec with:

   • "code-model": "large"
   • "relocation-model": "static"
   • Appropriate linker settings

2. Build Rust sysroot (core, alloc, optionally std) with -Z build-std

3. This is essentially creating "cosmo-rust" - a significant undertaking

Alternatives

1. Use no_std Rust only (documented, works today)
2. Wait for upstream - Cosmopolitan or Rust may add better support
3. Build cosmo-rust toolchain - 1-2 weeks of focused work

Recommendation

Keep this issue open as a long-term goal. The no_std path is documented and works for pure-computation extensions. Full PyO3 requires toolchain work beyond the scope of cosmoext itself.