v0.1.4

First stable release of smugmap — a small LD_PRELOAD shim that lets any unmodified Linux binary read files directly from S3, page by page on demand.

What it does

Intercepts open, mmap, pread, read (including their _64 LFS variants). Matched paths get a synthetic fd backed by /dev/null; a background thread serves page faults from S3 via userfaultfd and HTTP Range GETs. Only the bytes the program actually touches are fetched.

Works where FUSE can't:

• AWS Lambda (no /dev/fuse, no root)
• Rootless / unprivileged containers
• Kubernetes without host-path mounts
• Most CI runners

Install

Prebuilt:

• smugmap-x86_64-linux.so — for Intel/AMD Lambda, most VMs and CI
• smugmap-aarch64-linux.so — for Graviton Lambda, ARM containers

Drop it wherever, then LD_PRELOAD=/path/to/smugmap.so. See README for the JSON config format.

From source:

cargo build --release
sudo make install

What's inside

• SigV4 built in — reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION and AWS_SESSION_TOKEN, so Lambda execution roles work out of the box.
• HTTPS fallback — if you don't have creds at runtime, presigned URLs (or any HTTP endpoint with Range support) work too.
• Read-only — write opens (O_WRONLY, O_RDWR) are rejected with EROFS. SQLite and DuckDB retry with O_RDONLY automatically.
• LFS syscall coverage — pread64, fstat64, mmap64 aliased so C programs built with _FILE_OFFSET_BITS=64 (which is essentially all modern C) are intercepted correctly.
• Rust 2024 edition, ~200 KB .so, no runtime dependencies.

Runnable examples

Under examples/:

• sqlite/ — query a SQLite database in S3 without downloading
• duckdb/ — analytical queries over a Parquet file in S3
• llama/ — run llama.cpp inference against a GGUF model in S3, streaming weights on demand

Requirements

• Linux with vm.unprivileged_userfaultfd=1 (default on modern kernels and AWS Lambda ARM64)
• Bucket accessible via SigV4 credentials or a public/presigned URL