Skip to content

Releases: VarNotUsed/smugmap

Release list

v0.2.0

v0.2.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Sep 11:26
v0.2.0
987f49c

Adds inline JSON config so wrappers and Lambda handlers can skip the temp-file dance.

New

SMUGMAP_CONFIG_JSON env var — pass the config directly instead of pointing at a file:

SMUGMAP_CONFIG_JSON='[{"pattern":"*.db","url":"s3://my-bucket/db"}]' \
LD_PRELOAD=./smugmap.so \
  sqlite3 /remote/db "SELECT count(*) FROM events"

Takes precedence over SMUGMAP_CONFIG when both are set. SMUGMAP_CONFIG still works exactly as before.

Why

Programmatic setups (CDK constructs, npm wrappers, Lambda entry scripts, one-liner shell commands) previously had to:

  1. Serialize the config to JSON
  2. Write it to a temp file with chmod 600
  3. Set SMUGMAP_CONFIG to the path
  4. Clean up on exit

With SMUGMAP_CONFIG_JSON, all of that collapses to setting one env var. The Quick Start and every example in the repo now use this shorter form.

Compatibility

Fully backward-compatible. No breaking changes. Existing SMUGMAP_CONFIG=/path/to/file setups keep working with no changes required.

Assets

Platform Asset
Linux x86_64 smugmap-x86_64-linux.so
Linux aarch64 smugmap-aarch64-linux.so
Assets 4
Loading

v0.1.4

v0.1.4

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Sep 11:02
v0.1.4
d32257a

First stable release of smugmap — a small LD_PRELOAD shim that lets any unmodified Linux binary read files directly from S3, page by page on demand.

What it does

Intercepts open, mmap, pread, read (including their _64 LFS variants). Matched paths get a synthetic fd backed by /dev/null; a background thread serves page faults from S3 via userfaultfd and HTTP Range GETs. Only the bytes the program actually touches are fetched.

Works where FUSE can't:

  • AWS Lambda (no /dev/fuse, no root)
  • Rootless / unprivileged containers
  • Kubernetes without host-path mounts
  • Most CI runners

Install

Prebuilt:

  • smugmap-x86_64-linux.so — for Intel/AMD Lambda, most VMs and CI
  • smugmap-aarch64-linux.so — for Graviton Lambda, ARM containers

Drop it wherever, then LD_PRELOAD=/path/to/smugmap.so. See README for the JSON config format.

From source:

cargo build --release
sudo make install

What's inside

  • SigV4 built in — reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION and AWS_SESSION_TOKEN, so Lambda execution roles work out of the box.
  • HTTPS fallback — if you don't have creds at runtime, presigned URLs (or any HTTP endpoint with Range support) work too.
  • Read-only — write opens (O_WRONLY, O_RDWR) are rejected with EROFS. SQLite and DuckDB retry with O_RDONLY automatically.
  • LFS syscall coverage — pread64, fstat64, mmap64 aliased so C programs built with _FILE_OFFSET_BITS=64 (which is essentially all modern C) are intercepted correctly.
  • Rust 2024 edition, ~200 KB .so, no runtime dependencies.

Runnable examples

Under examples/:

  • sqlite/ — query a SQLite database in S3 without downloading
  • duckdb/ — analytical queries over a Parquet file in S3
  • llama/ — run llama.cpp inference against a GGUF model in S3, streaming weights on demand

Requirements

  • Linux with vm.unprivileged_userfaultfd=1 (default on modern kernels and AWS Lambda ARM64)
  • Bucket accessible via SigV4 credentials or a public/presigned URL
Loading