Add a high-level API#53
Merged
Merged
Conversation
stat(env) and stat(txn, dbi) wrap mdb_env_stat/mdb_stat — the latter
gives access to per-DBI page size and entry counts, needed by callers
like cuTile's DiskCache to derive utilization from the LMDB-owned mmap.
tryget(txn, dbi, key, T) returns Union{T,Nothing} for keys that may not
exist; get(txn, dbi, key, T, default) is the symmetric variant matching
Base.get(dict, key, default). Both use unchecked_mdb_get internally to
distinguish MDB_NOTFOUND from a real error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The new tier-2 cursor API mirrors py-lmdb's navigation surface:
seek! / seek_last! / seek_range! — position via MDB_FIRST / LAST /
SET_KEY / SET_RANGE
next! / prev! — move with MDB_NEXT / MDB_PREV
key / value / item — read at MDB_GET_CURRENT
walk(f, cur; from) — callback over every entry, with
raw MDB_val refs for zero-copy
scans (e.g. cuTile's eviction loop)
Each navigator returns Union{T,Nothing}: nothing on MDB_NOTFOUND, the
key as `T` otherwise. The legacy `LMDBIterator`/`DirectoryLister` and
the `keys(cur, T; prefix)` / `values(cur, T; prefix)` iterators stay in
place for now; Step 4 switches LMDBDict over to the new primitives,
after which the legacy iterator can be deprecated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tier-2 dbi.jl: replace!(txn, dbi, key, val[, V]) — atomic put-and-return-old pop!(txn, dbi, key, T) — atomic get-and-delete Tier-2 cur.jl (DupSort navigation, only meaningful in MDB_DUPSORT databases): seek_first_dup! / seek_last_dup! — MDB_FIRST_DUP / LAST_DUP next_dup! / prev_dup! — MDB_NEXT_DUP / PREV_DUP next_nodup! / prev_nodup! — MDB_NEXT_NODUP / PREV_NODUP Adds test/dupsort.jl exercising the dup navigation, count(cur), and the dup-aware delete!(txn, dbi, key, val) form (already present, now explicitly covered). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tier-2 env.jl additions:
reader_check(env) -> Int — reap stale reader slots; returns count.
Useful in long-running services.
reader_list(env) -> String — human-readable reader-slot dump (header
+ PID/thread/txid per slot).
copy(env, path; compact) — env to directory; mdb_env_copy / _copy2
copy(env, fd; compact) — env to fd; mdb_env_copyfd / _copyfd2
reader_list installs a @cfunction message-collector callback that writes
into an IOBuffer rooted across the ccall via GC.@preserve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The new high-level Environment(path; mapsize, maxreaders, maxdbs, flags, mode) bundles `create()` + `setindex!`s + `open(env, path)` into one call, mirroring py-lmdb / lmdb-rs. If anything fails between create and a successful open, the partially constructed env is closed before rethrowing. LMDBDict delegates to it instead of inlining the sequence. The `flags::Cuint` kwargs on `open` (env + dbi), `start`, `put!`, `delete!` were a usability footgun: passing `MDB_DUPSORT` (UInt8) or `MDB_NOSYNC` (UInt32) sometimes typechecked, sometimes didn't, depending on which constant the user reached for. They now accept any `Integer` and convert to `Cuint` internally; callers no longer need to wrap LMDB constants in `Cuint(...)`. The unused `EnvironmentFlags = Unsigned` typealias is dropped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test/liblmdb.jl exercises the @checked/unchecked_* contract: a bad mdb_env_open auto-throws LMDBError, while unchecked_mdb_get on a missing key returns MDB_NOTFOUND without throwing. test/integration.jl reproduces the cuTile-shaped power-user pattern (Environment kwargs ctor → write txn → cursor walk over raw MDB_val refs → tryget + is_notfound NOTFOUND-tolerant batch delete) so a future change that breaks the zero-copy walk idiom fails loudly. README walks through all three tiers with worked examples, replacing the old build-badge stub. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aligns with Base.delete!'s "if any" contract and the dominant LMDB binding convention (heed, py-lmdb, lmdb-js, lmdbxx). Lets cuTile-shaped batch-delete code use `LMDB.delete!(...) && (n += 1)` directly without try/catch around `is_notfound`. LMDBDict.delete! still returns `d` to preserve the AbstractDict signature.
Power-user packages (cuTile.DiskCache, in particular) need partial
reads — e.g. skip an 8-byte framing prefix — in a single copy. The
existing `mdb_unpack(::Type{T}, ::Ref{MDB_val})` is already
method-dispatched and reachable from outside the module; this commit
just makes that contract explicit (docstring on the generic, worked
example, lifetime caveat) and adds a regression-guard test that
exercises a custom unpack method from outside `LMDB`.
Same idea as heed's `BytesDecode<'txn>` trait, expressed via Julia's
multimethod dispatch. No new exports; no new API.
Mirrors the existing tier-2 typed-overload pattern (tryget(...,T),
seek!(cur,key,T), key(cur,T), item(cur,K,V)): the typed walk decodes
each ref through mdb_unpack(K,...) / mdb_unpack(V,...) and calls
f(k::K, v::V). Stop semantics (return false to halt) are preserved.
Plays the role heed's `iter()` plays for a `BytesDecode` impl: the
caller defines a marker type with a custom mdb_unpack and reads
already-decoded pairs, instead of poking at raw Ref{MDB_val} during
iteration.
The MDB_envinfo and MDB_stat fields had C-style ms_*/me_* prefixes that leaked into user code. Wrap them in NamedTuples with Julian field names (`mapsize`, `entries`, `branch_pages`, …) so callers stop reading C struct field names. Heed and lmdbxx do the same. Updated LMDBDict.length and the existing tests to the new field names. The raw MDB_envinfo / MDB_stat structs are still exported via tier-1 for callers that need the bit-level layout.
Calls f(buf::Vector{UInt8}) with an unsafe_wrap'd view over LMDB's
mmap-allocated write buffer of the requested size, so the caller can
fill bytes directly (e.g. unsafe_store! for a header + copyto! for a
payload) without first building an intermediate Vector and having LMDB
copy it.
This is the analogue of heed's `Database::put_reserved` and lmdbxx's
MDB_RESERVE-flagged dbi_put. The buf is only valid inside f and inside
the enclosing write txn; can't be used with DUPSORT/DUPFIXED.
`env[:Bogus] = 1` and `env[:Bogus]` previously fell through to a `@warn` and silently returned a placeholder value, hiding typos. They now throw `ArgumentError` with the supported option list. `Cursor.delete!` previously returned the raw status without either a docstring or a Bool/idempotent contract. Adds a docstring and a regression test. Note the asymmetry with `delete!(txn, dbi, key)`: `mdb_cursor_del` returns `EINVAL` (not `MDB_NOTFOUND`) on an unpositioned cursor, so the Bool/idempotent shape doesn't apply — position the cursor first if you want to recover from a missing entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #53 +/- ##
==========================================
+ Coverage 73.42% 80.59% +7.17%
==========================================
Files 9 9
Lines 538 701 +163
==========================================
+ Hits 395 565 +170
+ Misses 143 136 -7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Something beyond the Dict API. Motivated by / Example use in JuliaGPU/cuTile.jl@901c47f