feat(bindings): put_documents / add_documents across python, nodejs, wasm, ruby, php (#866)#871
Merged
Conversation
…wasm, ruby, php (#866) Expose the Engine::put_documents / add_documents batch API (#551) in all five language bindings. Each gains put_documents / add_documents (putDocuments / addDocuments in the camelCase hosts) taking a list of (id, document) pairs — the direct mirror of the core Vec<(String, Document)> signature and the batched form of each binding's existing singular put_document(id, doc). Semantics carry through from the core: sequential in-order application, one WAL fsync per batch, duplicate-id dedup within a put batch (last wins), chunk accumulation for add, and fail-fast with the applied prefix left in place (retrying is idempotent). Each converter names the offending position on a malformed entry, and an empty batch is a no-op. The BatchIngest error surfaces message-only through each binding's existing error mapper (the message carries failed_index / failed_id / applied via Display). Per-binding marshalling: Python iterates (str, dict) pairs; Node uses napi Vec<(String, Value)> tuple marshalling; wasm deserializes a JS array via serde_wasm_bindgen; Ruby adds manual define_method registration and validates [String, Hash] pairs; PHP walks nested arrays. The generated laurus-nodejs/index.d.ts and laurus-wasm/pkg/ are git-ignored, so they are regenerated by each build rather than committed. Runtime-tested: Python 4/4 (pytest), Node 58/58 (vitest, 3 new), Ruby 4/4 (minitest), PHP batch tests (phpunit); wasm builds for wasm32-unknown-unknown and via wasm-pack with both methods in the regenerated .d.ts (build-test only, matching the wasm suite). Closes #866. Refs #551
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes the batch ingestion API (#551 PR-4, the campaign's final PR) in all five language bindings, on top of the merged
Engine::put_documents/add_documents.Each binding gains
put_documents/add_documents(putDocuments/addDocumentsin the camelCase hosts), taking a list of(id, document)pairs — the most direct mirror of the coreVec<(String, Document)>signature and the natural batched form of each binding's existing singularput_document(id, doc). Semantics carry through from the core: sequential in-order application, one WAL fsync per batch, duplicate-id dedup within aputbatch (last wins), chunk accumulation foradd, and fail-fast with the applied prefix left in place (retrying is idempotent). Each binding's converter names the offending position on a malformed entry, and an empty batch is a no-op.put_documents([(id, dict), ...])(str, dict)pairsputDocuments([[id, obj], ...])Vec<(String, Value)>napi tuple marshalling;index.d.tsregeneratedputDocuments([[id, obj], ...])serde_wasm_bindgen;.d.tsregeneratedput_documents([[id, hash], ...])define_methodregistration addedputDocuments([[$id, $doc], ...])The
BatchIngesterror surfaces message-only through each binding's existing error mapper (the message already carriesfailed_index/failed_id/appliedviaDisplay); no new host exception types.Verification (real runtime tests, not just compile)
maturin develop+ pytest: 4/4 (test_batch_ingest.py: empty no-op, apply+dedup, chunk accumulation, malformed-entry position).napi build+ vitest: 58/58 (3 new inbatch_ingest.spec.mjs); regeneratedindex.d.tscarries both methods.rake compile+ minitest: 4/4 (test_batch_ingest.rb, incl.ArgumentErrornamingdocuments[1]).cargo build --release+ phpunit: batch tests green (LaurusTest.php).cargo build --target wasm32-unknown-unknownandwasm-pack build --target websucceed;putDocuments/addDocumentspresent in the regeneratedpkg/*.d.ts. (This binding has no runtime test harness — build-test only, consistent with the rest of the wasm suite.)api_reference.md(en+ja × 5).Generated artifacts (
laurus-nodejs/index.d.ts,laurus-wasm/pkg/) are git-ignored, so they are regenerated by each binding's build rather than committed.Docs
docs/{src,ja/src}/laurus-{python,nodejs,wasm,ruby,php}/api_reference.md:put_documents/add_documentsrows (or headings for wasm) with the pair-list shape and fail-fast/no-rollback semantics.Closes #866. Refs #551 (final PR of the batch-ingestion campaign; the #551 umbrella is closed during wrap-up once PR-3 (#865) and this PR are both merged).