feat: property tests, e2e ingest→query test, parallel ingest, host-fn log#88
feat: property tests, e2e ingest→query test, parallel ingest, host-fn log#88arandomogg wants to merge 1 commit into
Conversation
|
@arandomogg Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
There was a problem hiding this comment.
Four issues in one well-organized PR. Each component is defensively built:
-
#73 — Property-based decoder tests: 10 000 fast-check runs with i128 edge coverage (zero, −1 as max-u128 bit pattern, i128 min/max, i64 boundaries), plus a backward-compat fixture check across all four XDR fixtures (
transfer/mint/burn/clawback). Thetsconfig.test.json+ dualroots/testMatcharrangement keeps tests undertests/without polluting the build output. Right pattern. -
#74 — E2E ingest → query: Auto-skips when
DATABASE_URLis absent — unit CI stays green. When DB is present, inserts a fixture via Prisma, exercises three endpoints (incoming,outgoing,tx/:txHash), cleans up inafterAll. Idempotent. -
#83 — Parallel ingest:
partitionByContractshards deterministically by char-code hash so the same contract always lands in the same partition (preserves intra-partition ordering).pollParallelrunsfetchEventsSafe + parseEvents + upsertTransfersper partition viaPromise.all. The guardINGEST_WORKERS > 1 && SAC_CONTRACT_IDS.length > 1keeps single-contract / default deployments on the existing sequential path — zero regression risk. -
#84 — Host-fn log:
HostFnLogPrisma model with the right indexes (contractId,(contractId, functionName),ledger,txHash).parseHostFnEventserialisestopics[1..]andvalueviascValToNative+toJsonSafeso BigInts become decimal strings (valid JSON on every Postgres version). XDR base64 fallback for unknown ScVal types means no data loss on future Soroban additions. ThepollOnceintegration wraps the log write in.catchso a log failure never interrupts indexing — good defensive choice.GET /host-fn/:contractIdwithfunctionName/limit/offsetfilters and hard cap of 200 follows the existing route patterns.
Migration note is correct — prisma db push runs at startup so docker-compose deploys auto-apply.
|
Approved. Merge failed because #87 just landed right before this and both PRs added new models to Quick rebase: git fetch origin
git rebase origin/main
# Conflict will be in prisma/schema.prisma — just keep both blocks
# (HostFnLog from your branch + NftTransfer/NftMetadata from main),
# then continue.
git add prisma/schema.prisma
git rebase --continue
git push --force-with-leaseOnce pushed, I'll merge immediately — no further review needed, the approval stays. |
closes #73
closes #74
closes #83
closes #84
Summary
#73 — Property-based decoder tests (
tests/decoder.property.test.ts)Added
fast-checkas a devDependency. A newtsconfig.test.json(extendsthe root config with
rootDir: ".") lets Jest compile files under the newtop-level
tests/directory without disturbing the existing build output.The Jest config gains a second root and a matching
testMatchglob.The property suite runs 10 000 arbitrary inputs through
parseEvent:random i128 amounts and asserts that only
null, aTransferRecord, ora proper
Erroris produced — never an unhandled crash.pattern as signed i128), i128 min/max, i64 boundary values.
null).(
transfer,mint,burn,clawback) still decode to the expected type.#74 — End-to-end ingest → query test (
tests/integration/e2e.test.ts)The suite is skipped automatically when
DATABASE_URLis absent (unit CIpass unaffected). When a database is available (local docker-compose or CI
service):
(simulating what the indexer writes after decoding an RPC event).
GET /transfers/incoming/:address,GET /transfers/outgoing/:address, andGET /transfers/tx/:txHash—each asserting the fixture row is present with the correct fields.
afterAllso the test is idempotent.The fixture file (
tests/integration/fixtures/horizon-event.json) is 395bytes.
#83 — Parallel partition ingest (
src/indexer/parallel.ts)partitionByContract(contractIds, n)deterministically shards contract IDsinto
nbuckets by char-code hash so the same contract always lands in thesame partition across poll cycles.
pollParallelfans out onefetchEventsSafe + parseEvents + upsertTransferscycle per partition via
Promise.all, so N partitions run their RPC callsand DB writes concurrently instead of sequentially — yielding roughly N×
throughput with no ordering violations within a partition.
Activated in
startIndexerwhenINGEST_WORKERS > 1and more than onecontract is being watched. Single-contract or default deployments continue
to use the existing sequential
pollOncepath unchanged.#84 — InvokeHostFn log (
src/indexer/host-fn-log.ts,prisma/schema.prisma,src/api.ts)Added a new
HostFnLogPrisma model (@@schema("wraith")) with fields forfunctionName,args(JSON),result(JSON?),gasUsed(BigInt?,nullable — populated externally when transaction metadata is available),
ledger,ledgerClosedAt,txHash, and a uniqueeventId.parseHostFnEventconverts anyRawEventinto aHostFnRecordby decodingtopics[0]as the function name and serialising the remaining topics andvalue via
scValToNative. BigInt fields are stringified so the JSON columnstores valid JSON on all PostgreSQL versions. An
xdrfallback is used forunknown future ScVal types.
pollOncenow callsupsertHostFnLogsafter the existing token-transferpath, wrapped in a
.catchso a log failure never interrupts indexing.GET /host-fn/:contractId(optional?functionName=,?limit=,?offset=)is wired into the Express app before the existing 404 handler, following the
same pattern as the other transfer endpoints.
Migration note
prisma db push/prisma migrate devis required to create the newHostFnLogtable. The startup migration insrc/index.tsalready runsprisma db push, so no manual step is needed for docker-compose deployments.