Skip to content

feat: property tests, e2e ingest→query test, parallel ingest, host-fn log#88

Open
arandomogg wants to merge 1 commit into
Miracle656:mainfrom
arandomogg:ingest
Open

feat: property tests, e2e ingest→query test, parallel ingest, host-fn log#88
arandomogg wants to merge 1 commit into
Miracle656:mainfrom
arandomogg:ingest

Conversation

@arandomogg
Copy link
Copy Markdown

closes #73
closes #74
closes #83
closes #84

Summary

#73 — Property-based decoder tests (tests/decoder.property.test.ts)

Added fast-check as a devDependency. A new tsconfig.test.json (extends
the root config with rootDir: ".") lets Jest compile files under the new
top-level tests/ directory without disturbing the existing build output.
The Jest config gains a second root and a matching testMatch glob.

The property suite runs 10 000 arbitrary inputs through parseEvent:

  • A comprehensive property that combines random event-type symbols with
    random i128 amounts and asserts that only null, a TransferRecord, or
    a proper Error is produced — never an unhandled crash.
  • A focused property for every i128 edge case: zero, −1 (max-u128 bit
    pattern as signed i128), i128 min/max, i64 boundary values.
  • A property for arbitrary unknown-symbol events (should return null).
  • Backward-compatibility fixture check: all four existing XDR fixtures
    (transfer, mint, burn, clawback) still decode to the expected type.

#74 — End-to-end ingest → query test (tests/integration/e2e.test.ts)

The suite is skipped automatically when DATABASE_URL is absent (unit CI
pass unaffected). When a database is available (local docker-compose or CI
service):

  1. A pre-recorded SEP-41 transfer fixture is inserted via Prisma directly
    (simulating what the indexer writes after decoding an RPC event).
  2. Three endpoints are exercised — GET /transfers/incoming/:address,
    GET /transfers/outgoing/:address, and GET /transfers/tx/:txHash
    each asserting the fixture row is present with the correct fields.
  3. The inserted row is cleaned up in afterAll so the test is idempotent.

The fixture file (tests/integration/fixtures/horizon-event.json) is 395
bytes.

#83 — Parallel partition ingest (src/indexer/parallel.ts)

partitionByContract(contractIds, n) deterministically shards contract IDs
into n buckets by char-code hash so the same contract always lands in the
same partition across poll cycles.

pollParallel fans out one fetchEventsSafe + parseEvents + upsertTransfers
cycle per partition via Promise.all, so N partitions run their RPC calls
and DB writes concurrently instead of sequentially — yielding roughly N×
throughput with no ordering violations within a partition.

Activated in startIndexer when INGEST_WORKERS > 1 and more than one
contract is being watched. Single-contract or default deployments continue
to use the existing sequential pollOnce path unchanged.

#84 — InvokeHostFn log (src/indexer/host-fn-log.ts, prisma/schema.prisma, src/api.ts)

Added a new HostFnLog Prisma model (@@schema("wraith")) with fields for
functionName, args (JSON), result (JSON?), gasUsed (BigInt?,
nullable — populated externally when transaction metadata is available),
ledger, ledgerClosedAt, txHash, and a unique eventId.

parseHostFnEvent converts any RawEvent into a HostFnRecord by decoding
topics[0] as the function name and serialising the remaining topics and
value via scValToNative. BigInt fields are stringified so the JSON column
stores valid JSON on all PostgreSQL versions. An xdr fallback is used for
unknown future ScVal types.

pollOnce now calls upsertHostFnLogs after the existing token-transfer
path, wrapped in a .catch so a log failure never interrupts indexing.

GET /host-fn/:contractId (optional ?functionName=, ?limit=, ?offset=)
is wired into the Express app before the existing 404 handler, following the
same pattern as the other transfer endpoints.

Migration note

prisma db push / prisma migrate dev is required to create the new
HostFnLog table. The startup migration in src/index.ts already runs
prisma db push, so no manual step is needed for docker-compose deployments.

@drips-wave
Copy link
Copy Markdown

drips-wave Bot commented May 30, 2026

@arandomogg Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Copy link
Copy Markdown
Owner

@Miracle656 Miracle656 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four issues in one well-organized PR. Each component is defensively built:

  • #73 — Property-based decoder tests: 10 000 fast-check runs with i128 edge coverage (zero, −1 as max-u128 bit pattern, i128 min/max, i64 boundaries), plus a backward-compat fixture check across all four XDR fixtures (transfer/mint/burn/clawback). The tsconfig.test.json + dual roots/testMatch arrangement keeps tests under tests/ without polluting the build output. Right pattern.

  • #74 — E2E ingest → query: Auto-skips when DATABASE_URL is absent — unit CI stays green. When DB is present, inserts a fixture via Prisma, exercises three endpoints (incoming, outgoing, tx/:txHash), cleans up in afterAll. Idempotent.

  • #83 — Parallel ingest: partitionByContract shards deterministically by char-code hash so the same contract always lands in the same partition (preserves intra-partition ordering). pollParallel runs fetchEventsSafe + parseEvents + upsertTransfers per partition via Promise.all. The guard INGEST_WORKERS > 1 && SAC_CONTRACT_IDS.length > 1 keeps single-contract / default deployments on the existing sequential path — zero regression risk.

  • #84 — Host-fn log: HostFnLog Prisma model with the right indexes (contractId, (contractId, functionName), ledger, txHash). parseHostFnEvent serialises topics[1..] and value via scValToNative + toJsonSafe so BigInts become decimal strings (valid JSON on every Postgres version). XDR base64 fallback for unknown ScVal types means no data loss on future Soroban additions. The pollOnce integration wraps the log write in .catch so a log failure never interrupts indexing — good defensive choice. GET /host-fn/:contractId with functionName/limit/offset filters and hard cap of 200 follows the existing route patterns.

Migration note is correct — prisma db push runs at startup so docker-compose deploys auto-apply.

Merging. Closes #73, #74, #83, #84.

@Miracle656
Copy link
Copy Markdown
Owner

Approved. Merge failed because #87 just landed right before this and both PRs added new models to prisma/schema.prisma, creating a textual conflict (your HostFnLog block and the new NftTransfer + NftMetadata blocks from #87 both wanted to insert into the same region of the file).

Quick rebase:

git fetch origin
git rebase origin/main
# Conflict will be in prisma/schema.prisma — just keep both blocks
# (HostFnLog from your branch + NftTransfer/NftMetadata from main),
# then continue.
git add prisma/schema.prisma
git rebase --continue
git push --force-with-lease

Once pushed, I'll merge immediately — no further review needed, the approval stays.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants