ETH-transactions-storage: optional address filter, lighter indexes, and a safer public API

[adamant-al]

ETH-transactions-storage is a self-hosted Ethereum indexer: it follows an Ethereum node, stores native ETH and ERC-20 transfer(address,uint256) activity in PostgreSQL, and exposes it as a read-only REST API through PostgREST. Ethereum nodes do not answer “show me this address’s history,” so wallets, dapps, treasuries, and operators usually depend on a third-party explorer. This project is the alternative you run yourself — no API key vendor, no tracking, no telemetry.

That stack now has an optional address filter, merged to dev in PR #29. Full-chain indexing remains the default (and is what ADAMANT wallets use). Filtered mode is for everyone who only needs a known set of addresses and does not want to store the rest of the chain.

Why this exists

A public Ethereum indexer is a large database. On a one-year mainnet dataset (~490M rows) the old full index set was hundreds of gigabytes. Many operators do not need that:

• A wallet or custodial backend that only serves its own users
• A project treasury that watches a handful of operational addresses
• A self-hosted explorer for an app-specific address set
• A lab or CI environment that should stay small

The filter keeps the same API contract. Clients still query /ethtxs, /max_block, and /aval. You change what is stored, not how it is read.

Address filter

Disabled by default (ADDRESS_FILTER_ENABLED=false). Enable it and point ADDRESS_FILTER_FILE at a private list (default filter/addresses.txt, gitignored, not copied into the Docker image).

Behavior:

• One 0x-prefixed 40-hex-character address per line; blank lines and # comments are ignored
• Matching is case-insensitive
• Native transfers match txfrom or txto
• Supported ERC-20 transfer(address,uint256) calls match the sender (txfrom), the token contract (txto), and the ABI-encoded recipient (contract_to)
• The list is reloaded before every synchronization pass, so valid additions and removals take effect without restarting the indexer
• Invalid, empty, or missing lists fail closed: indexing stops until the file is corrected, rather than silently storing everything
• Receipt RPC is skipped for transactions the filter rejects

What it does not do (existing indexer limits, unchanged):

• Internal ETH transfers
• ERC-20 flows that are not a direct transfer(address,uint256) (transferFrom, routers, multisigs, batch/aggregator calls)
• Automatic historical backfill when you add an address

Enabling the filter does not delete already stored rows. Rebuild is a manual operator step: stop the indexer, truncate both ethtxs and sync_state in one transaction (or rewind both to block N), set START_BLOCK, restart. Truncating only ethtxs will not rescan — the checkpoint still says the chain is done.

Durable sync progress

Filtered and empty blocks used to look like “nothing happened,” so the indexer could rescan them. dev now keeps a single-row public.sync_state checkpoint, updated in the same PostgreSQL transaction as the inserts for that block.

• /max_block still returns { max, version }; max is GREATEST(MAX(ethtxs.block), sync_state.last_block)
• Startup still rewinds the last processed block, now atomically with the checkpoint
• create_tables.sql is idempotent: it creates sync_state, initializes it from the existing highest transaction block, and grants DML to api_user / app_user when those roles exist
• web_anon cannot read or write sync_state directly

Also on dev: indexes, API hardening, operations

The address filter sits on top of the other dev work from PR #28 (not yet a GitHub Release; latest tag remains v2.4.1):

• Minimal 5-index set covering ADAMANT Web and iOS query shapes, saving roughly 90–110 GB per 1-year dataset versus the legacy 8-index set
• web_anon: PostgREST anonymous role is SELECT-only on ethtxs, aval, and max_block
• db-max-rows = 10000: caps serialized result size so an unbounded GET /ethtxs cannot OOM the API
• nginx guards for public deployments: method allow-list (GET/HEAD/OPTIONS), require txfrom or txto on /ethtxs, reject Prefer: count=exact and huge offsets
• .env workflow: documented template, secrets stay out of Git, Compose no longer ships a hardcoded database password
• Safer DB diagnostics: connection URIs work correctly; passwords are redacted from logs
• AGENTS.md: contributor and operator contract for the repo

Existing systemd hosts should keep their current unit during the code/schema upgrade. Apply create_tables.sql with ON_ERROR_STOP before starting the new indexer. Do not copy the repository ethsync.service until a production .env with equivalent values exists.

Who this is for

ADAMANT uses this indexer so adamant-im and adamant-iOS can show Ethereum and ERC-20 history without a centralized explorer. The same binary is a general-purpose open-source service: wallets, payment processors, token issuers, and anyone who wants address-indexed Ethereum history under their own PostgreSQL and their own access policy.

Self-host it, keep full-chain mode for a public API, or turn the filter on and store only the addresses you actually serve.

Links

• Repository
• Address filter PR
• Indexes, API hardening, and query bounds
• Original address-filter request
• Setup and upgrade notes

Questions and operator reports welcome below. Historical rescan/backfill is out of scope for this change; if you need it, say so in the thread.