docs planning MAP_NATIONAL_SOURCE_LIMITS

National source limits for bots and workers

Status: reference for Phase 2+ ingestion and matching workers. Observed on 2026-10-26. This document records the limits we actually hit or verified; it does not invent an official limit where the publisher does not state one.

ANP Revendedores API

• Endpoint: https://revendedoresapi.anp.gov.br/v1/combustivel
• Credentials: none. No 401/403 was observed.
• Observed behavior on 2026-10-26: 12 municipality requests at roughly one request per second succeeded; the 13th (AC / MANCIO LIMA) returned HTTP 429 on all three attempts with 10/30/90 s backoff. The exact request and STOP record are in inputs/api/STOP.txt and inputs/api/harvest-index.ndjson on the Phase 1 VPS root.
• Published official limit: not found. Treat the limit as unknown and conservative.
• Worker policy:
  • one process and one request at a time; never parallelize this API;
  • start at one request every 5 s; if a 429 appears, stop the lane rather than increasing the rate;
  • retries only with 10/30/90 s backoff, then STOP on repeated 401/403/429;
  • cache every raw response by URL and SHA-256, and resume from the saved index;
  • never use station-discover for harvesting (it stages rows in the database); Phase 1/2 workers are file-only;
  • never print or commit credentials; this API needs none.
• Do not retry a stopped lane automatically. A new run needs an owner decision and a fresh observation window.
• 2026-10-27: uf=<UF> without municipio returns the whole UF in pages of 5,000 (AC 193 records; the 60 records of the 9 harvested AC municipalities were byte-identical). The national harvest used 29 requests at one per 5 s: 136 s, 45,807 records, 39,017 valid points, no 429. Prefer the UF query; keep the per-municipality query for targeted refreshes only.

OSM / Geofabrik

• brazil-latest.osm.pbf was 2,098,807,156 bytes on 2026-10-25; OSM timestamp 2026-10-25T21:21:21Z.
• Download once per batch, verify the Geofabrik MD5 and our SHA-256, and keep one copy per private root. No credentials.
• prepare_osm default limits: 8 GiB input, 400,000,000 elements, 200,000 matched. The observed Brazil run used 363,382,865 elements, 875,904 KiB peak RSS and 7m14s. Do not raise these limits without a measured need and a new review.
• One prepare_osm worker per PBF; do not re-parse the PBF per municipality.

IBGE CNEFE 2022

• Official per-UF CSV ZIPs under https://ftp.ibge.gov.br/Cadastro_Nacional_de_Enderecos_para_Fins_Estatisticos/Censo_Demografico_2022/Arquivos_CNEFE/CSV/UF/.

• Pilot sizes: SP 1,082,015,432 B, PR 242,066,031 B, MT 37,772,175 B; data rows were 22,953,725 / 6,122,025 / 1,985,314 (header excluded).

• Public files, no credentials. Download each UF once, hash it, and stream the CSV; do not load a full UF into memory.

• Keep only the station municipalities' rows in a derived extract. The raw ZIPs stay in the private root, never in Git.

• 2026-10-27: all 27 UF ZIPs (3.7 GB) downloaded in 86 s for the 24 missing UFs. Use NV_GEO_COORD 1-3 (address level) and COD_ESPECIE 6 (other establishments) only; levels 4-6 are face/locality/sector centroids and species 3/4/5/8 are agro/school/health/religious rows. The national L5 extract kept 124,545 of 111,102,875 rows in 147 s with 5 workers.

Receita Federal CNPJ - Estabelecimentos

• Official landing page: https://www.gov.br/receitafederal/dados. The current public SERPRO+ share is a monthly folder (2026-10 in this batch).
• The ten Estabelecimentos0..9.zip partitions are split by CNPJ root, not by UF, so all ten are needed to cover PR/MT/SP. The 2026-10 batch was 5,416,929,969 bytes and 73,809,386 rows.
• Public share, no account; do not log the public share token. Four parallel file downloads completed in this batch; do not increase concurrency without a new observation.
• Cache by SHA-256 and month. Do not re-download a month for every worker.

Overture Places

• Release used: 2026-10-21.0; places theme 11,895,717,186 bytes in 16 Parquet files. In this release only parts 4 and 5 contained Brazil places.
• Use DuckDB with a bbox filter and the IBGE UF polygons; extract once to a private GeoParquet. D3 is undecided: Overture is evidence only, never sole proof.
• Do not scan all 16 files per worker when the release layout already prunes to the Brazil parts.

National matcher (prepare_national_matches)

• Local files only: no network, database, clock or native engine.
• Hard bounds: 200,000 API records, 500,000 OSM references, 200,000 registry rows, 10,000 map assertions (the current abastevo.map-assertions-v1 contract cap), 10,000 review rows and 4 MiB reviews.ndjson (the current station-review caps), 64 MiB per API JSON file, 1 GiB OSM/registry file.
• Matching: 0.0025 degree grid (about 250 m), 3x3 query, CNPJ and normalized address indexes, Haversine distance. L1 address radius 50 m; L1 CNPJ review distance 200 m (the station-review contract cap); L2 address uses 50 m when an ANP point exists and no distance when it does not.
• Ambiguous OSM ties are never resolved by picking one candidate. L3 ANP points remain decisions only while D1 is open; L5 never gets a pin.
• The sample Phase 1 inputs produced 361 L1 reviews, 565 L2 map assertions, 1,574 L3 decisions and 43,188 L5 decisions in 1.12 s wall and 142,644 KiB peak RSS. A 1M-point index test completed in 2.98-8.72 s in release mode with 594,360 KiB peak RSS.

Open geocoder (prepare_open_geocode)

• Local files only: no network, database, clock or native engine.
• Hard bounds: 5,000,000 CNEFE points, 500,000 RFB address rows, 200,000 input decisions, 10,000 map assertions, 2 GiB CNEFE extract, 256 MiB RFB/decisions file, 1 GiB OSM/registry file. The default CNEFE ambiguity radius is 100 m.
• Matching order: OSM CNPJ tag -> L2; unique OSM normalized street+number with equal municipality and compatible name/brand -> L2; exact CNEFE street+number+municipality with establishment evidence -> L4; otherwise L5. RFB replaces a stale registry address only for the same CNPJ. L4 candidates stay unpublished while owner decision D2 is open.
• Real pilot (Londrina/PR, Cuiaba/MT, Ribeirao Preto/SP) on 2026-10-26: 985,846 CNEFE points, 424 RFB addresses and 43,188 A2 L5 inputs produced 186 L4 candidates and 43,002 L5 decisions in 10.12 s / 679,240 KiB peak RSS; 396/424 addresses came from RFB. Deterministic reruns were byte-identical.

Coverage and apply preparation (prepare_national_coverage)

• File-only: no database, network, clock or deployment. It reads the A2/B2 outputs and the prepared registry; 256 MiB per file, 200,000 decisions and 10,000 combined map assertions.
• It reconciles one final tier per registry station, requires B2 to cover every A2 L5 station and forbids B2 from publishing an A2 non-L5 station. It builds one combined L2 corpus and one abastevo.map-prepared-v1 envelope; the envelope must pass the Go mapcontract validation before any staging load.
• The apply plan contains station-review/map-load commands with <STAGING_DSN> placeholders only. No credential is read, logged or stored.
• 2026-10-26 sample: 45,688 stations, 926 published (2.03%), 1,760 candidates, 43,002 no point; 361 L1 reviews and 565 L2 assertions ready; envelope valid. L3 1,574 (D1) and L4 186 (D2) remain blocked.

Stop conditions for any worker

Stop and report, do not raise limits or retry blindly, when:

• a source requests credentials or returns repeated 401/403/429;
• a hash, size or row count cannot be recorded;
• an input exceeds the documented bound;
• prepare_osm exceeds 8 GiB or 400M elements;
• the map assertion corpus exceeds 10,000 records;
• a proposed action would write to the database, apply a migration, deploy, or change nginx/k8s/app in a files-only phase.