Repository navigation
docs architecture sources national limits
reference for Phase 2+ ingestion and matching workers. This document records the limits we actually hit or verified; it does not invent an official limit where the publisher does not state one.
- Endpoint:
https://revendedoresapi.anp.gov.br/v1/combustivel - Credentials: none. No 401/403 was observed.
- Observed behavior on 2026-10-26: 12 municipality requests at roughly one
request per second succeeded; the 13th (
AC / MANCIO LIMA) returned HTTP 429 on all three attempts with 10/30/90 s backoff. The exact request and STOP record are ininputs/api/STOP.txtandinputs/api/harvest-index.ndjsonon the Phase 1 VPS root. - Published official limit: not found. Treat the limit as unknown and conservative.
- Worker policy:
- one process and one request at a time; never parallelize this API;
- start at one request every 5 s; if a 429 appears, stop the lane rather than increasing the rate;
- retries only with 10/30/90 s backoff, then STOP on repeated 401/403/429;
- cache every raw response by URL and SHA-256, and resume from the saved index;
- never use
station-discoverfor harvesting (it stages rows in the database); Phase 1/2 workers are file-only; - never print or commit credentials; this API needs none.
- Do not retry a stopped lane automatically. A new run needs an owner decision and a fresh observation window.
- 2026-10-27:
uf=<UF>withoutmunicipioreturns the whole UF in pages of 5,000 (AC 193 records; the 60 records of the 9 harvested AC municipalities were byte-identical). The national harvest used 29 requests at one per 5 s: 136 s, 45,807 records, 39,017 valid points, no 429. Prefer the UF query; keep the per-municipality query for targeted refreshes only.
-
brazil-latest.osm.pbfwas 2,098,807,156 bytes on 2026-10-25; OSM timestamp 2026-10-25T21:21:21Z. - Download once per batch, verify the Geofabrik MD5 and our SHA-256, and keep one copy per private root. No credentials.
-
prepare_osmdefault limits: 8 GiB input, 400,000,000 elements, 200,000 matched. The observed Brazil run used 363,382,865 elements, 875,904 KiB peak RSS and 7m14s. Do not raise these limits without a measured need and a new review. - One
prepare_osmworker per PBF; do not re-parse the PBF per municipality.
-
Official per-UF CSV ZIPs under
https://ftp.ibge.gov.br/Cadastro_Nacional_de_Enderecos_para_Fins_Estatisticos/Censo_Demografico_2022/Arquivos_CNEFE/CSV/UF/. -
Pilot sizes: SP 1,082,015,432 B, PR 242,066,031 B, MT 37,772,175 B; data rows were 22,953,725 / 6,122,025 / 1,985,314 (header excluded).
-
Public files, no credentials. Download each UF once, hash it, and stream the CSV; do not load a full UF into memory.
-
Keep only the station municipalities' rows in a derived extract. The raw ZIPs stay in the private root, never in Git.
-
2026-10-27: all 27 UF ZIPs (3.7 GB) downloaded in 86 s for the 24 missing UFs. Use
NV_GEO_COORD1-3 (address level) andCOD_ESPECIE6 (other establishments) only; levels 4-6 are face/locality/sector centroids and species 3/4/5/8 are agro/school/health/religious rows. The national L5 extract kept 124,545 of 111,102,875 rows in 147 s with 5 workers.
- Official landing page:
https://www.gov.br/receitafederal/dados. The current public SERPRO+ share is a monthly folder (2026-10in this batch). - The ten
Estabelecimentos0..9.zippartitions are split by CNPJ root, not by UF, so all ten are needed to cover PR/MT/SP. The 2026-10 batch was 5,416,929,969 bytes and 73,809,386 rows. - Public share, no account; do not log the public share token. Four parallel file downloads completed in this batch; do not increase concurrency without a new observation.
- Cache by SHA-256 and month. Do not re-download a month for every worker.
- Release used:
2026-10-21.0; places theme 11,895,717,186 bytes in 16 Parquet files. In this release only parts 4 and 5 contained Brazil places. - Use DuckDB with a bbox filter and the IBGE UF polygons; extract once to a private GeoParquet. D3 is undecided: Overture is evidence only, never sole proof.
- Do not scan all 16 files per worker when the release layout already prunes to the Brazil parts.
- Local files only: no network, database, clock or native engine.
- Hard bounds: 200,000 API records, 500,000 OSM references, 200,000 registry
rows, 10,000 map assertions (the current
abastevo.map-assertions-v1contract cap), 10,000 review rows and 4 MiBreviews.ndjson(the currentstation-reviewcaps), 64 MiB per API JSON file, 1 GiB OSM/registry file. - Matching: 0.0025 degree grid (about 250 m), 3x3 query, CNPJ and normalized
address indexes, Haversine distance. L1 address radius 50 m; L1 CNPJ review
distance 200 m (the
station-reviewcontract cap); L2 address uses 50 m when an ANP point exists and no distance when it does not. - Ambiguous OSM ties are never resolved by picking one candidate. L3 ANP points remain decisions only while D1 is open; L5 never gets a pin.
- The sample Phase 1 inputs produced 361 L1 reviews, 565 L2 map assertions, 1,574 L3 decisions and 43,188 L5 decisions in 1.12 s wall and 142,644 KiB peak RSS. A 1M-point index test completed in 2.98-8.72 s in release mode with 594,360 KiB peak RSS.
- Local files only: no network, database, clock or native engine.
- Hard bounds: 5,000,000 CNEFE points, 500,000 RFB address rows, 200,000 input decisions, 10,000 map assertions, 2 GiB CNEFE extract, 256 MiB RFB/decisions file, 1 GiB OSM/registry file. The default CNEFE ambiguity radius is 100 m.
- Matching order: OSM CNPJ tag -> L2; unique OSM normalized street+number with equal municipality and compatible name/brand -> L2; exact CNEFE street+number+municipality with establishment evidence -> L4; otherwise L5. RFB replaces a stale registry address only for the same CNPJ. L4 candidates stay unpublished while owner decision D2 is open.
- Real pilot (Londrina/PR, Cuiaba/MT, Ribeirao Preto/SP) on 2026-10-26: 985,846 CNEFE points, 424 RFB addresses and 43,188 A2 L5 inputs produced 186 L4 candidates and 43,002 L5 decisions in 10.12 s / 679,240 KiB peak RSS; 396/424 addresses came from RFB. Deterministic reruns were byte-identical.
- File-only: no database, network, clock or deployment. It reads the A2/B2 outputs and the prepared registry; 256 MiB per file, 200,000 decisions and 10,000 combined map assertions.
- It reconciles one final tier per registry station, requires B2 to cover every
A2 L5 station and forbids B2 from publishing an A2 non-L5 station. It builds
one combined L2 corpus and one
abastevo.map-prepared-v1envelope; the envelope must pass the Gomapcontractvalidation before any staging load. - The apply plan contains
station-review/map-loadcommands with<STAGING_DSN>placeholders only. No credential is read, logged or stored. - 2026-10-26 sample: 45,688 stations, 926 published (2.03%), 1,760 candidates, 43,002 no point; 361 L1 reviews and 565 L2 assertions ready; envelope valid. L3 1,574 (D1) and L4 186 (D2) remain blocked.
Stop and report, do not raise limits or retry blindly, when:
- a source requests credentials or returns repeated 401/403/429;
- a hash, size or row count cannot be recorded;
- an input exceeds the documented bound;
-
prepare_osmexceeds 8 GiB or 400M elements; - the map assertion corpus exceeds 10,000 records;
- a proposed action would write to the database, apply a migration, deploy, or change nginx/k8s/app in a files-only phase.
- Home
- Roadmap
- docs-README
- docs-android-build
- docs-android-design-system
- docs-android-forum
- docs-android-games
- docs-android-map-engine
- docs-android-navigation
- docs-android-ocr-image
- docs-android-ocr-live-framing
- docs-android-premium-area
- docs-android-premium-integration
- docs-android-release
- docs-android-ui-containers
- docs-architecture-android-stack
- docs-architecture-android
- docs-architecture-backend
- docs-architecture-data-sources
- docs-architecture-overview
- docs-architecture-sources-ev-research
- docs-architecture-sources-national-limits
- docs-architecture-sources-national-stations
- docs-architecture-sources-register
- docs-assets-app-tutorial-CAPTURE_CONTRACT
- docs-assets-app-tutorial-FABLE_LANDING_POLISH_PROMPT
- docs-assets-app-tutorial-README
- docs-assets-attribution
- docs-assets-brand-README
- docs-assets-flags-ATTRIBUTION
- docs-assets-fuel-icons-README
- docs-assets-games-fuel-v1-README
- docs-assets-home-icons-README
- docs-assets-landing-fleet-android-20261012-BRIEF
- docs-assets-landing-hero-20261011-BRIEF
- docs-assets-landing-illustrated-20261012-BRIEF
- docs-assets-landing-illustrated-20261012-README
- docs-assets-landing-minimal-20261012-BRIEF
- docs-assets-landing-open-data-20261026-BRIEF
- docs-assets-landing-redesign-20261011-PREPARATION_EVIDENCE
- docs-assets-landing-redesign-20261011-README
- docs-assets-landing-redesign-20261011-REFERENCE_BRIEF
- docs-assets-landing-sync-20261012-BRIEF
- docs-assets-map-icons-README
- docs-backend-anp-ingestion
- docs-backend-api
- docs-backend-community-pricing
- docs-backend-data-model
- docs-backend-domain-model
- docs-backend-driver-private-data
- docs-backend-forum-architecture
- docs-backend-forum-bulletins
- docs-backend-forum-sources
- docs-backend-payments-provider-admission
- docs-backend-payments-provider-register
- docs-backend-photos-capture
- docs-backend-photos-subset-submission
- docs-backend-premium-api
- docs-backend-premium-cloud-protocol
- docs-backend-testing
- docs-backend-toolchain
- docs-brand-app-icons
- docs-brand-identity
- docs-decisions-001-kotlin-compose-stack
- docs-decisions-002-price-import-immutability
- docs-decisions-003-nominatim-reverse-geocode
- docs-decisions-005-modular-monolith-and-postgresql
- docs-decisions-006-official-catalog-and-exact-contracts
- docs-decisions-007-anonymous-proof-of-possession
- docs-decisions-008-privacy-and-retention
- docs-decisions-009-community-consensus-and-projections
- docs-decisions-010-private-object-evidence
- docs-decisions-012-licensing-and-hosted-business
- docs-decisions-015-kotlin-multiplatform-shared-domain
- docs-decisions-016-android-commercial-community-ios-deferred
- docs-decisions-017-station-profile-and-verified-representation
- docs-decisions-021-community-first-contribution
- docs-decisions-022-comaps-derived-native-map-engine
- docs-decisions-023-admin-suite-granted-tiers-contributors
- docs-decisions-025-private-abastevo-empresas-organization-boundary
- docs-decisions-026-multilingual-public-site
- docs-decisions-README
- docs-enterprise-README
- docs-enterprise-architecture
- docs-enterprise-backlog-INDEX
- docs-enterprise-backlog-ep00-governance
- docs-enterprise-backlog-ep01-platform
- docs-enterprise-backlog-ep02-organizations
- docs-enterprise-backlog-ep03-identity-access
- docs-enterprise-backlog-ep04-sync
- docs-enterprise-backlog-ep05-fleet-meters
- docs-enterprise-backlog-ep06-trips
- docs-enterprise-backlog-ep07-gps-navigation
- docs-enterprise-backlog-ep08-fuelings
- docs-enterprise-backlog-ep09-documents
- docs-enterprise-backlog-ep10-expenses-finance
- docs-enterprise-backlog-ep11-maintenance
- docs-enterprise-backlog-ep12-fuel-stock
- docs-enterprise-backlog-ep13-intelligence
- docs-enterprise-backlog-ep14-station-claims
- docs-enterprise-backlog-ep15-station-commerce
- docs-enterprise-backlog-ep16-subscriptions
- docs-enterprise-backlog-ep17-data
- docs-enterprise-backlog-ep18-imports-notifications-integrations
- docs-enterprise-backlog-ep19-back-office
- docs-enterprise-backlog-ep20-release
- docs-enterprise-business-rules
- docs-enterprise-contract
- docs-enterprise-conventions
- docs-enterprise-data-offering
- docs-enterprise-decisions
- docs-enterprise-domain-model
- docs-enterprise-glossary
- docs-enterprise-journeys
- docs-enterprise-modules-back-office
- docs-enterprise-modules-data
- docs-enterprise-modules-documents
- docs-enterprise-modules-expenses-finance
- docs-enterprise-modules-fleet
- docs-enterprise-modules-fuel-stock
- docs-enterprise-modules-fuelings
- docs-enterprise-modules-gps-navigation
- docs-enterprise-modules-identity
- docs-enterprise-modules-imports-notifications-integrations
- docs-enterprise-modules-intelligence
- docs-enterprise-modules-maintenance
- docs-enterprise-modules-organizations
- docs-enterprise-modules-station-claims
- docs-enterprise-modules-station-commerce
- docs-enterprise-modules-subscriptions
- docs-enterprise-modules-trips
- docs-enterprise-offline-sync
- docs-enterprise-permissions
- docs-enterprise-portal-routes
- docs-enterprise-privacy-security
- docs-enterprise-product-vision
- docs-enterprise-source-admission
- docs-enterprise-state-model
- docs-enterprise-testing
- docs-enterprise-ux-app
- docs-enterprise-ux-web
- docs-operations-README
- docs-operations-backup
- docs-operations-deploy-agent
- docs-operations-dou-editions
- docs-operations-edge-cache
- docs-operations-enterprise-activation
- docs-operations-map-static-overlay
- docs-operations-moderation-feedback
- docs-operations-moderation-profiles
- docs-operations-monitoring
- docs-operations-operator-access
- docs-operations-payments
- docs-operations-premium-billing
- docs-operations-recovery
- docs-operations-registry-daily-sync
- docs-operations-registry-runs
- docs-product-README
- docs-product-admin-tiers
- docs-product-community-community-first
- docs-product-community-experience
- docs-product-community-metrics
- docs-product-community-moderation
- docs-product-community-safety
- docs-product-community-terms-pt-br
- docs-product-driver-suite-README
- docs-product-driver-suite-android-state
- docs-product-driver-suite-cloud-sync
- docs-product-driver-suite-cost-profiles-and-maintenance
- docs-product-driver-suite-driver-model-and-earnings
- docs-product-driver-suite-foundation-decisions
- docs-product-driver-suite-gps-lifecycle
- docs-product-driver-suite-mobile-hub
- docs-product-driver-suite-odometer-review
- docs-product-driver-suite-outbox-wire
- docs-product-driver-suite-quick-refuel
- docs-product-driver-suite-reports
- docs-product-driver-suite-shifts-and-distance
- docs-product-driver-suite-signed-commands
- docs-product-ev-README
- docs-product-ev-charging-stop-planner
- docs-product-ev-energy-model
- docs-product-ev-financial-method
- docs-product-ev-journey-ui
- docs-product-ev-preparation
- docs-product-ev-typed-data
- docs-product-exploration
- docs-product-forum-contract
- docs-product-forum-policies
- docs-product-glossary
- docs-product-landing-content
- docs-product-landing-guides
- docs-product-landing-organic-growth-i18n-plan
- docs-product-licensing
- docs-product-map-discovery
- docs-product-map-favorites-and-trips
- docs-product-map-reports-routing-guidance
- docs-product-notification-content
- docs-product-official-prices
- docs-product-open-source-business
- docs-product-overview
- docs-product-payments-README
- docs-product-payments-offer-policy
- docs-product-premium-README
- docs-product-premium-financial-method
- docs-product-premium-source-policy
- docs-product-stations-catalog
- docs-product-stations-fuel-feedback
- docs-product-stations-fuel-quality
- docs-product-stations-identity-and-ev-catalog
- docs-product-stations-profile-contract
- docs-product-stations-profile-representation
- docs-product-stations-registry-sources
- docs-product-vehicle-registration
- docs-product-vehicles-and-occurrences
- docs-security-android-terms-acceptance
- docs-security-free-account-access
- docs-security-identity-profile
- docs-security-media-location-policy
- docs-security-privacy-notice
- docs-security-privacy-policy
- docs-security-retention
- docs-security-risks
- docs-security-security-privacy
- docs-store-README
- docs-store-play-readiness
- docs-store-play-store-listing
- docs-store-ugc-safety
Source: AlexandreZanata/Abastevo @ e4a67580fc368a59bc1c0f1e0466fc011313f330