Skip to content

docs product ENTERPRISE_SOURCE_ADMISSION

AlexandreZanata edited this page Oct 26, 2026 · 1 revision

Enterprise source admission register

Status: FROZEN FOR LOCAL CONSTRUCTION; public-release gates remain explicit. Date: 2026-10-24. Task: ESP-01. Owner: enterprise data/static portal plan. Rules: B-BR-ESP-02–09/16.

This register freezes the fields, keys, clocks, coverage, retention, correction windows and quotas used by the ESP export/portal. It does not enable a source, schedule, import, public route or deployment. Unknown rights, schemas or coverage are recorded as unknown, never inferred from a page label or an existing parser.

Rights policy

The ANP site footer declares CC BY-ND 3.0 for site content. That footer is not a dataset license and does not authorize derived public data products. Dataset rights come only from the dataset metadata or an explicit ANP statement:

Source Declared dataset license Public derived release
S-ESP-01 registry none found; metadata PDF has no text layer RIGHTS_PENDING
S-ESP-02 prices none found in the historical/series metadata RIGHTS_PENDING
S-ESP-03 enforcement CC BY, CC0, ODbL or PDDL (metadata menu, no selection) RIGHTS_PENDING (applicable license indeterminate; see re-examination)
S-ESP-04 fines CC BY (metadata) event-level ADMITTED under CC BY attribution; station links only when linked
S-ESP-05 PMQC none found in the October 2026 metadata RIGHTS_PENDING
S-ESP-06 registry API not inspected/admitted NOT_ADMITTED

RIGHTS_PENDING means the profiles may feed local implementation and internal analysis, but the public static release must not publish their rows, station pages, rankings or downloads until the maintainer records a compatible license or written permission. This blocks the price/registry/PMQC slices of ESP-18, not the unrelated official-record slices whose declared license and runtime gates pass. No profile is enabled by this register.

Three representations

  • Original artifact: fetched bytes, URL, media type, byte count and SHA-256, private temporary storage, never under the public root or in Git.
  • Normalized fact: module-owned revision/row with source URL/checksum, parser/profile version and source clock.
  • Derived public output: allowlisted JSON/HTML/CSV generated by ESP from a pinned manifest; no raw document, private field or internal note.

Public field policy (all profiles)

Allowed: public station UUID and full CNPJ (14 digits, leading zeros/letters preserved), public business name/address/municipality/UF, reviewed coordinates only, fuel/product, exact source unit, condition when known, milli-BRL amount, source dates, source URL/checksum, revision/edition identity, and the applicable clock. Excluded everywhere: CPF, private account/owner data, contributor GPS, user photos, internal notes, raw source paths, credentials and unverified coordinates. Unknown values stay missing; they are never zero, average or a nearby station's value.

S-ESP-01 — ANP retailer registry (reuse Directory/RST)

  • Observed 2026-10-24: page 200 text/html, 264,737 bytes; published 2022-01-25, updated 2026-10-23 09:21. CSV link .../arquivos-dados-cadastrais-dos-revendedores-varejistas-de-combustiveis-automotivos/dados-cadastrais-revendedores-varejistas-combustiveis-automoveis.csv. Metadata PDF 159,866 bytes SHA-256 774259f8…ef8c98, created 2022-05-12 by Print-to-PDF, no extractable text.
  • Ingestion: existing Directory full-snapshot reconciliation, exact establishment CNPJ → canonical UUID; Rust prep remains the source preparer.
  • Native key/dedupe: full CNPJ + establishment identity. A missing row never retires a station; absence is not evidence of closure.
  • Public fields: UUID, CNPJ (absent when the active binding is invalidated), display name, address components, IBGE municipality/UF, location_quality, reviewed coordinates, source-reported registry operation_status (active/suspended/revoked), identity revision ID and source anchor date.
  • Clocks: source snapshot update, last successful check, Abastevo publication.
  • Coverage: current national registry; not proof of historical membership or of survey sampling.
  • Quotas/retention: one full CSV per daily reconciliation, ≤100 MB; raw CSV private and removed after accepted publication; URL/checksum/bytes/profile retained. Source refusal/partial snapshot preserves the last accepted state.
  • Correction window: each daily snapshot reconciles the current registry; corrections use Directory supersession and never rewrite history.
  • Status: ingest=EXISTING, public_release=RIGHTS_PENDING.

S-ESP-02 — ANP weekly prices and historical series (reuse Official)

  • Observed 2026-10-24: recent page 200 text/html, 446,356 bytes; updated 2026-10-23 16:29. We use the station detail (revendas_lpc_*.xlsx) and retain the municipal summary (resumo_semanal_lpc_*.xlsx) only as official summary, never as station facts.
  • Current naming gap: the newest listed week uses revendas_lpc_2026_10_11-2026_10_17.xlsx; older weeks use revendas_lpc_2026-10-04_2026-10-10.xlsx. The existing discovery builds only the second shape (%s_%s with ISO dates), so it cannot discover the newest week. ESP-02 must accept the observed variants without inventing a period.
  • Historical series: page 200 text/html, 328,336 bytes; updated 2026-10-20. Files are semester ZIPs such as ca-2026-01.zip, not the single CSV described in docs/data-sources.md. Metadata PDF 191,035 bytes SHA-256 cfe58226…2b8350, updated 2026-03-06, declares CSV format, weekly/monthly/ semester cadence and fields; it declares no license.
  • Ingestion: existing Official XLSX parser, staged revisions, review gate, import jobs and read port. No second parser/import path.
  • Native key/dedupe: import identity = source URL + source SHA-256 + parser version; observations are keyed by station, product, unit and collection date inside one published revision. Supersession retains corrections.
  • Public fields: UUID, CNPJ, product, unit, condition.kind, amount in milli-BRL, currency BRL, collected date, survey start/end, revision ID, supersession reference, source URL/checksum, parser version and Abastevo publication time. Municipal summaries keep their official product/period/unit and are labeled as ANP summary statistics.
  • Excluded: raw workbook text from HTML pages; municipal mean is never attached to a station; no national mean is derived by averaging municipal means.
  • Clocks: collection date, survey week, source page update, last successful check and Abastevo publication are separate. Stored coverage is measured; a new build does not refresh source age.
  • Discovery/validators: scheduled run covers the current and previous survey week; the source is not assumed to expose a reliable ETag, so each file is content-hashed and the hash is the dedupe identity. A 404 means "not published", not "no prices". An operator correction sweep may request up to five recent weeks in the job payload; older corrections use a controlled backfill.
  • Correction window: current and previous week on the schedule; a bounded correction sweep may reach the previous four weeks, then a documented controlled backfill. This is an Abastevo operating policy, not an ANP guarantee.
  • Quotas/retention: scheduled run probes two weeks × two file kinds; a correction sweep is capped at five weeks; raw files private and removed after publication. Backfill is one archive/member per run with byte/row caps.
  • Coverage: station-level survey subset, not every station or every day. Rankings mean "prices found in the identified survey period", with counts, ties and exclusions.
  • Status: ingest=EXISTING, public_release=RIGHTS_PENDING.

S-ESP-03 — ANP enforcement actions (reuse FQ-06)

  • Observed 2026-10-24: page 200 text/html, 264,240 bytes; updated 2026-10-20 12:43; monthly with about two months between action month and publication. Resources: dados-fisc-1998-2018.xlsx and dados-fisc-a-partir-2019.xlsx. Metadata DOCX 17,715 bytes SHA-256 04917ce1…c5066cf declares CC BY/CC0/ODbL/PDDL.
  • Ingestion: FQ-06 Rust adapter and FQ-04b event ledger. The raw ANP XLSX columns are not yet bound to the adapter; the synthetic enforcement-v1 contract is not evidence of the live file. No live Go stream exists.
  • Native key/dedupe: native action key (ACAOID); same-key content changes are retained as correction pairs. Distinct keys against one CNPJ remain distinct; rows are not infractions.
  • Public fields: native key, subject CNPJ, scope, source status verbatim, occurrence date when present, extraction date/as-of, evidence reference and source period. No inferred guilt, interdiction, finality or risk score.
  • Clocks: occurrence date (optional), extraction date, source page update, last check and Abastevo publication. Missing later records prove no outcome.
  • Coverage: two visible files (1998–2018, 2019–2025 label), actual row/field coverage unverified. A page date is not coverage.
  • Quotas/retention: one archive per discovery run, bounded bytes/rows; private raw cleanup; supersession retains corrections. Default-off.
  • Correction window: current published edition plus the two preceding monthly editions, then controlled backfill. No page label creates a period.
  • Status: ingest=ADAPTER_LOCAL, runtime=OWED, public_release=RIGHTS_PENDING. The license cell holds four alternatives joined by "ou" with no selection control; absence of a checkbox alone does not decide between alternative licensing and an unfilled form, so no specific admission is sustained. The slice stays blocked until the live stream, coverage, safe station association AND a determined license are accepted. This doubt changes nothing for the independent fines slice S-ESP-04.

S-ESP-04 — ANP fines (reuse FQ-07)

  • Observed 2026-10-24: page 200 text/html, 265,975 bytes; updated 2026-08-24 11:27; annual extraction snapshots. Metadata PDF 108,518 bytes SHA-256 0b2c4fd2…c0e36b declares CC BY, annual cadence, CSV/XLSX.
  • Verified resources: seven exact files accepted by FQ-07R5E (grouped 2016–2019, annual 2020 and 2022–2025, and the 2021 CSV served as multas-2021-junho22.xlsx). Unsupported years/formats/query/fragments refuse.
  • Ingestion: FQ-07 native preparer, typed bridge, storage, publication and worker. Imports/schedules remain off. CPF is minimized before publication.
  • Native key/dedupe: content observation identity, proceeding reference and party CNPJ; due date is not decision date, extraction month is not an invented day, and a fine number is not a content key. Same-content repeats collapse.
  • Public fields: native key, proceeding reference, party CNPJ, amount (minor units and source text), source status verbatim, decision/due dates when supplied, extraction month/report year/as-of, evidence reference and source contract. No CPF, internal notes or unlabelled fields.
  • Clocks: due/decision dates as supplied, extraction month/year, report year, last check and Abastevo publication. A fine is not a definitive quality finding.
  • Coverage: annual snapshots 2016–2025; one acceptance run reconciled 35,911 source records (35,612 observations, 293 CPF quarantines, 6 metadata). This is not complete station history or current legal status.
  • Quotas/retention: one resource per discovery step, whole-source lease, bounded bytes/time; raw artifacts removed before publication; only hashes and source notes retained. Default-off.
  • Correction window: current annual edition plus the previous edition; corrections supersede, never rewrite. Older edits require a controlled backfill.
  • Status (ESP-11, 2026-10-24): ingest=REAL_STAGING_ACCEPTED for the seven FQ-07 files (35,911 input = 35,612 accepted + 293 CPF quarantines + 6 metadata) through the real FQ-08 read/export path; runtime=DEFAULT_OFF for scheduled imports; public_release=EVENT_LEVEL_ADMITTED under CC BY attribution. The accepted 2025 edition is event-level and all 3,375 records are unmatched (empty acceptance Directory); no station link is invented. Station pages show a record only when association_state=linked; fine association live revalidation/rebound remains OWED. No score, guilt, current interdiction or definitive-finality claim is admitted. FQ-06 remains blocked.

S-ESP-05 — PMQC monitoring (optional portal slice)

  • Observed 2026-10-24: page 200 text/html, 270,945 bytes; updated 2026-10-13 17:19; monthly ZIPs with CSV/JSON members; metadata DOCX 84,545 bytes SHA-256 9eaed144…144e3f (October 2026, full conforming/non-conforming publication). No dataset license field found.
  • Ingestion: FQ-02B native 19-column parser, lossless sample/assay stream, FQ-04 ledger and FQ-08 reads. Monitoring is not enforcement.
  • Native key/dedupe: IdNumeric + Produto identifies a sample; assay rows are separate; multiple assays never inflate the sample count.
  • Public fields: sample/assay reference, collection date/time, fuel, result text, unit, conformity, evidence reference, station CNPJ/municipality/UF when safely associated. Coordinates/address are excluded from the portal until a separate review; no laboratory inference.
  • Clocks: source collection date/time verbatim, source period, last check and Abastevo publication. Empty result is missing evidence, not zero.
  • Coverage: 2016–2025 annual and January–September 2026 links observed; duplicated February/July-style hrefs are resolved by URL/hash, never by label.
  • Quotas/retention: existing ZIP/member caps (≤100 MiB compressed, ≤1 GiB declared aggregate, ≤30 MiB per CSV); one archive per run; private cleanup.
  • Correction window: current month plus the two preceding monthly archives; annual archives for backfill. No invented month.
  • Status: ingest=LOCAL, runtime=DEFAULT_OFF, public_release=RIGHTS_PENDING, slice OPTIONAL.

S-ESP-06 — ANP retailer API (conditional)

  • Status: NOT_ADMITTED. No contract, quota, coordinate quality or uninterrupted-access evidence was inspected. The portal must not call it and must not replace a verified CSV field with an unverified API field. Reconsider only with a measured missing need and a new admission record.

Operational limits shared by the ESP pipeline

  • No visitor request triggers a fetch, import or build. Ingestion/build runs are operator-scheduled or invoked by the owned worker only.
  • Each source run has one bounded slot; general safety/API work keeps its own budget. Existing FQ source leases and default-off switches are reused.
  • Redirects stay on the admitted host/prefix; no query/fragment/port tricks; response bytes, archive entries, member size, rows, time and concurrency are bounded. 429/5xx backs off with jitter; a failed/partial/unknown layout keeps the last accepted publication.
  • Raw source artifacts, temp files and prepared quarantine samples stay outside Git and the public root. Logs contain no CPF, precise contributor position, private document or full source path.
  • Corrections rebuild every affected page, index, ranking and download; withdrawn records cannot return through a rollback to an older release.

Change control

A profile change (URL pattern, schema, key, clock, field allowlist, rights, correction window or quota) requires a new register revision and affected tests. A source whose rights or live contract is unknown stays RIGHTS_PENDING or NOT_ADMITTED; it does not inherit admission from a sibling ANP dataset.

Re-examination 2026-10-25 (no status change)

Primary sources were re-fetched and re-inspected before any publication decision. No compatible new license was found, so every status above stands. Code license, site footer, download availability and derived redistribution rights were kept distinct; the deploy authorization is not a third-party license.

Source Document fetched 2026-10-25 Bytes / SHA-256 License finding
S-ESP-01 registry .../arquivos-dados-cadastrais-dos-revendedores-varejistas-de-combustiveis-automotivos/metadados-revendedores-varejistas-combustiveis-automoveis.pdf 159,866 / 774259f8…ef8c98 (unchanged) No license field; full OCR of the single page (title, description, catalog, org, keywords, language, daily frequency, CSV format, source, contact, column metadata) contains zero license mentions. RIGHTS_PENDING stands.
S-ESP-02 prices old .../arquivos/shpc/metadados-serie-historica-precos-combustiveis.pdf now answers 404 ({"error_type":"NotFound"}); current series page (276,528 bytes) links no dataset metadata; dados.gov.br API answers 401 — Only the generic site footer (CC BY-ND 3.0, not a dataset license). RIGHTS_PENDING stands.
S-ESP-03 enforcement .../arquivos-acoes-de-fiscalizacao/metadados-acoes-fiscalizacao.docx 17,715 / 04917ce1…c5066cf (unchanged) License cell is an unselected menu (CC BY / CC0 / ODbL / PDDL, zero checkbox elements in the XML). Admission reading unchanged with this caveat recorded; runtime stays blocked.
S-ESP-04 fines .../arquivos/mav/metadados-multas-aplicadas-2016a2019.pdf 108,518 / 0b2c4fd2…c0e36b (unchanged) Licença de uso: Creative Commons Atribuição (CC BY), public visibility, annual cadence re-verified. Event-level admission stands.
S-ESP-05 PMQC .../arquivos/pmqc/pmqc-metadados (moved URL, identical bytes) 84,545 / 9eaed144…144e3f (unchanged) All 202 paragraphs scanned: no license field. RIGHTS_PENDING stands.
S-ESP-06 registry API — — NOT_ADMITTED stands.

Raw documents stay outside Git under private /tmp; only bytes, hashes and license-relevant extracts are recorded here.

Clarification request draft (NOT sent — for the maintainer to forward)

Subject: Licença de uso para redistribuição derivada — dados abertos ANP

Para publicação de derivados estáticos com atribuição, solicitamos confirmação escrita, por conjunto abaixo, da licença aplicável à redistribuição de dados derivados (não apenas download):

  1. Série histórica / levantamento semanal de preços de combustíveis (revendas LPC): existe licença de uso do conjunto que autorize derivados públicos? O antigo PDF de metadados da série não é mais servido (404).
  2. Dados cadastrais dos revendedores varejistas: existe licença de uso do conjunto que autorize derivados públicos? O PDF de metadados não declara licença (verificado inclusive por OCR).
  3. PMQC (monitoramento da qualidade): existe licença de uso do conjunto? O DOCX de metadados não declara licença.
  4. Ações de fiscalização: qual das quatro opções do campo "licença de uso" do DOCX de metadados (CC BY, CC0, ODbL ou PDDL) vale para o conjunto? O campo é um menu sem seleção marcada.

Sem resposta, os slices correspondentes permanecem RIGHTS_PENDING e nenhum conteúdo deles será publicado. Nada aqui autoriza contato com terceiros pelo agente.

Alignment 2026-10-25 — S-ESP-03 license indeterminacy

The re-examination proved the "LICENÇA DE USO" cell is a four-alternative menu (CC BY / CC0 / ODbL / PDDL) with zero checkbox or selection controls in the document XML. That byte-level fact cannot decide between "alternative licensing" and "unfilled form". Per the rule that no specific admission stands without foundation, the table and the S-ESP-03 section above now record RIGHTS_PENDING for public derived release instead of the earlier CC BY reading; runtime was and stays blocked. Historical records that quoted the earlier reading are preserved as history, not rewritten. The S-ESP-04 fines slice is independent and unchanged. Gates need no change: neither portalcheck nor the publication preflight ever admitted enforcement pages or slices (only official-sanctions), so no enforcement content could have passed them.

Fuel prices wiki

Source: AlexandreZanata/brazil-fuel-prices @ bbd96568180563ced975b96f8f89f7a3c242522e

Clone this wiki locally