v0.20.1 — free-text deadline fix (fino al)

[0.20.1] — 2026-06-17 — free-text deadline fix: "… fino al <date>"

Fixed

• Expired bandi were shown as open. wordpress._parse_scadenza (the Lazio/
  Sicilia adapters' deadline parser) only accepted a date introduced by
  scadenza/termine/entro/presentazione/chiusura. The dominant Lazio window form is
  "… dalle ore 12 del <start> e fino alle ore 17 del <end>", where the closing date
  is introduced by "fino al/alle/ad" — not in the key list. So those bandi were
  mapped deadline=None → status="open" and surfaced as live long after they had
  closed. Added "fino a" to _DEADLINE_KEYS. Measured on the bundled data: the
  fix correctly closes ~20 expired Lazio/Sicilia items (e.g. "Voucher Digitalizzazione
  PMI 2025", window closed 14 Feb 2025), with zero recall risk — none of the
  newly-closed items is labelled relevant/borderline by any profile.
• README Quickstart no longer showcases an expired match. mayai --sample had
  documented "Voucher Digitalizzazione PMI 2025" (deadline —) as match #2; it is now
  correctly withheld (closed), so the Quickstart shows 2 genuinely-open matches. The
  pinned reproducibility test asserts the expired bando is dropped.

Notes

• The eval corpus was deliberately left untouched. Its frozen Lazio slice (60 rows)
  predates the fix and the committed fixtures cover only 15 of them, so a faithful
  refresh of the other 45 would need the full source payloads (not available offline);
  recomputing from the corpus's truncated stored text was verified to mis-date some
  near-boundary rows, so it was rejected. Leaving the stale-open items in the corpus
  is the conservative choice — they are extra negatives that make the matcher's job
  harder, never easier.

v0.20.0 — useful@k metric + __version__ fix

[0.20.0] — 2026-06-16 — useful@k metric (honest reading) + __version__ fix

Added

• useful@k eval metric — the lived-experience companion to strict precision@k.
  Strict P@k counts ONLY relevant; useful@k also counts borderline (adjacent,
  worth-a-look) items, which a user genuinely finds useful. Surfaced as the USE@5
  column in bandiradar eval and in the README. Why it matters: on the
  human-reviewed gold the LLM's strict P@5 is 0.39 but useful@5 is 0.65; and 3 of
  the 11 profiles have ZERO relevant items in the corpus (only borderline), so their
  strict P@5 is structurally 0 and drags the aggregate. On the 8 profiles where the
  corpus holds relevant bandi, the LLM scores strict P@5 0.54 / useful@5 0.83 —
  i.e. where the question is well-posed the matcher is good; the low aggregate is
  mostly corpus coverage, not a broken ranker (recall is fine — 55 of 60 wanted items
  land in the top-10). README "Matching quality" rewritten to show this decomposition.

Changed

• Embeddings semantic prefilter — re-measured on the human-reviewed gold; verdict
  corrected. The old docs called embeddings "net-negative"; that was on the
  permissive pre-review gold. On the corrected gold, semantic ≥ 0.4 lifts recall
  0.89 → 0.95 (rescuing 3 of the 4 lexical prefilter-drops) at a flat FPR
  (0.52 → 0.53) for a 1.6× larger candidate set (150 → 240) — a REAL recall gain.
  It still ships optional and off by default, because the 1.6× candidate set means
  ~1.6× more LLM scorings per run (cost) and 2 of the 4 rescued drops are correctly-weak
  borderline items. Docs (README Honest limits, CLAUDE.md) updated from "net-negative"
  to the honest trade-off; 0.3 over-expands (2.4×), 0.5 barely helps. No code/default
  change — measurement + documentation only.

Fixed

• bandiradar.__version__ was hardcoded to "0.12.0" and had drifted from the
  package version for seven releases — it also rode into the HTTP User-Agent
  (bandiradar/0.12.0). It now derives from the installed package metadata
  (importlib.metadata.version), so it can never drift from pyproject again; a
  regression test pins __version__ to the pyproject version.

Security

• Bumped three transitive dependencies to clear newly-disclosed advisories so the
  pip-audit CI gate stays green: cryptography 48.0.0 → 49.0.0
  (GHSA-537c-gmf6-5ccf), python-multipart 0.0.30 → 0.0.32 (CVE-2026-53540), and
  starlette 1.2.1 → 1.3.1 (CVE-2026-54282, CVE-2026-54283). Lockfile-only; no
  source or default-behaviour change. Full test suite + mypy + bandit re-run green.

v0.19.0 — type checking: mypy clean + CI gate

[0.19.0] — 2026-06-15 — Type checking: mypy clean + CI gate

Added

• mypy gate in CI (uv run mypy, config in [tool.mypy]). The codebase is now
  type-clean (0 errors) — the 80 real issues mypy found were FIXED, not silenced:
  • source classes now declare kind: Kind (was bare str) so they satisfy the
    Source protocol;
  • the recipe self-heal paths narrow recipe/hrecipe with asserts (the invariant
    is real — those paths run only when a recipe is set);
  • default_status keeps now typed (the _ensure_utc(now) or ... form);
  • core returns the proper FetchStatus/FetchErrorKind literals;
  • with_retry accepts Callable[..., Response] so the loop-capture idiom
    lambda x=x: … type-checks;
  • the LLM-extracted kind simplifies to "tender" if … else "incentive" (no Any);
  • plus a handful of small Optional-narrowing fixes. Only two justified
    # type: ignore remain (httpx's overloaded stream(**kwargs) and the header dict).
• mypy added to the dev group.

Notes

• No runtime behaviour change (asserts are invariant guards; the kind and status
  logic are equivalent). This completes the CI-hardening chapter the external review
  asked for (lint + format + tests + coverage + security + audit + types).

v0.18.0 — CI hardening: coverage floor + security gates

[0.18.0] — 2026-06-15 — CI hardening: coverage floor + security gates

Added

• Coverage gate — CI now runs pytest --cov --cov-fail-under=88. Measured
  src-only coverage is 90% (the gap is live/network paths that can't run offline);
  the build fails if it regresses below 88%.
• bandit security scan in CI. Every existing finding was REVIEWED, not blanket-
  ignored: the 4 false positives (RSS XML is built not parsed, X-Relay-Token is a
  header name not a secret, the storage SQL is parameterized via ?, a deliberate
  skip-bad-PDF-page) carry an inline justification + a scoped # nosec; B101
  (assert) is skipped project-wide in [tool.bandit]. A NEW unreviewed finding (e.g.
  unparameterized SQL) fails the build.
• pip-audit dependency audit in CI (PyPA advisory DB) — currently clean.
• Dev tooling (pytest-cov, bandit, pip-audit) added to the dev group; coverage
  • bandit config live in pyproject.toml.

Notes

• This is a CI / dev-tooling change: no runtime behaviour change (only inline # nosec
  comments touch shipped source). mypy is the next, separate hardening step (80 real
  type issues to fix first — a green gate, not a red one).

v0.17.1 — revert TED no-deadline=>closed (hid open calls)

[0.17.1] — 2026-06-15 — Revert the TED "no deadline ⇒ closed" rule (it hid open calls)

Fixed

• Reverted the 0.17.0 TED status rule. It was wrong and harmful. The premise ("a
  biddable TED call always states a deadline") does not hold: the TED search API,
  as read by the adapter (only deadline-receipt-tender-date-lot), returns no
  deadline for many genuine open calls. Measured on the corpus: of the 41
  deadline-less TED notices, only 2 are awards/results — 32 are open calls
  ("PROCEDURA APERTA", forniture, gare). So "no deadline ⇒ closed" was hiding ~32
  live, biddable tenders — the worst error for an opportunities radar (a false
  negative costs the user a real chance). A missing deadline once again defaults to
  open (recall-safe). The eval corpus's TED statuses were restored; the
  human-reviewed gold and the (unchanged) matching numbers from 0.17.0 stand — the
  rule never affected the eval (TED tenders are dropped at Stage 1 by seeks/keyword
  for these profiles anyway), only production.
• Lesson / real root cause: the actual "closed-shown-as-open" items are the
  Lazio past-edition bandi (no structured deadline in the WP body), not TED. The
  proper TED improvement is to capture the deadline that exists on the notice
  (read beyond the lot-level field) — which needs live TED data to validate, so it is
  deferred rather than guessed.

v0.17.0 — human-reviewed gold + honest matching numbers; TED status fix

[0.17.0] — 2026-06-14 — Human-reviewed gold + honest matching numbers; TED status fix

Changed

• The eval gold is now HUMAN-REVIEWED (was auto-proposed + rule-corrected). Every
  label was checked by hand: expired/closed windows → not, out-of-region items →
  not, over-generous relevant → borderline, plus a few real false positives
  removed and two active borderline positives added where a profile had none. The
  _meta no longer says "pending review".

• Matching numbers re-measured on the strict gold — and they are LOWER, which is the
  point. The earlier auto-proposed gold overstated precision (it counted
  expired/out-of-region items as relevant). Honest figures now (LLM pointwise Haiku):

  min_score	P@5	P@10	recall	FPR
  40 (precision)	0.39	0.34	0.61	0.06
  20 (balanced)	0.35	0.24	0.86	0.21

  (was 0.70 / 0.66 / 0.57 / 0.02 on the lenient gold). The matcher is a low-FPR,
  recall-oriented engine, ~2× the heuristic's precision (heuristic P@5 0.22), with
  wide per-profile variance (costruzioni/sardegna_impresa P@5 1.00; small-pool
  profiles 0.00). README's "Matching quality" section is rewritten to these.

Fixed

• TED: a notice with no submission deadline now maps to closed (not the global
  optimistic "open"). A genuine above-threshold biddable call always states a
  deadline; a TED notice without one is almost always an award/result (already
  closed) or a prior-information notice (not yet biddable) — neither belongs in an
  "open calls" radar. Rule lives in ted.to_opportunities (fixes production live
  too); the eval corpus's 41 deadline-less TED rows were recomputed to closed
  (gold untouched). This was found while reviewing the gold: many corpus items showed
  status: open though the source clearly wasn't biddable.

v0.16.1 — fixes from an external code review

[0.16.1] — 2026-06-14 — Fixes from an external code review

Fixed

• --db now controls ALL persistence for LLM scrapers. Previously a fetch
  through core.run_fetch(store=...) saved the OPPORTUNITIES on the caller's DB but
  the LLM extraction cache + the crawl recipe store on a DEFAULT/BANDIRADAR_DB DB
  (the scraper opened its own Store(None)). core now threads the caller's store
  into an LLM source's fetch (requires_llm sources only), binding the extraction
  cache + recipe store to the same DB. (The production monitor was unaffected — it
  points BANDIRADAR_DB at its state DB so both resolved to the same file — but the
  --db flag was leaky.)
• validate_refs now requires a non-empty post_id. A crawl ref with a url +
  title but a None/empty post_id was counted as "usable", yet post_id builds
  RawDoc.id/Opportunity.id downstream — so it would collide as source:None. Such
  a ref is now treated as drift (the listing's id field moved), surfacing via crawl
  health instead of producing colliding ids.
• Reproducible Quickstart. --sample runs now pin the reference time to a fixed
  core.SAMPLE_NOW (2026-06-08, the fixtures' capture date), so the demo output no
  longer drifts as the calendar moves (bundled fixtures have fixed deadlines; items
  were silently "closing" over time). The README's match output is regenerated to the
  reproducible result and documents the pin.

v0.16.0 — self-maintaining: 9 of 10 scrapers auto-heal

[0.16.0] — 2026-06-14 — Self-maintaining: 9 of 10 scrapers auto-heal

Added

• campania, fvg, liguria now auto-heal — the last clean HTML scrapers join
  the regex-recipe self-heal, bringing auto-heal to 9 of 10 LLM scrapers. The key
  insight: the HtmlCrawlRecipe already separates the PARSE (a DATA regex,
  healable) from the FETCH (code in _listing_html), so a bespoke fetch is no
  obstacle:
  • liguria — the POST + per-session CSRF dance stays in _listing_html; the parse
    is a plain item_regex.
  • fvg — the #contributi contribution filter lives INSIDE the regex; the
    multi-page fetch concatenates pages for the recipe to parse.
  • campania — its image-widget anchors carry no text, so the new
    HtmlCrawlRecipe.title_template synthesizes the crawl label from the slug
    (humanize - → space), exactly as the hand parser did.
• Each new recipe reproduces its source's hand parser refs exactly (cassette
  tests), the same golden-gated bar the others meet.

Notes

• Only puglia stays detect-only: its "Bando aperto" badge filter is a
  conditional on a SIBLING element, not reducible to one item regex — and the host is
  CI-blocked regardless. The fetch/parse split made the planned "assisted-heal" path
  unnecessary for everything except puglia (docs/self-heal-html-design.md).

v0.15.0 — self-healing for HTML-listing scrapers (regex-recipe)

[0.15.0] — 2026-06-14 — Self-healing for HTML-listing scrapers (regex-recipe)

Added

• HTML-listing auto-heal (Phase 2a). The fragile listing PARSE of an HTML
  scraper is now modellable as DATA — an HtmlCrawlRecipe whose item_regex (named
  groups post_id/title) + url_template reproduce the refs — so on drift the LLM
  re-derives the item_regex and a candidate is adopted ONLY if it reproduces the
  golden exactly, the SAME gated propose/dispose as the JSON recipe. veneto,
  sardegna and piemonte migrated → auto-heal now covers 6 of 10 LLM scrapers.
• crawl.py: HtmlCrawlRecipe, pure apply_html_recipe (regex + title cleanup +
  dedup, reproduces each hand parser exactly), html_recipe_reproduces_golden, and
  is_safe_regex — a ReDoS guard for LLM-proposed patterns (length cap +
  nested-quantifier refusal + compile check; the golden gate remains the correctness
  socket, so a refused/wrong regex is simply never adopted).
• heal.py: propose_html_recipe + heal_html_crawl (HTML twin of heal_crawl,
  guards the candidate regex). recipe_store now persists either recipe kind
  (a _kind tag; legacy rows load as JSON — backward-compatible).
• LlmScraperSource gained a third listing flavour: html_recipe + _listing_html
  (regex-recipe auto-heal) alongside the JSON recipe and the bespoke-code detect-only
  path. The listing FETCH (multi-page, params, POST/auth) stays in the source; only
  the PARSE is the healable DATA.

Notes

• campania (title synthesized from the slug, not in the markup) and the bespoke
  fvg/puglia/liguria (filter / badge / POST+CSRF) stay detect-only — their
  parse doesn't reduce to one regex. Phase 2b ("assisted-heal": LLM proposes, golden
  pre-validates, human one-click) is the planned path (docs/self-heal-html-design.md).

v0.14.0 — self-healing generalized to JSON-listing scrapers

[0.14.0] — 2026-06-14 — Self-healing generalized to JSON-listing scrapers

Added

• Auto-heal for calabria + basilicata (was Toscana-only). Both expose a
  WP-REST JSON listing with the same item shape as Toscana, so they now declare a
  CrawlRecipe (default_recipe) and reuse the exact same gated self-heal: on crawl
  drift the LLM re-derives the dotted paths (DATA, not code) and a candidate is
  adopted ONLY if it reproduces the golden exactly. Auto-healing coverage 1 → 3
  sources.
• LlmScraperSource is now heal-capable (the recipe path was lifted out of the
  standalone toscana source into the shared base). A subclass picks its listing
  flavour: set default_recipe + _listing_json for a JSON listing (auto-heal), or
  implement _listing_refs for an HTML listing (bespoke code → drift DETECTED and
  human-flagged, never auto-healed). Backward-compatible: the 7 HTML scrapers
  (veneto, piemonte, sardegna, fvg, campania, puglia, liguria) are
  unchanged — default_recipe defaults to None → the existing detect-only path.

Notes

• The HTML scrapers stay detect-only by design (an HTML parse is code, not
  re-derivable DATA). Generalizing auto-heal to them is the subject of a separate
  design (docs/self-heal-html-design.md): a declarative HTML-selector recipe for
  the clean single-anchor sources vs an "assisted-heal" (LLM proposes, golden
  pre-validates, human one-click) for the bespoke ones.