Skip to content

v0.10.0

Latest

Choose a tag to compare

@TBarregren TBarregren released this 16 Aug 20:04
· 59 commits to main since this release

The client half of a coordinated release with Kntnt Extractor 0.6.0. The verified Extractor API-version ceiling rises from 6 to 7, which is the act of declaring this client correct against that build — so install the Extractor on production first, then this, and do neither while an extraction is in flight.

Three things carry the release. A define whose value the Extractor withholds is no longer ported: null on the wire is a masking value, and writing define('NAME', null) locally makes defined('NAME') true and silently suppresses whatever fallback the owning plugin runs for "not configured". This was latent until the Extractor replaced its secret deny-list with an allow-list, at which point a masked third-party key would have reached the writer. The poll loop moved out of prose and into scripts/poll_extraction.py — one blocking invocation, one terminal verdict — instead of being rewritten by the agent from eight pinned literals on every run. And the main extraction lost its overall wall-clock budget: a 3600 s cap expired on a healthy, visibly-advancing run that then had to be re-polled by hand, and the stall window is the check that actually distinguishes dead from slow.

The run report is also more honest about what it cannot do: it now states the Extractor API version production reported and, when that is below this client's ceiling, names each behaviour degraded on that host — three at the time of writing. The remedy for all of them is upgrading production's Extractor, never a client-side workaround.

A limit worth stating: raising the ceiling to 7 was verified by source inspection from three angles, not by unsealing a version-7 container. No such container existed when the verification was done. The record of exactly what was and was not checked is in plans/004-raise-the-verified-api-version-ceiling-to-7.md, now tracked in the repository rather than living only on the maintainer's machine.

Added

  • The canonical discovery document now carries the Extractor API version the health check observed, as a required top-level api_version sibling of environment — plumbing that previously stopped at the health check itself, discarded the moment the comparison against the floor and ceiling was made. agents/discovery-classify.md passes the number through verbatim rather than re-fetching it, and both skills' run report now states the version production reported and, when it is below this client's ceiling, names each behaviour that is consequently degraded on this host: the identity report's absence below API version 4, the absent progress.chunks_done and widened stall window below API version 6, and strict being silently ignored by any Extractor that predates it. The report was previously silent on all three even though production runs API version 5 today. This plumbing changes no decision — the version pin's floor and ceiling are unchanged — it only makes an existing fact reportable.

Changed

  • Two-phase discovery's "the bootstrap is small" claim is now written down as a premise that holds only where wp_postmeta is small, so a bloated table is a known case rather than a surprise. A skip-wp_postmeta fallback was considered and is not implemented: 493 MB of table versus 279 MB of thumbnails compared InnoDB allocated size to files, while the whole bootstrap container measured 30 MB — a loss on transferred bytes, with only server-side cost remaining (ADR-0017).
  • Both skills now state that a fast preflight is not a promise about the main extraction. The two-table probe proves the loopback and continuation path; a pass in a few seconds says nothing about how a large table or a large file will behave.
  • The poll loop is now scripts/poll_extraction.py: one blocking invocation, one terminal verdict, the seven discipline literals in code so an agent cannot re-derive the cadence, timeouts, backoff, confirmed-vanished check, stall window, or the preflight and bootstrap budgets from prose. The Application Password is KNTNT_EXTRACTOR_APP_PASSWORD in that one process's environment — never argv, never printed. The poll-owning agents and both skills invoke the helper; the consistency suites bind the pinned phrases to the script's constants.
  • The main extraction has no overall wall-clock budget. A 3600 s cap expired on a healthy, visibly-advancing many-file job at less than a quarter of the file phase; sealing cost is per file, not per byte, so a constant cannot be right. The helper now omits the budget argv on that loop and stops only on the stall window — the check that distinguishes dead from slow. Preflight and bootstrap keep their 10- and 15-minute budgets.
  • The main extraction is submitted with strict: false, so a file that vanished between the GET /files walk and the POST is skipped and reported rather than failing the whole job. extract-transfer sends the member, surfaces any skipped_files the create or poll returns, and unseals against the remaining file list — the container only holds what the plugin packaged. A missing table is still a hard 404; the error data now names every missing table and every missing file, so the recovery is no longer a full re-walk. strict defaults to today's hard fail on an older Extractor that ignores the member, and the new fields are additive, so the verified API-version ceiling stays ≤ 6.
  • The verified Extractor API-version ceiling is raised from 6 to 7. The Extractor's GET /environment define-disclosure allow-list is the change that moved api_version, and source inspection of the Extractor's own ADRs, its normative docs/container-format.md, and its artifact-writing commit history since v0.5.1 found no change to the sealed container's byte layout, segment framing, segments-per-resource, sealed index, or reassembly order — the bump is a deliberate compatibility interlock (Extractor ADR-0018), not a shape claim, and scripts/unseal.py needed no change. See ADR-0021.

Fixed

  • A define whose value the Extractor withheld (null on the wire from GET /environment) is no longer ported. scripts/classify.py now classifies such a define auto-excluded under a new withheld class instead of offering it at the wp_config_defines gate, so scripts/wpconfig_block.py never writes define('NAME', null); into the local wp-config.php — a define php -l accepts and the smoke test never catches, but which makes defined('NAME') report true and suppresses whatever fallback the plugin runs for "not configured". This was latent until now: every name the Extractor currently masks is also routed to an auto-excluded class by name, but the Extractor is replacing its secret deny-list with an allow-list, after which a masked plugin define (a third-party API key, say) would have reached the gate and the writer with a null value. Both clone and pull now name every withheld define in the run report, so the operator learns which values did not come down instead of finding out later that a plugin is silently unconfigured. scripts/wpconfig_block.py also gained its own rejection of a None define value as defence in depth, for a caller outside the normal classifier-to-writer path. See ADR-0020.
  • The resolved exclusion set left cache-plugin trees, backup-tool working directories, and the Extractor's own uploads staging in scope. On a live site that meant LiteSpeed's hashed CSS, BackWPup restore logs, and a previous failed job's .sealed.building were selected for clone — and the last of those can vanish mid-run because POST /extractions reclaims the directory the selection just named. ALWAYS_EXCLUDED now covers the known cache-plugin and backup-tool trees (wp-content/litespeed, wp-content/et-cache, wp-content/w3tc-*, wp-content/uploads/backwpup*) and the plugin's own three uploads directories as a self-reference, not a cache. The names that live inside the uploads directory — the backup tool's scratch and the plugin's three — are re-anchored on the classifications' uploads_prefix, so a site that moved its uploads directory (a non-default WP_CONTENT_DIR, an UPLOADS define) excludes them where they actually are and not only at the standard location, which is the layout classify.py already honours everywhere else. Glob-bearing directory prefixes match in filter_manifest.py and baseline_diff.py the same way, so a suffixed live directory is dropped and a sibling that does not fit the glob is kept; a consistency suite now pins those two matchers as one implementation, since a disagreement between them is exactly the deletion-diff poisoning issue #35 closed at the assembly end. The assembler is still the one source both the selection and the baseline consume.
  • A healthy job against an API-version-5 Extractor could abort as a false stall: below API version 6, progress.chunks_done does not exist, so stall detection fell back to the two coarse counters — but those move only when a whole table or a whole file finishes, and a 186-table site working through one large table stood still for minutes at a time on the unconditional 10-minute stall window. scripts/poll_extraction.py now widens the stall window to 40 minutes from the first poll that observes chunks_done absent, says so in the run's output, and reports the correct give-up minutes for whichever window actually fired. See ADR-0018.
  • Both poll-owning subagents shaped a FAILED verdict specifically so the orchestrator could clean up the still-active job afterwards, and the orchestrator was never told to do it: an unseal failure after a complete download, and an exhausted stall window, both left the production artifact or the active job in place on a live client site until the plugin's own TTL, and a failed bootstrap's cleartext dump of real user (and, when a recognised CRM was carried, subscriber) rows had nothing scheduled to report or remove it. Both SKILLs now carry a Closing out a failed phase subsection that cancels a job that never reached ready, consumes one that reached ready and downloaded but failed to unseal, and reports (never auto-consumes) one whose download itself failed — plus reports and, by default, deletes the diagnostic bootstrap dump. See ADR-0022.