v0.2.1

Fixed

• The scanner scores TDMRep from the tdm-reservation and tdm-policy headers, /.well-known/tdmrep.json and the <meta> tag, which is what the compiler emits. The non-standard Link: rel="tdm-policy" header still counts, for a third of the points, so sites on the 0.1 Worker or plugin are not marked as having nothing.
• The scanner detects the Cloudflare Content-Signal line in robots.txt and the matching response header, and awards usage points for either.
• The scanner parses robots.txt per RFC 9309: consecutive User-Agent: lines form one group, tokens match case-insensitively and Allow wins a tie. It uses the evaluator the compiler tests already use, moved to scanner/robots_eval.py.
• The JSON report's robots.found is a boolean rather than the file body.

Added

• tests/test_scan.py covers the scanner, including a scan of the compiler's own output served from a local HTTP server.

v0.2.0

Fixed

• The compiler resolves the manifest once into a Plan that every emitter reads, so robots.txt, AIPREF, TDMRep and Content-Signal can no longer disagree.
• A rule inherits each category it does not declare, an explicit unknown included, from the longest enclosing rule and then from the defaults (spec 6.6).
• A train-genai deny now yields train-ai=n.
• robots.txt emits every rule path as Allow or Disallow, so RFC 9309 longest match keeps nested denies.
• Path-level ai-input denies are emitted even when the default allows.
• Globs are lowered to RFC 9309 patterns: * and $ are kept, /x/* becomes /x/, no / is appended and a leading / is required.
• The compiler validates the manifest first and exits 2 on errors unless --force is given.
• An unknown state raises an error instead of lowering to a fallback.
• TDMRep is emitted as the W3C protocol defines it, as /.well-known/tdmrep.json with per-path rules plus tdm-reservation and tdm-policy headers and the <meta> equivalent.
• The compiler no longer emits the non-standard Link: rel="tdm-policy" header. The Worker, playground, WordPress plugin and scanner still use it and are untouched here.
• Site-wide surfaces (aipref-header, content-signal, the noindex in x-robots-tag, the TDMRep headers) no longer publish the default value on paths that override it and list the per-path values as comments instead.
• Per-path lines are emitted only for categories a rule declares or inherits from an enclosing rule, so an unrelated or empty rule cannot reopen an overlapping deny.
• Rules that lower to the same path are merged, and two rules on one path no longer crash the compiler.
• A rule on / overrides the defaults.
• A /**/ in the middle of a pattern expands to two patterns so the segment boundary survives, with at most four /**/ per path.
• Equal-length patterns put Allow first, as RFC 9309 does, and the compiler warns when such a tie carries different decisions.
• Rules with matchers no target compiles contribute only their deny decisions.
• The validator CLI accepts several files and has --help.
• The validator requires match.path and rejects paths with whitespace, #, ?, brackets or a non-trailing $.
• The validator rejects control characters in publisher and endpoint strings, and a category declared in both tiers with different states.
• The validator warns on unknown experimental categories and on matchers no target can compile.
• Malformed manifests no longer crash the validator.

Added

• A content-signal emit target writes the Cloudflare Content Signals line for robots.txt and the matching response header.
• compiler/agents.json, a curated user-agent table with operator sources, adds the training crawlers Meta-ExternalAgent, MistralAI-Training, Ai2Bot-Dolma, cohere-training-data-crawler, TikTokSpider, DeepSeekBot, omgili, Webzio-Extended and SBIntuitionsBot, the search crawlers Amzn-SearchBot, MistralAI-Index, SBIntuitions-SearchBot and Meta-WebIndexer, and the user-triggered fetchers Perplexity-User, MistralAI-User, DuckAssistBot, Meta-ExternalFetcher, Amzn-User and Google-Agent.
• tests/ holds golden-file tests for every example and edge case, crawl-decision tests through a small RFC 9309 evaluator, and validator and CLI tests, which a GitHub Actions workflow runs.

Changed

• Diffbot moves from training to search, as its operator documents.
• examples/news-publisher.json drops its content_types matcher and points tdm_policy at a policy document rather than at tdmrep.json.
• The README's full example moves ai-input to the experimental block so it validates.
• The schema $id points at the real repository slug, match.path is required and constrained, and endpoints must be http(s) URLs.
• The tdmrep target writes tdmrep.json and tdmrep-headers.txt instead of tdmrep-header.txt, so scripts that read --out need updating.
• An unknown --emit target is an error (exit 3) rather than a skipped warning.