Documents the RFC 9309 rule matching, the refusal that no longer retries and
carries its own error code, the reachable opt-out on the extract API, and the
5xx change. The README feature list now says what the matching actually does
and names the extract-side option.

Adds a real-site integration suite (excluded from CI, network-bound) that
drives the whole path against live robots.txt files: an allow-all site
(`User-agent: *` with an empty `Disallow:`), a disallowed path, the opt-out,
and a real file whose two opening User-agent lines share one group and whose
longer `Allow` uncovers a subtree. Each fact records what the site's file said
when it was written, so a later failure can be told from a site that changed.