Agent readiness: crawler tokens, dataset instructions, Signposting, llms.txt - #294
Merged
Merged
Conversation
…290) * Allow AI crawlers in robots.txt * Add real-lastmod sitemap.xml route * Add tests and fixture for robots/sitemap * Move escapeXml to a neutral xml module * Assert the crawler block stays contiguous * Reuse isManagedDatasetId in the sitemap filter * Skip the sitemap on noindex hosts
* lib: export license helpers from jsonld * types: add zarr status fields to Dataset * lib: add use-this-data model and tests * add UseThisData section component * layout: add generic headLinks to Base * dataset: render Use this data section * dataset: add markdown mirror endpoint * fix: render Use this data outside unpublished ternary * fix: make page unpublished flag authoritative * fix: mark up commands and URLs, not sentences
* Add /llms.txt endpoint with tests * Link llms.txt from the footer * Fix llms.txt license claim, was off by 10x * Give llms.txt search link a working q example * Require v prefix on llms.txt version placeholder * Un-link llms.txt dataset-page placeholder
* lib: add signposting link model and tests * dataset: emit Signposting Link header and tags
This was referenced Sep 3, 2026
Deploying nemar-website with
|
| Latest commit: |
6367d81
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://d742ae1e.nemar-website.pages.dev |
| Branch Preview URL: | https://feature-issue-284-epic-agent.nemar-website.pages.dev |
This was referenced Sep 3, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Epic #284, four phases, evidence-ordered from the 2026-09-03 findability research (nemarOrg/nemar-cli .context/research-agent-findability.md). Descriptive only: no API response shape, Zarr index contract, data host layout, or served field changes anywhere in this PR.
Phase 1 (#285): robots.txt names every AI crawler token over one stacked Allow rule with a Sitemap line, production hosts only, non-production still disallowed; sitemap.xml with per-dataset lastmod taken from the dataset's own last change rather than build time; no canonical to openneuro.org.
Phase 2 (#286): a server-rendered Use-this-data section on the dataset page built from the data the page already fetches (what it is, license and required citations, where the bytes are, how to download, how to assess fit without downloading, and a Zarr block only when has_zarr is set), plus a /dataset/.md mirror generated from the same model and linked with rel=alternate. Most AI crawlers execute no JavaScript, so this is the only channel they have.
Phase 3 (#287): FAIR Signposting as both HTTP Link headers and head link elements from one model, so the two serializations cannot disagree: cite-as on the concept DOI, describedby for the metadata and API documents, item, license, type, and author ORCIDs.
Phase 6 (#289): a For-agents guide and generated llms.txt on docs.nemar.org (merged separately as nemarOrg/docs#16), plus a small nemar.org llms.txt with a footer link so the file is auditable. llms.txt ships as a no-regret addition, not as a findability claim: the evidence says it is not read at scale.
Closes #285
Closes #286
Closes #287
Closes #289
Phase 4 (#288, registry listings and DataCite completeness) stays open as an operations checklist. Phase 5 is nemarOrg/nemar-cli#937: its OpenAPI half landed on dev as nemarOrg/nemar-cli#1239, and the website contract-type adoption remains open there.
Test plan