#69 - Introduce Concepts page and restructure API Reference docs

[chloecrozier]

Addresses some of #69

Summary

• Restructures the monolithic docs/api-guide.md and docs/configuration.md into a dedicated docs/api-reference/ section: index.md (API guide), configuration.md (YAML schema), cpp.md (C++ usage + function reference).
• Adds a new top-level docs/concepts.md page that consolidates DAQIRI terminology — kernel bypass, GPUDirect, packet / burst / segment, flow / queue, memory region, zero-copy ownership, and RX reorder — so readers have a single glossary to keep open in parallel with the rest of the docs.
• Reflects team discussion on the doc-vs-tutorial split: reference pages list and connect, the Concepts page defines terms, tutorials walk a goal-driven plot.

What's in this PR

API Reference section (docs/api-reference/)

• index.md — the API guide is now a short orientation page (~60 lines): one-line description of the library, configuration-first model, the 6-step application lifecycle, and a See Also footer. The glossary content that used to live here has moved into docs/concepts.md.
• configuration.md — moved from docs/configuration.md; YAML schema reference, unchanged in scope.
• cpp.md — moved from docs/api-guide.md and reorganized:
  • Bridge sentence at the top ties the page to the API guide's lifecycle and points readers at Concepts.
  • RX and TX step headings disambiguated (### RX Step 1 — … / ### TX Step 1 — …) so the page TOC reads cleanly.
  • Reordered RX Bursts promoted from a buried level-3 subsection to a top-level section sibling of Receiving / Transmitting, so reorder is visible in the page TOC.
  • Writing Bursts to Storage moved to follow Transmitting Packets (was awkwardly stuck between RX and TX, breaking the data-path narrative).

New Concepts page (docs/concepts.md)

• Restores the kernel-bypass / GPUDirect content from the deleted tutorials/background.md, including the backend trade-off framing and the Backend Maturity admonition that had been dropped in the earlier restructure.
• Adds definitions for BurstParams, Segments, Header-Data Split, Flows, Queues, Memory Regions, Zero-Copy Ownership, and RX reorder that were previously embedded inside the API guide.
• Page intro tells readers explicitly to keep it open in a second tab — matches the "glossary, openable in parallel" framing from the team discussion.

Nav and entry points

• mkdocs.yml — Concepts is now a top-level nav entry, sibling of Getting Started / API Reference / Tutorials.
• docs/getting-started.md — Concepts added as step 1 of "Next Steps".

Sync + onboarding docs

• .claude/rules/docs-sync.md, AGENTS.md, README.md — all three updated so future contributors know docs/concepts.md exists and which source changes should trigger updates to it (backend additions, new user-facing concepts, etc.).

Removed

• docs/tutorials/background.md — content lives on docs/concepts.md now, no longer a standalone tutorial page (it was never really plot-driven).

Discussion notes

This PR implements the first iteration agreed on in the docs sync: separate the conceptual glossary from the API reference, keep the API reference focused on lifecycle + schema + language reference, and defer the larger doc-strategy decisions to a follow-up. The doc-vs-tutorial test we settled on — reference pages list and connect, tutorials walk a goal-driven plot — drove the splits below.

Out of scope / Next steps

These are intentionally deferred so this PR stays narrowly scoped to the restructure:

• Concept diagrams. At minimum a NIC → mempool → burst → segment data-flow diagram with the GPUDirect path drawn as a CPU-bypass arrow. Belongs in docs/concepts.md under Packets, Bursts, and Segments and GPUDirect. (This is the immediate next piece of work.)
• Landing page wiring. Adding a Concepts entry to the docs/index.html nav + footer is intentionally not in this PR — it can ride along with the landing-page refresh that picks up the diagrams above.
• Plot-driven base tutorials. Per the "few canonical goals" pattern: candidates include "capture UDP straight into GPU memory", "TX a known waveform from GPU memory", and "RDMA loopback between two hosts". One tutorial per goal, with the API Reference handling variations.
• Split the C++ function-reference tables off cpp.md. The page currently does two jobs (workflow narrative + alphabetical-ish reference) with opposite reading patterns. A dedicated cpp-functions.md would shorten the workflow page and make the reference easier to deep-link.
• mkdocs nav label cleanup. Tighten the inconsistent labels (API Guide / Configuration YAML Reference / C++ API Usage) into a uniform Overview / Configuration (YAML) / C++ API scheme that future-proofs for a Python API sibling.

Test plan

• python3 -m mkdocs build --strict — clean, no broken-link or anchor warnings (matches .github/workflows/docs.yml gate).
• python3 scripts/check_html_links.py site — all hand-maintained HTML links resolve.
• python3 scripts/check_doc_refs.py — 8 binaries × 19 YAML configs across 10 doc files all resolve; decision-tree YAML coverage complete.
• Visual check of docs/concepts.md rendered through MkDocs Material — admonitions (!!! warning, ??? example, ??? info) render, anchors link back into api-reference/cpp.md and api-reference/configuration.md.
• Nav order verified: Getting Started → Concepts → API Reference → Tutorials.

[greptile-apps]

Greptile Summary

This PR restructures the DAQIRI documentation by splitting the monolithic docs/api-guide.md and docs/configuration.md into a new docs/api-reference/ section (index.md, configuration.md, cpp.md), adds a top-level docs/concepts.md glossary page consolidating terminology previously scattered across the API guide and the now-deleted docs/tutorials/background.md, and updates all sync files (mkdocs.yml, docs/index.html, AGENTS.md, README.md, .claude/rules/docs-sync.md) accordingly.

• New docs/concepts.md: Restores kernel-bypass / GPUDirect background from the removed tutorial, adds BurstParams / burst / segment / flow / queue / memory region / zero-copy ownership / RX reorder definitions, and includes the Backend Maturity admonition; replaces docs/tutorials/background.md.
• docs/api-reference/: API guide oriented to the six-step application lifecycle (index.md), full YAML schema reference (configuration.md), and C++ workflow + function reference (cpp.md); RX/TX step headings disambiguated and reorder section promoted to top-level.
• Script updates: check_html_links.py drops daqiri-api.html from HTML_PAGES (file deleted); check_doc_refs.py comment corrected from CLAUDE.md to AGENTS.md; docs-sync.md source paths corrected from src/common.h to include/daqiri/common.h, matching the actual repo layout.

Confidence Score: 5/5

This is a documentation-only restructure with no source code changes; all cross-references, nav entries, sync files, and CI scripts are consistently updated.

Every file that needed updating due to the rename/move was touched: mkdocs.yml nav, docs/index.html links, README.md Documentation table, AGENTS.md layout list, .claude/rules/docs-sync.md path mappings, and the two check scripts. No broken internal links remain, no stale references to the deleted docs/api-guide.md, docs/configuration.md, or docs/daqiri-api.html were found, and all commits carry DCO sign-offs.

No files require special attention.

Important Files Changed

Filename	Overview
docs/concepts.md	New top-level glossary consolidating terminology from the old API guide and deleted background tutorial; content is accurate, cross-links resolve, admonitions are well-structured.
docs/api-reference/index.md	Short orientation page covering the six-step lifecycle and configuration-first model; all See Also links point to correct relative paths.
docs/api-reference/cpp.md	C++ workflow and function reference moved from api-guide.md; RX/TX step headings disambiguated, reorder promoted to top-level section, cross-links updated.
.claude/rules/docs-sync.md	Source-to-doc mapping corrected: src/common.h → include/daqiri/common.h, CLAUDE.md → AGENTS.md, old api-guide.md/daqiri-api.html paths replaced with api-reference/* equivalents.
mkdocs.yml	Nav restructured: Concepts added as top-level entry, API Reference expanded into three sub-pages, Background tutorial removed; consistent with new file layout.
scripts/check_html_links.py	daqiri-api.html removed from HTML_PAGES and docstring updated; correct since the file was deleted in this PR.
docs/index.html	Landing page updated: nav link changed from daqiri-api.html to api-reference/, hero actions updated with Key Concepts and API Reference buttons, tutorial list renumbered, footer Docs column updated with new api-reference/* links.
AGENTS.md	Documentation section updated to reflect new structure (api-reference/*, concepts.md); drift hotspot paths corrected; daqiri-api.html removed.
README.md	Documentation table updated with new URLs (concepts/, api-reference/, api-reference/configuration/, api-reference/cpp/); Background tutorial link removed.

_{Reviews (7): Last reviewed commit: "#69 - updating queue and intro sections" | Re-trigger Greptile}

[RamyaGuru]

Looks good, thanks! I just added a button on the landing page to link to the Concepts page. (It was the one thing that greptile review picked up)

[dleshchev]

inline comments

[dleshchev]

Linux kernel-bypass?

[chloecrozier]

Dropped the kernel-bypass label from the intro since the term already is explained in its own section on the Concepts page

[dleshchev]

One thought: daqiri is configurable both through yank config file and config struct; may be we need to use language that is inclusive of the struct

[chloecrozier]

fixed to be inclusive of both configuration options

[dleshchev]

Just "Socket" here ?

[dleshchev]

May be here or may be in the other docs, we should mention that at the moment we support gpu-only or cpu-only packet reorder, i.e. for hds types of situations the reorder (of packet parts in two memory regions) is not supported yet - will be added in the future

[chloecrozier]

I went ahead and added this clarification in docs/concepts.md

[greptile-apps]

HEAD commit missing DCO sign-off

The tip commit 26bd7b7b — "#69 - updating wordage based on comments" — has no Signed-off-by: trailer. Every commit in this repo must carry one per CONTRIBUTING.md's "Signing Your Work" section. The unsigned commit will be rejected by the DCO gate.

The contributor needs to amend and force-push:
git commit -s --amend (then git push --force-with-lease)

Rule Used: Every commit in the PR must contain a 'Signed-off-... (source)

[cliffburdick]

Maybe do different wording here or erase the first two sentences.

[chloecrozier]

Generalized to:

Concepts

DAQIRI glossary — defines the terms used across the
API Guide,
Configuration Reference, and
tutorials: kernel bypass,
GPUDirect, packet / burst / segment, flow / queue,
memory region, zero-copy ownership, and RX reorder.

I also removed some similar styled wording later in the doc

[cliffburdick]

Technically regular sockets with TCP also offers reliability

[chloecrozier]

Updated to be more inclusive of other backends. I also addressed the queue cpu pinning comment we discussed in our meeting this morning.