Feat/agent friendly docs#676
Merged
Merged
Conversation
The workflow invoked the validator via `pnpm run check:sql-exec:changed -- --base-branch main`. pnpm 9 forwards the literal `--` to the script, so commander treated `--base-branch` and `main` as positional file arguments and `--changed-only` lost its base-branch input. Result: 0 files scanned, 0 SQL blocks, exit 0 — CI reported PASS without actually validating any changed docs. - Call node directly in the workflow so args reach commander verbatim. - Reject `--changed-only` + positional args in the validator as a defensive guard against future reintroduction.
check-sql-syntax.yml had the same `pnpm run ... -- --base-branch` pattern that caused the execution workflow to silently pass. The defensive guard added in this PR's previous commit correctly flagged it (exit 1, no silent pass), but the job still needs to run — switch to direct node invocation so commander receives `--base-branch` verbatim.
… SQL The splitter used plain `;` as a statement terminator, which broke any documented compound statement whose body contains inner `;` — most visibly `CREATE TASK ... AS BEGIN <stmt>; <stmt>; END;`. A single compound got sliced into 3+ fragments and fed to the parser/runner piecewise, yielding `syntax error near ""` on the first fragment and `syntax error near "END;"` on the orphaned tail, even though the MatrixOne parser accepts the compound as one statement. Track block depth across lines: - BEGIN opens a block (compound form); exclude `BEGIN WORK`, `BEGIN TRANSACTION`, and bare single-line `BEGIN;` — matching the SPBEGIN lookahead in pkg/sql/parsers/dialect/mysql/scanner.go. - END closes a block. - IF / LOOP / WHILE / CASE / REPEAT also open blocks when they are at the start of a line (or preceded only by a `label:`), matching the compound-statement grammar in mysql_sql.y. Expression uses like `SELECT CASE WHEN ... END` or `IF(a,b,c)` stay inline and are not counted. - String literals and comments are skipped during the scan. Only split on `;` when depth == 0. Applies to both splitSqlStatements and splitSqlStatementsWithAnnotations so syntax and execution checkers both benefit.
Ship the first pass of the Agent-friendly documentation roadmap:
- doc-validator baseline cleanup: drop stale `supportedVersions`, clarify
that the real version comes from `MO_TARGET_BRANCH` / `mo-test-env.sh`,
add `:all` scripts/targets and a KNOWN_ISSUES entry for the sql-runner
`Unknown database` context-loss bug.
- Bypass that same validator bug in two partition-heavy pages via
`<!-- validator-ignore-exec -->`. Full-corpus exec scan is now 0-fail
across 414 pages / 1618 SQL statements on 3.0-dev nightly.
- Introduce `mysql_compat` frontmatter on every SQL-Reference page (126
pages) with values derived from the authoritative overview
`Overview/feature/mysql-compatibility.md` and cross-checked against
MySQL 8.0. Distribution: 38 full / 35 partial / 53 mo_only / 0 unknown.
- Enforce that frontmatter in CI via `scripts/check-compat-frontmatter.js`
(wired into `check-sql-syntax.yml`).
- Auto-generate `Reference/mysql-compatibility-matrix.md` from the
frontmatter via a pre-build mkdocs hook.
- Emit the agent-delivery triple via a post-build hook:
* per-page `.md` mirrors under `site/MatrixOne/**`
* `site/llms.txt` (llmstxt.org format, curated featured pages)
* `site/llms-full.txt` (full corpus concatenated, ~88k lines)
- Update README with an AI-agents section and CONTRIBUTING with a
SQL-Reference contributor checklist.
Verified locally: `mkdocs build` succeeds and emits all three artefacts;
`pnpm run check:frontmatter` and `node scripts/doc-validator/index.js
--check=execution` both pass.
Five new scripts supporting ongoing SQL-block hygiene work: - sql-coverage-report.js: classifies every fenced block (executed / ignore-exec / ignore-all / impure / non-sql-language / admin / external-dependency) so we can see how many SQL examples actually reach the execution checker. - triage-ignore-all.js: for every validator-ignore block, attempts native-parser validation plus sandboxed execution against the running MatrixOne container. Includes baseline cleanup that drops non-system accounts/pitrs/snapshots/publications/stages/databases before each run, plus per-block pattern-based cleanup, so repeated triage runs start from a clean baseline. Handles mysql-style transcripts, syntax templates, query-timeout hangs, and connection recycling. - unmark-safe-ignore.js: bulk-removes validator-ignore markers on blocks that triage classified as safe-to-run. HOLD_IGNORE carve-out for syntax-example blocks that must stay ignored. - remark-failing-as-ignore-exec.js: runs the execution checker on a set of files, locates every failing statement, and inserts validator-ignore-exec on the enclosing SQL fence so those blocks still get syntax validation without tripping exec result diffs. - retag-dialects.js: encodes dialect decisions (Flink / PL-SQL / PostgreSQL / T-SQL) and rewrites the fence language accordingly.
The Flink CDC tutorials interleave SQL from four different engines
(Flink SQL, Oracle PL/SQL, PostgreSQL, SQL Server T-SQL) with
MatrixOne SQL. Previously every block was fenced as `sql` and carried
a `<!-- validator-ignore -->` to keep the MatrixOne parser from
rejecting them.
Retag the 17 third-party fences to their actual dialect so the
MatrixOne validator skips them naturally and readers get correct
syntax highlighting:
- Flink SQL DDL (CREATE TABLE ... WITH ('connector' = ...)) -> flink
- Oracle DDL (NUMBER / VARCHAR2) -> plsql
- PostgreSQL (replica identity full) -> postgresql
- SQL Server (NVARCHAR, master.dbo.*, exec sp_*) -> tsql
Drop the now-redundant validator-ignore markers on the retagged
fences. MatrixOne-native blocks in the same files keep their `sql`
fence and remain checked.
…verage Cross-referencing the triage-ignore-all report against the live MatrixOne 3.0-dev container, 62 previously-ignored SQL blocks across 21 files parse AND execute cleanly. Strip their `<!-- validator-ignore -->` markers so they participate in both the syntax check and the execution check. 38 of those blocks then flagged on the full execution scan — not because the SQL is wrong, but because the documented expected-output hard-codes a database name like `db1` that won't match the per-file `doc_test_*` sandbox, or the block depends on cross-block state dropped between invocations. Those get `<!-- validator-ignore-exec -->` added, keeping them in the syntax checker while skipping execution. One block in comment.md (the `// ...` SQL-comment-syntax example) was reclassified back to `<!-- validator-ignore -->` because its content is intentionally unparseable documentation about comment syntax itself. End state: syntax check 431/431 files / 3690/3690 statements green; exec check 432/432 files / 1699/1699 statements green. Coverage report: ignore-all 211 -> 126 (-85), executed 494 -> 523 (+29).
…esult columns Two sql-runner bugs were causing perfectly valid SQL examples to be forced into `<!-- validator-ignore-exec -->` purgatory. 1) Unknown database on CREATE TABLE … PARTITION BY RANGE/LIST/HASH/KEY. MatrixOne defaults to `lower_case_table_names = 1`, so the planner lowercases identifiers internally when re-resolving the current database. The sandbox name built from the file path kept mixed case (e.g. `doc_test_docs_MatrixOne_Performance_Tun_…`), so MO looked up its lowercase twin and reported "Unknown database doc_test_docs_matrixone_…" even though SELECT DATABASE() still returned the original mixed-case name. Force the generated name to lowercase at construction time in utils/db-connection.js::createTestDatabase. 2) Expected-output assertions treated environment-dependent columns as if they were literal data. `SHOW FUNCTION STATUS` documented output hardcodes `Db = db1`; our sandbox runs in `doc_test_*`. Same story for `created_time`, `modified`, `role_id`, `size`, … Introduce a VOLATILE_COLUMNS allow-list in sql-runner.js and relax compareTableOutput() to only check presence (non-undefined) for those columns instead of a strict value match. With both fixes in place, the corpus execution scan goes from 1681 to 1896 passing statements (+215) and frees 30 net validator-ignore-exec markers across 31 files without introducing any new failures (tracked separately in the follow-up document commit).
After shipping runner fixes, many `<!-- validator-ignore-exec -->` markers are no longer load-bearing. try-unmark-ignore-exec.js reclaims them automatically: strip every marker, run the execution checker over the affected files, and re-apply markers only on the blocks that still fail. Leaves a file untouched if nothing would change. Meant to be re-run whenever we tighten the sandbox or add new expected-output normalization — it keeps the ignore budget honest without manual bookkeeping.
…fixes
Running the new try-unmark-ignore-exec reclamation pass after the
sandbox-naming and volatile-column fixes: 30 blocks across the
corpus no longer fail the execution checker and have their
`<!-- validator-ignore-exec -->` markers removed. 3 blocks still
need exec-skip (results depend on state the sandbox can't provide)
and have a fresh marker re-applied by remark-failing-as-ignore-exec.
Net coverage change (from scripts/sql-coverage-report.js):
executed 523 -> 546 (+23)
ignore-exec 430 -> 407 (-23)
Syntax check: 432/432 files / 3690/3690 statements green.
Execution check: 432/432 files / 1896/1896 statements green.
The remaining markers on the retained blocks are rewritten from the
inline form
```sql <!-- validator-ignore-exec -->
to the standalone form
<!-- validator-ignore-exec -->
```sql
so the tooling treats every marker uniformly. No semantic change.
`Data-Manipulation-Language/load-data.md` no longer exists in the SQL-Reference tree — the page was split into `load-data-infile.md` and `load-data-inline.md` at some point and the top-level SQL-Type index was never updated. markdown-link-check flagged it as the only true dead link across this PR's 315 touched files (every other failure was a GitHub rate-limit noise). Point the index at both surviving pages.
…comparison The mysql2 Node.js driver decodes MatrixOne BOOL columns and boolean-valued expressions (comparisons, BETWEEN, IS) as JS numbers 1/0, while the `mysql` CLI renders them as true/false — which is the form the docs mirror. Accept both representations in valuesMatch so CLI-style expected output validates correctly. Clears 10 SQL-validation failures across data-types.md, is.md, and the 8 comparison-operator pages (=, <>, <, >, <=, >=, BETWEEN, NOT BETWEEN).
This was referenced May 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Which issue(s) this PR fixes:
issue #
What this PR does / why we need it:
Overhaul the MatrixOne English docs repository to serve both human readers and AI agents (Cursor / Claude Code / ChatGPT / MCP clients), and tighten the SQL
validation pipeline so documented examples stay in lockstep with the
3.0-devnightly image.Highlights
Agent-friendly delivery (
mkdocs buildnow emits)site/llms.txt— curated index in llmstxt.org format with a blockquote of MatrixOne-specific hints for agents writing SQL.site/llms-full.txt— whole corpus concatenated (~88 k lines) for long-context models.site/MatrixOne/**/*.md— raw markdown mirror of every HTML page, so agents can append.mdto any URL instead of parsing HTML.CONTRIBUTING.mdgain an "For AI Agents" section pointing at these endpoints.MySQL compatibility matrix, driven by frontmatter
docs/MatrixOne/Reference/SQL-Reference/**now declaresmysql_compat(full/partial/none/mo_only/unknown) withoptional
differs_from_mysqlandmo_onlylists.scripts/check-compat-frontmatter.js(wired intocheck-sql-syntax.yml).docs/MatrixOne/Reference/mysql-compatibility-matrix.mdrebuilds on everymkdocs buildvia a pre-build hook.Overview/feature/mysql-compatibility.mdcross-referenced with MySQL 8.0 reference.
SQL validator fixes that unblock real execution checks
sql-runner.js: added aVOLATILE_COLUMNSallow-list (db,created_time,modified,role_id,size, …) so documented example output isn'tflagged as stale when the sandbox database name or a timestamp legitimately differs.
db-connection.js: sandbox database names are now forced to lowercase, avoidingUnknown database doc_test_*failures onCREATE TABLE … PARTITION BY RANGE/LIST/HASH/KEY(MatrixOne lowercases identifiers vialower_case_table_names = 1).KNOWN_ISSUES.mdrecords the cause and reproduction for future regressions.Third-party SQL retagged with their real dialect
```sqlto```flink/```plsql/```postgresql/```tsqlso the MatrixOne parser no longer chokes on them and readers get correct syntax highlighting.Tooling added for ongoing hygiene
scripts/sql-coverage-report.js— classifies every fenced block (executed / ignore-exec / ignore-all / impure / non-sql-language / admin /external-dependency).
scripts/triage-ignore-all.js— per-block parse + sandboxed execution with baseline cleanup between runs (drops residual non-system accounts / pitrs /snapshots / stages / publications / databases).
scripts/unmark-safe-ignore.js/scripts/remark-failing-as-ignore-exec.js/scripts/try-unmark-ignore-exec.js— reclaim ignore markers that are nolonger load-bearing whenever we tighten the runner.