Harden corpus packing and graph bundle ingress#2
Merged
Conversation
|
🤖 Hi @apstenku123, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
Pull request overview
Hardens several ingestion and evaluation paths to be fail-loud and boundary-correct, spanning packed-row boundary semantics, commit processing robustness, parquet audits, and Megatron graph-sidecar loading (including a v2 schema and compact fixed-row restoration).
Changes:
- Enforce fail-loud behavior for commit processing (missing inputs, parse errors unless explicitly allowed) and PR store lookups via readonly connections.
- Fix packed-row boundary semantics (doc_ids/loss_mask correctness) and add a parquet repair tool plus stronger parquet auditing.
- Extend Megatron indexed dataset support to restore padding for compact fixed rows and load graph routes v2 (including edge-triples + domain side-channels), plus runtime-gated compile eval improvements.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tools/clang_indexer/process_commits.py | Readonly PR store connect; strict input/parse error handling; restore canonical PR metadata into records. |
| tests/test_streaming_conveyor_progress.py | New/updated conveyor tests for commit-plan completion proofs, dedup args, and recompress/promote ordering. |
| tests/test_repair_packed_document_boundaries.py | Adds tests for packed-boundary repair logic and hardlink-safe rewriting. |
| tests/test_process_commits_fail_loud.py | Adds tests for fail-loud parse/missing-input behavior and readonly PR store usage. |
| tests/test_pack_enriched_rows.py | Adds regression tests for platform provenance union and row-local doc_ids semantics. |
| tests/test_megatron_indexed.py | Adds tests for compact fixed-row restoration and graph routes v2 + sidecars. |
| tests/test_inference_generation.py | Adds test for accepting DenseCppLM-style (logits, None) tuple output. |
| tests/test_eval_domain_routed_codegen.py | Adds runtime oracle / timeout coverage and compiler selection helper. |
| tests/test_domain_graph_routes.py | Validates default edge weight behavior in domain graph routing config. |
| tests/test_cpp_jsonl_generation_compile_eval.py | Extends constraints + trimming tests (reserved tokens; brace parsing). |
| tests/test_audit_sidecar_parquet.py | Adds row-group read enforcement and new boundary/collapsed-doc-id regressions. |
| tests/test_ast_fim.py | Tests instruction-aware char fallback kind for AST iFIM. |
| scripts/streaming_reindex.py | Makes exit status depend solely on manifest failures (no “processed > 0” escape). |
| scripts/streaming_reindex_commits.py | Same manifest-failure-only exit status logic for commits reindexing. |
| scripts/streaming_conveyor.py | Commit-plan EOF proofing; recompress-before-promote; dedup retry semantics; recompressor tracking tweaks. |
| scripts/repair_packed_document_boundaries.py | New tool to repair packed-row boundaries using source token lengths as oracle. |
| scripts/pr_ingest/pr_store.py | Adds readonly connection mode via SQLite URI mode=ro. |
| scripts/nanochat_data/pack_enriched_rows.py | Removes platform union cap; makes doc_ids row-local rather than stable provenance IDs. |
| scripts/fix_packed_parquet_boundaries.py | Updates source-doc-length shifting logic to be independent of doc_ids IDs. |
| scripts/eval_domain_routed_codegen.py | Adds runtime execution option, compile/run timeouts, and runtime-failure reporting. |
| scripts/cpp_jsonl_generation_compile_eval.py | Bans reserved tokens when tokenizer supports it; robust brace-aware body trimming. |
| scripts/audit_sidecar_parquet.py | Audits by row group; validates packed boundaries via source lengths; adds chunk-span validation. |
| evals/domain_routed_prompts.jsonl | Enables run_binary for the simple C++ “add” prompt. |
| cppmega_mlx/nn/domain_graph_routes.py | Ensures default edge weights are copied from defaults rather than empty mapping. |
| cppmega_mlx/inference/generation.py | Accepts (logits, None) inference tuple while rejecting other tuple/dict structures. |
| cppmega_mlx/data/megatron_indexed.py | Adds compact fixed-row restoration and graph routes v2 (edge triples + domain side channels). |
| cppmega_mlx/data/ast_fim.py | Extends AstFimKind and records char_ifim fallback explicitly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+242
to
+245
| finally: | ||
| with self._lock: | ||
| self._handled.update(id(fut) for _path, fut in jobs) | ||
| if failures: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+16
to
+41
| name: macOS MLX data, model, and eval contracts | ||
| runs-on: [self-hosted, macOS, ARM64, cppmega-mlx-macos] | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Run focused MLX contract suite | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| python_bin=/Volumes/external/sources/cppmega.mlx/.venv/bin/python | ||
| test -x "${python_bin}" | ||
| "${python_bin}" -m pytest -q \ | ||
| tests/test_ast_fim.py \ | ||
| tests/test_audit_sidecar_parquet.py \ | ||
| tests/test_cpp_jsonl_generation_compile_eval.py \ | ||
| tests/test_domain_graph_routes.py \ | ||
| tests/test_eval_domain_routed_codegen.py \ | ||
| tests/test_inference_generation.py \ | ||
| tests/test_megatron_indexed.py \ | ||
| tests/test_pack_enriched_rows.py \ | ||
| tests/test_streaming_conveyor_progress.py \ | ||
| tests/test_process_commits_fail_loud.py \ | ||
| tests/test_repair_packed_document_boundaries.py | ||
| git diff --check | ||
|
|
||
| linux-portable: |
Comment on lines
+42
to
+65
| name: Linux portable syntax and conveyor contracts | ||
| runs-on: [self-hosted, Linux, X64, cppmega-mlx] | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.13" | ||
| - name: Install portable test dependencies | ||
| run: python -m pip install --disable-pip-version-check pytest numpy pyarrow tokenizers zstandard | ||
| - name: Run portable script contracts | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| # cppmega.mlx imports Apple's MLX runtime at package import time. The | ||
| # Linux runner exercises only portable scripts and deliberately skips | ||
| # the macOS-only root conftest/package initialization. | ||
| python -m pytest -q --noconftest \ | ||
| tests/test_audit_sidecar_parquet.py \ | ||
| tests/test_streaming_conveyor_progress.py \ | ||
| tests/test_process_commits_fail_loud.py \ | ||
| tests/test_repair_packed_document_boundaries.py | ||
| python -m compileall -q scripts tools/clang_indexer | ||
| git diff --check |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prerequisite baseline from the 2026-07-13 full review.
Includes:
Verification:
This PR intentionally precedes the six isolated objective/graph/inference/USR/domain/Nebius case PRs.