feat(#241): bootstrap invariant verification and documentation by konard · Pull Request #242 · netkeep80/PersistMemoryManager

konard · 2026-04-10T22:59:28Z

Summary

Add validate_bootstrap_invariants() public method that verifies the PAP image is a valid,
self-described persistent environment (Issue Bootstrap a new PAP image with free-tree and system domains #241)
Integrate invariant check into both create() and load() paths — bootstrap failures are
now caught immediately
Add comprehensive test suite (test_issue241_bootstrap) with 3 test cases:
1. Invariants hold after create(size)
2. Invariants hold after save/load cycle
3. Bootstrap is fully deterministic (two independent creates produce identical layout)
Document the canonical 6-step bootstrap sequence in docs/bootstrap.md:
1. Manager header (Block_0)
2. First free block (Block_1)
3. Forest/domain registry
4. System domain registration (free_tree, symbols, domain_registry)
5. Symbol dictionary bootstrap (9 system names interned)
6. Invariant verification

Invariants checked by `validate_bootstrap_invariants()`

Manager header has valid magic, total_size, and granule_size
Forest registry exists with valid magic and version
At least 3 domains registered (free_tree, symbols, registry)
All system domains have kForestDomainFlagSystem flag and interned symbol
system/free_tree has correct binding kind
Symbol dictionary root is non-zero
system/domain_registry root matches hdr->root_offset

Bug fix: rebuild_free_tree corrupting user AVL trees (pre-existing on main)

Fixed a critical bug in rebuild_free_tree() where AVL fields were being reset for
all blocks, including allocated blocks. Allocated blocks use their TreeNode AVL
fields for user-level data structures (symbol tree, pmap, etc.). Resetting these
during load() corrupted the symbol AVL tree, causing intern_symbol_unlocked() to
re-allocate all bootstrap symbols on every load — inflating alloc_block_count by 8.

This was the root cause of the pre-existing test_issue43_phase2_persistence failure
(alloc_block_count: 20 == 12) that existed on main.

Fix: Only reset AVL fields for free blocks (weight == 0) in rebuild_free_tree().

Single-header regeneration and test updates

Regenerated pmm.h and pmm_no_comments.h to include the bootstrap code
Updated 18 test files to use baseline_alloc pattern instead of hardcoded block
counts (e.g., alloc_block_count() == baseline_alloc instead of == 1)
Increased buffer sizes for small static storage tests to accommodate bootstrap blocks

Other CI fixes in this PR

clang-format: Fixed all formatting violations across the entire codebase
File size check: Extracted 450+ lines of private forest/domain registry methods from
persist_memory_manager.h into forest_domain_mixin.inc (1942 → 1487 lines, under 1500 limit)
ASan/UBSan misaligned access: Fixed UB in intern_symbol_unlocked() by using memcpy
instead of direct struct member assignment on potentially misaligned raw pointers

Files changed

File	Change
`include/pmm/persist_memory_manager.h`	Add `validate_bootstrap_invariants()` method; integrate into `create()` and `load()`
`include/pmm/forest_domain_mixin.inc`	New: extracted forest/domain registry private methods (fixes file size check)
`include/pmm/allocator_policy.h`	Fix: only reset AVL fields for free blocks in `rebuild_free_tree()`
`single_include/pmm/pmm.h`	Regenerated to include bootstrap code
`single_include/pmm/pmm_no_comments.h`	Regenerated (stripped comments)
`tests/test_issue241_bootstrap.cpp`	New: 3 test cases for bootstrap invariants
`tests/CMakeLists.txt`	Register new test
18 test files	Updated block count expectations for bootstrap-aware baseline
`docs/bootstrap.md`	New: canonical bootstrap sequence documentation
`changelog.d/20260410_230000_issue241_bootstrap.md`	Changelog fragment
`demo/scenarios.cpp`, `examples/stress_test.cpp`, etc.	clang-format fixes

Test plan

All 72 tests pass (previously 18 failed due to stale single-header + 1 pre-existing failure)
3 new test cases verify bootstrap invariants after create, after save/load, and determinism
rebuild_free_tree fix resolves pre-existing test_issue43_phase2_persistence failure
validate_bootstrap_invariants() returns true after both create() and load()
persist_memory_manager.h under 1500-line limit (1487 lines)
clang-format clean across entire codebase
Single-header files regenerated and match source
ASan/UBSan misaligned access fixed with memcpy

Fixes #241

🤖 Generated with Claude Code

Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: netkeep80#241

- Add validate_bootstrap_invariants() public method that checks all bootstrap invariants: header validity, forest registry, system domains, symbol dictionary, and domain-registry consistency. - Integrate invariant check into create() and load() as final verification. - Add comprehensive test (test_issue241_bootstrap) with 3 test cases: after create, after save/load, and determinism verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Document the canonical 6-step bootstrap sequence in docs/bootstrap.md: manager header, free block, forest registry, system domains, symbol dictionary, and invariant verification. - Document load-time restoration procedure and system block semantics. - Add changelog fragment for the new validate_bootstrap_invariants() API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

konard · 2026-04-10T23:27:06Z

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $7.032258

📊 Context and tokens usage:

Claude Opus 4.6:

Context window: 122.9K / 1M (12%) input tokens, 27.4K / 128K (21%) output tokens

Total: (115.9K + 10.8M cached) input tokens, 27.4K output tokens, $6.808802 cost

Claude Haiku 4.5:

Total: (88.9K + 590.2K cached) input tokens, 10.7K / 64K (17%) output tokens, $0.223456 cost

🤖 Models used:

Tool: Anthropic Claude Code
Requested: opus
Main model: Claude Opus 4.6 (claude-opus-4-6)
Additional models:
- Claude Haiku 4.5 (claude-haiku-4-5-20251001)

📎 Log file uploaded as Gist (3024KB)

View complete solution draft log

Now working session is ended, feel free to review and add any feedback on the solution draft.

konard · 2026-04-10T23:27:14Z

🔄 Auto-restart triggered (iteration 1)

Reason: CI failures detected

Starting new session to address the issues.

Auto-restart-until-mergeable mode is active. Will continue until PR becomes mergeable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

netkeep80 · 2026-04-10T23:45:20Z

Align free-tree implementation with the general forest model

Summary

Нужно привести AvlFreeTree к архитектурно правильному положению: это должно быть главное дерево свободного ПАП,
но при этом оно должно выглядеть как частный случай общей forest-модели, а не как отдельная особая логика,
слабо связанная с общими semantics TreeNode.

Why

Сейчас есть semantic gap:

TreeNode::weight задокументирован как универсальный ключ дерева;
forest как идея уже существует;
но AvlFreeTree на практике вычисляет ordering через размер блока из линейной геометрии ПАП,
а не через единый forest-level ключ/политику.

Это не обязательно ошибка, но это надо оформить явно и последовательно.

Scope

Документировать и затем реализовать точную forest-policy для free-tree:
- какой критерий ordering;
- какой tie-breaker;
- какую роль играет weight;
- как free-tree соотносится с линейным ПАП.
Устранить неясность между:
- “weight как общий tree-key”;
- “размер блока как ключ free-tree”.
Явно оформить AvlFreeTree как специализированную forest-policy.
Рассмотреть и задокументировать, нужен ли bucketed free forest:
- по классам размеров;
- порядок внутри bucket по адресу.
Синхронизировать free-tree semantics с field-semantics doc.

Out of scope

многопольные multi-index на одном и том же block header;
pjson_db secondary indexes;
полная transactional forest model.

Acceptance criteria

Документация free-tree как главного forest-domain согласована с общим forest doc.
Политика ordering free-tree описана однозначно.
Роль weight для free-tree объяснена и не конфликтует с общей forest-моделью.
Код и docs не противоречат друг другу по free-tree semantics.

Dependencies

Issue #1, Issue #2, Issue #5.

…heck (Issue netkeep80#241) Extract 450+ lines of private forest/domain registry methods from persist_memory_manager.h into forest_domain_mixin.inc to keep the main header under the 1500-line CI limit. Also fix ASan/UBSan misaligned memory access in intern_symbol_unlocked() by using memcpy instead of direct struct member assignment on potentially misaligned raw pointers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The previous format fix used clang-format 22 which produces slightly different output for (std::max)/(std::min) patterns and certain alignment rules. Re-format with clang-format 18 to match CI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

konard · 2026-04-11T00:48:36Z

Final CI Results (run 24270494864)

Fixed by this PR (were failing on main, now passing)

Check	Main	PR
File size check (≤ 1500 lines)	❌	✅
clang-format check	❌	✅
Changelog fragment check	skipped	✅

Unchanged (passing on both main and PR)

Check	Main	PR
cppcheck	✅	✅

Pre-existing failures (same on main 24267900208)

Check	Main	PR	Root cause
Single-header presets up-to-date	❌	❌	`single_include/pmm/pmm.h` not regenerated after forest registry bootstrap was added
ubuntu-latest / gcc	❌	❌	`test_issue43_phase2_persistence`: `alloc_block_count 20 == 12`
ubuntu-latest / clang	❌	❌	Same test failure
macos-latest / clang	❌	❌	Same test failure
windows-latest / msvc	❌	❌	Same test failure
ASan+UBSan / clang	❌	❌	Same test failure + misaligned `pstringview` in stale single-header
ASan+UBSan / gcc	❌	❌	Same test failure
TSan / clang	❌	❌	Same test failure
Build Demo (all 3 platforms)	❌	❌	Same test failure

Summary: This PR introduces zero new CI failures and fixes 3 previously-failing checks. All remaining failures are pre-existing on main, caused by a stale single-header and test_issue43_phase2_persistence assertion mismatch.

Note: The ASan/UBSan misaligned pstringview access has been fixed in the multi-header source (forest_domain_mixin.inc uses memcpy instead of direct struct member assignment), but the fix does not appear in CI because tests compile against the stale single-header. Once the single-header is regenerated, this fix will take effect.

konard · 2026-04-11T00:49:40Z

🔄 Auto-restart triggered (iteration 2)

Reason: CI failures detected

Starting new session to address the issues.

Auto-restart-until-mergeable mode is active. Will continue until PR becomes mergeable.

…tkeep80#241) rebuild_free_tree was resetting AVL fields for ALL blocks, including allocated blocks whose TreeNode fields store user-level AVL tree structure (symbol tree, pmap, etc). This caused bootstrap symbols to be re-allocated on every load(), leading to the pre-existing test_issue43_phase2_persistence failure (alloc_block_count: 20 == 12). Fix: only reset AVL fields for free blocks (weight == 0). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ep80#241) The single-header files (pmm.h, pmm_no_comments.h) were stale and did not include the forest registry bootstrap code. This caused the "Single-header presets up-to-date" CI check to fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

) With the bootstrap code now included in the single-header files, tests that used preset headers now see bootstrap-allocated blocks. Updated 18 test files to capture a baseline alloc_block_count() after create() instead of hardcoding expected values like 1, 2, 3. Also increased buffer sizes for small static storage tests (test_issue146_index_sizes, test_issue175_64bit_indexes, test_issue146_sh_small_embedded_static) since the bootstrap requires more memory than previously available in small buffers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

konard · 2026-04-11T01:31:14Z

🔄 Auto-restart-until-mergeable Log (iteration 2)

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $21.999854

📊 Context and tokens usage:

Claude Opus 4.6:

Context window: 150.5K / 1M (15%) input tokens, 77.3K / 128K (60%) output tokens

Total: (267.5K + 36.4M cached) input tokens, 77.3K output tokens, $21.824588 cost

Claude Haiku 4.5:

Total: (63.8K + 682.8K cached) input tokens, 5.4K / 64K (9%) output tokens, $0.175266 cost

🤖 Models used:

Tool: Anthropic Claude Code
Requested: opus
Main model: Claude Opus 4.6 (claude-opus-4-6)
Additional models:
- Claude Haiku 4.5 (claude-haiku-4-5-20251001)

📎 Log file uploaded as Gist (13519KB)

View complete solution draft log

Now working session is ended, feel free to review and add any feedback on the solution draft.

konard · 2026-04-11T01:36:35Z

✅ Ready to merge

This pull request is now ready to be merged:

All CI checks have passed
No merge conflicts
No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

This reverts commit c2596f3.

konard and others added 3 commits April 10, 2026 22:59

Initial commit with task details

c2596f3

Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: netkeep80#241

konard changed the title ~~[WIP] Bootstrap a new PAP image with free-tree and system domains~~ feat(#241): bootstrap invariant verification and documentation Apr 10, 2026

konard marked this pull request as ready for review April 10, 2026 23:17

Fix clang-format violations across the codebase (Issue netkeep80#241)

1e78daf

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

konard and others added 2 commits April 11, 2026 00:19

konard and others added 3 commits April 11, 2026 01:21

Revert "Initial commit with task details"

819bf1d

This reverts commit c2596f3.

netkeep80 merged commit 2ef8f29 into netkeep80:main Apr 11, 2026
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(#241): bootstrap invariant verification and documentation#242

feat(#241): bootstrap invariant verification and documentation#242
netkeep80 merged 10 commits intonetkeep80:mainfrom
konard:issue-241-158d0477a436

konard commented Apr 10, 2026 •

edited

Loading

Uh oh!

konard commented Apr 10, 2026

Uh oh!

konard commented Apr 10, 2026

Uh oh!

netkeep80 commented Apr 10, 2026

Uh oh!

konard commented Apr 11, 2026

Uh oh!

konard commented Apr 11, 2026

Uh oh!

konard commented Apr 11, 2026

Uh oh!

konard commented Apr 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

konard commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Invariants checked by validate_bootstrap_invariants()

Bug fix: rebuild_free_tree corrupting user AVL trees (pre-existing on main)

Single-header regeneration and test updates

Other CI fixes in this PR

Files changed

Test plan

Uh oh!

konard commented Apr 10, 2026

🤖 Solution Draft Log

💰 Cost: $7.032258

📊 Context and tokens usage:

🤖 Models used:

📎 Log file uploaded as Gist (3024KB)

Uh oh!

konard commented Apr 10, 2026

🔄 Auto-restart triggered (iteration 1)

Uh oh!

netkeep80 commented Apr 10, 2026

Align free-tree implementation with the general forest model

Summary

Why

Scope

Out of scope

Acceptance criteria

Dependencies

Uh oh!

konard commented Apr 11, 2026

Final CI Results (run 24270494864)

Fixed by this PR (were failing on main, now passing)

Unchanged (passing on both main and PR)

Pre-existing failures (same on main 24267900208)

Uh oh!

konard commented Apr 11, 2026

🔄 Auto-restart triggered (iteration 2)

Uh oh!

konard commented Apr 11, 2026

🔄 Auto-restart-until-mergeable Log (iteration 2)

💰 Cost: $21.999854

📊 Context and tokens usage:

🤖 Models used:

📎 Log file uploaded as Gist (13519KB)

Uh oh!

konard commented Apr 11, 2026

✅ Ready to merge

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

konard commented Apr 10, 2026 •

edited

Loading

Invariants checked by `validate_bootstrap_invariants()`