Axiom Engine v0.2.1-beta.1 – Fragment-Aware Idle Mesh

Highlights

• Fragment‑aware ledger and pruning: facts now carry explicit fragment metadata, surfaced in tooling and used by pruning.
• Smarter idle cycles: deterministic idle suite with per‑node visibility, fragment auditing, and health anomaly detection.
• More robust P2P + chain sync: better logging, chain alignment, and data‑quality checks across peers.
• Standalone build improvements: fixed NLP model loading inside PyInstaller binaries; updated multi‑OS build script.
• New docs: clarified how to run nodes, inspect the ledger, and understand pruning and metacognition.

---

Core changes

1. Fragment‑aware facts and pruning

• Schema changes in ledger.py

  • facts now includes:
    • fragment_state TEXT NOT NULL DEFAULT 'unknown'
    • fragment_score REAL NOT NULL DEFAULT 0.0
    • fragment_reason TEXT
  • New index: idx_facts_fragment_state (created safely after migration).
  • initialize_database() runs defensive ALTER TABLE steps so existing DBs are migrated in place.
  • insert_uncorroborated_fact(...) accepts and stores:
    • fragment_state, fragment_score, fragment_reason.

• Extractor upgrades in crucible.py

  • New deterministic helper _compute_fragment_metadata(doc, raw_sent) computes:
    • A rule‑based fragment_score using:
      • Short or moderately short sentences.
      • Missing named entities.
      • Pronoun‑leading sentences (“he”, “she”, “they”, “it”, “this”, “that”, …).
      • Non‑terminal punctuation / lowercase starts.
    • A fragment_state ('suspected_fragment' or 'unknown') and a human‑readable fragment_reason.
  • extract_facts_from_text():
    • Still generates ADL summaries (_generate_adl_summary).
    • Stores fragment metadata with each new fact via insert_uncorroborated_fact.

• View and inspect fragments (view_ledger.py)

  • Stats header now shows (when columns exist):
    • fragments (suspect)
    • fragments (confirmed)
  • Recent record listing:
    • Uses fragment metadata to render integrity:
      • FRAGMENT! for confirmed_fragment.
      • FRAGMENT? for suspected_fragment or fragment_score ≥ 0.5.
      • Falls back to the legacy length heuristic if fragment metadata is absent.

• Idle fragment audit in node.py

  • New idle task _idle_fragment_audit:
    • Runs at most once every 5 minutes per node.
    • Randomly samples up to 40 non‑disputed facts.
    • Decompresses content and applies model‑free heuristics (length, pronoun start, punctuation).
    • Updates fragment_state, fragment_score, fragment_reason when they materially change.
    • Logs:
      • "[Idle-Fragment:<port>] Audited N fact(s); updated classifications for M."

• Cross‑node fragment consensus

  • New endpoint GET /fragment_opinion?fact_id=...:
    • Returns { seen, status, trust_score, fragment_state, fragment_score }.
  • _idle_fragment_audit:
    • For suspected_fragment facts, queries up to 3 peers at /fragment_opinion.
    • If peers either don’t know the fact or also mark it fragment‑like → promotes to confirmed_fragment.
    • If peers treat it as healthy (rejected_fragment or trusted with good trust) → demotes to rejected_fragment.
    • Entirely rule‑based; no LLM/ML involvement.

• Metacognitive pruning (metacognitive_engine.py)

  • prune_integrity_check(...) now considers fragment_state:
    • Candidates: facts older than 90 days with trust_score ≤ 2.
    • Delete when:
      • ADL summary is too short (len(adl_summary) < ADL_INTEGRITY_THRESHOLD), or
      • fragment_state == 'confirmed_fragment'.
  • This preferentially purges stale, low‑trust confirmed fragments and shallow facts.

---

2. Idle cycles, health, and diagnostics

• Idle suite in node.py

  • Background loop now:
    • Runs the main ingestion cycle every AXIOM_MAIN_CYCLE_INTERVAL (default 900s).
    • Runs an idle suite every AXIOM_IDLE_SUITE_INTERVAL (default 30s).
  • Idle tasks (each with its own throttle):
    • _idle_learning_cycle
    • _idle_conversation_training
    • _idle_code_introspection
    • _idle_data_quality
    • _idle_fragment_audit
    • _idle_health_snapshot
    • _idle_self_checks

• Per‑node visibility and cleaner logging

  • Logs include node port, e.g. [Idle-Health:8009], [Idle-Data:8010].
  • New helper _maybe_log_idle_throttle(...):
    • Logs throttle decisions at debug level, rate‑limited to once per 60s per task.
  • Idle suite logging:
    • "[Idle-Suite:<port>] Start." is only logged when at least one idle task actually runs.
    • "[Idle-Suite:<port>] End." is emitted only in that case; fully throttled suites stay silent.

• Health anomaly detection

  • _idle_health_snapshot:
    • Writes snapshots via compute_health_snapshot(db_path).
    • If total_blocks > 0 and total_facts == 0, logs a warning pointing at a possible sync anomaly (“chain without facts”).

• Idle diagnostics endpoint

  • GET /debug/idle_state returns:
    • node_port, node_role, advertised_url, db_path.
    • main_cycle_interval_sec, idle_suite_interval_sec.
    • Ages since last:
      • main cycle, idle learning, code introspection, data quality.
      • fragment audit, health snapshot, self‑checks.

---

3. P2P sync and chain robustness

• Ledger sync (p2p.py)

  • sync_with_peer(...):
    • Verifies hashes using decompressed text.
    • Compresses incoming fact content with zlib before inserting into facts.fact_content.
    • Logs and skips facts when compression fails, instead of inserting bad data.

• Chain sync (p2p.py + node.py)

  • sync_chain_with_peer(...) uses the same DB path as the node, ensuring local chain height reflects the correct ledger file.
  • bootstrap_sync() and background sync in _run_main_cycle():
    • No longer swallow exceptions; log warnings with peer URL and error details.

---

4. Standalone build + NLP model loading

• Build script (build_standalone.py)

  • Version bumped to 0.2.1-beta.1:
    • get_version() reads from pyproject.toml with fallback to 0.2.1-beta.1.
    • macOS DMG names include the version and arch: Axiom_Node_v0.2.1-beta.1_<arch>.dmg.
  • Cross‑platform behavior:
    • macOS / Linux: builds dist/AxiomNode.
    • Windows: builds dist/AxiomNode.exe.
  • Final instructions now print the actual executable path (./dist/AxiomNode or ./dist/AxiomNode.exe).

• NLP loader (axiom_model_loader.py)

  • New load_nlp_model() strategy:
    • First tries import en_core_web_sm; en_core_web_sm.load().
    • Falls back to spacy.load("en_core_web_sm").
  • Works in both:
    • Normal Python environments.
    • PyInstaller one‑file builds, as long as en_core_web_sm is installed when building.

---

Documentation updates

New / updated docs under docs/:

• idle-cycles-and-behavior.md
  • Describes the idle suite, per‑node tagged logs, /debug/idle_state, and health anomaly detection.
• ledger-pruning-refinement-fragment-awareness.md
  • Explains fragment metadata, idle fragment audits, and fragment‑aware pruning.
• metacognitive-engine.md (updated)
  • Now documents pruning of both shallow‑ADL and confirmed_fragment facts.
• running-nodes-and-ledger-tools.md (new)
  • How bootstrap vs peers choose DB paths.
  • How to start nodes with PORT, AXIOM_DB_PATH, BOOTSTRAP_PEER.
  • How the idle suite works and how to inspect state via /debug/idle_state.
  • Usage of view_ledger.py (--stats, --brain, --limit, --db).
  • Operational guidance for pruning, fragments, and when (not) to reset DBs.
• Existing conceptual docs (blockchain.md, p2p-mesh-verification.md, explain-synapse-weighting.md) remain valid and describe the blockchain layer, mesh sync behavior, and deterministic synapse‑based reasoning.

---

Upgrade notes

• Database migration

  • No manual migration required:
    • On startup, each node will:
      • Ensure all tables and indexes exist.
      • Add fragment columns if missing.
      • Compress legacy plaintext facts via migrate_fact_content_to_compressed().
  • You do not need to delete existing axiom_ledger*.db unless you explicitly want a clean corpus.

• Standalone builds

  • Before running build_standalone.py, ensure:
    • en_core_web_sm is installed in the build environment.
    • uv is available (recommended) or PyInstaller is installed.
  • Build on each target OS separately (no cross‑compiling).

---

Checks

• Linting passes on:
  • node.py
  • ledger.py
  • p2p.py
  • crucible.py
  • metacognitive_engine.py
  • view_ledger.py