Skip to content

feat(P2): Daydreamer background consolidation, community detection, and P2P sharing#71

Merged
devlux76 merged 14 commits intomainfrom
copilot/p2-subtask-issues
Mar 14, 2026
Merged

feat(P2): Daydreamer background consolidation, community detection, and P2P sharing#71
devlux76 merged 14 commits intomainfrom
copilot/p2-subtask-issues

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 13, 2026

Implements all P2 milestone tasks: cooperative background memory consolidation (Daydreamer), label-propagation community detection with quota enforcement, and privacy-safe P2P curiosity/sharing infrastructure.

MetadataStore interface extensions

  • getAllVolumes(), getAllShelves() — needed by FullNeighborRecalc and PrototypeRecomputer to enumerate hierarchy
  • deleteEdge(from, to) — required for Hebbian pruning
  • core/crypto/uuid.tsrandomUUID() with crypto.randomUUID() + fallback

Daydreamer modules (daydreamer/)

  • IdleScheduler — cooperative task loop via requestIdleCallback / setImmediate; priority queue, FIFO tie-break, error-isolated per task
  • HebbianUpdater — LTP (strengthenEdges), LTD+pruning (decayAndPrune); enforces maxDegree; triggers batchComputeSalience + runPromotionSweep for changed nodes
  • FullNeighborRecalc — pairwise cosine recalc for dirty volumes; pair budget bounded by computeCapacity(graphMass) per idle cycle (O(√(t log t))); clears dirty flag, recomputes salience
  • PrototypeRecomputer — volume medoid selection (minimises avg pairwise distance) and centroid, then shelf routing prototypes; appends to VectorStore; runs tier-quota sweep
  • ClusterStability — deterministic label propagation on MetroidNeighbor graph; persists communityIdPageActivity; exports detectOversizedCommunities / detectEmptyCommunities

Sharing modules (sharing/)

  • types.tsCuriosityProbe (with required mimeType + modelUrn fields), GraphFragment, SubgraphSlice, EligibilityDecision
  • EligibilityClassifier — regex-based PII/credential detection (credentials, credit card, IBAN, SSN, email, medical); deterministic with typed reason codes; classifyPage / filterEligible
  • SubgraphExporter — BFS expansion from seed nodes, eligibility filter, creator identity stripped, provenance tagged; exportForProbe + exportForExchange
  • SubgraphImporter — schema validation, sender identity stripping, safe page+edge import; importFragment / importSlice
  • CuriosityBroadcaster — rate-limited probe queue; dispatches incoming graph_fragment responses to registered handler
  • PeerExchange — opt-in signed subgraph exchange; inject any P2PTransport implementation
// Curiosity broadcast flow
broadcaster.enqueueProbe({ m1, partialMetroid, queryContextB64, knowledgeBoundary, mimeType, modelUrn, timestamp });
await broadcaster.flush(Date.now()); // rate-limited; sends one probe per call

// Import received fragment — identity-stripped, schema-validated
const result = await importFragment(fragment, { metadataStore, vectorStore });
// result: { nodesImported, edgesImported, rejected }

Tests

93 new tests across 8 test files (tests/daydreamer/, tests/sharing/, tests/integration/Daydreamer.test.ts). Integration test validates Williams Bound invariant (resident count ≤ H(t)) holds after multiple Daydreamer passes.

Original prompt

This section details on the original issue you should resolve

<issue_title>P2</issue_title>
<issue_description>All P2 tasks, issues and sub issues belong here, either directly or by reference. Please subtask every issue with P2 in its name for ease of tracking.</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

Copilot AI and others added 2 commits March 13, 2026 22:37
…Sharing modules

Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>
…bility

Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>
Copilot AI changed the title [WIP] Subtask P2 issues for better tracking feat(P2): Daydreamer background consolidation, community detection, and P2P sharing Mar 13, 2026
Copilot AI requested a review from devlux76 March 13, 2026 22:41
@devlux76 devlux76 mentioned this pull request Mar 14, 2026
Closed
@github-actions github-actions bot added layer: foundation Core types, model profiles, crypto layer: storage OPFS vector store and IndexedDB metadata store layer: daydreamer Background consolidation (LTP/LTD, recalc) layer: testing Test coverage and integration tests layer: documentation API docs, developer guide, architecture diagrams labels Mar 14, 2026
@devlux76 devlux76 marked this pull request as ready for review March 14, 2026 00:40
Copilot AI review requested due to automatic review settings March 14, 2026 00:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements major Daydreamer background-maintenance modules and introduces a first pass at privacy-safe P2P sharing (curiosity probes + subgraph exchange), along with required MetadataStore API extensions and comprehensive Vitest coverage.

Changes:

  • Add Daydreamer modules: idle scheduling, Hebbian edge updates, full neighbor-graph recalculation, prototype recomputation, and community detection via label propagation.
  • Add sharing modules: eligibility classification, curiosity probe broadcasting, subgraph export/import, and peer exchange; centralize sharing schema in sharing/types.ts.
  • Extend MetadataStore with volume/shelf enumeration and single-edge deletion; update IndexedDB backend and tests accordingly.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
TODO.md Marks P2 Daydreamer + sharing tasks as completed.
PLAN.md Updates implementation status tables/exit criteria for Daydreamer + sharing.
core/types.ts Extends MetadataStore with getAllVolumes, getAllShelves, and deleteEdge.
storage/IndexedDbMetadataStore.ts Implements getAllVolumes, getAllShelves, and deleteEdge for IndexedDB backend.
core/crypto/uuid.ts Adds randomUUID() helper with platform + fallback implementations.
daydreamer/IdleScheduler.ts Adds cooperative idle-time scheduler with priority queue.
daydreamer/HebbianUpdater.ts Adds LTP/LTD/pruning and triggers salience recompute + promotion sweep.
daydreamer/FullNeighborRecalc.ts Adds dirty-volume full neighbor recomputation bounded by a budget.
daydreamer/PrototypeRecomputer.ts Adds recomputation of volume/shelf prototypes and attempts to update hotpath.
daydreamer/ClusterStability.ts Adds label propagation community detection + helper detectors.
sharing/types.ts Introduces canonical types for probes, fragments, slices, and transport messages.
sharing/EligibilityClassifier.ts Adds deterministic eligibility rules to block PII/credentials/etc.
sharing/SubgraphExporter.ts Exports eligibility-filtered slices using BFS expansion.
sharing/SubgraphImporter.ts Imports fragments/slices with schema validation + identity stripping.
sharing/PeerExchange.ts Adds opt-in proactive slice exchange over transport.
sharing/CuriosityBroadcaster.ts Adds rate-limited probe queue + fragment handler dispatch.
tests/sharing/*.test.ts Adds tests for eligibility, broadcaster, and subgraph exchange round-trip.
tests/daydreamer/*.test.ts Adds tests for idle scheduler, Hebbian updater, full recalc, prototypes, and cluster stability.
tests/integration/Daydreamer.test.ts Adds integration coverage for Daydreamer passes using fake-indexeddb.
tests/SalienceEngine.test.ts Updates MetadataStore mock to satisfy new interface methods.

devlux76 and others added 11 commits March 13, 2026 19:10
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@devlux76 devlux76 merged commit 5cde29f into main Mar 14, 2026
1 of 3 checks passed
@devlux76 devlux76 deleted the copilot/p2-subtask-issues branch March 14, 2026 01:23
@devlux76 devlux76 mentioned this pull request Mar 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

layer: daydreamer Background consolidation (LTP/LTD, recalc) layer: documentation API docs, developer guide, architecture diagrams layer: foundation Core types, model profiles, crypto layer: storage OPFS vector store and IndexedDB metadata store layer: testing Test coverage and integration tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P2

3 participants