Skip to content

feat: Croissant ML dataset discovery via federation-native PAP agents#291

Merged
toadkicker merged 3 commits intomainfrom
feat/455e-building-on-more
Apr 15, 2026
Merged

feat: Croissant ML dataset discovery via federation-native PAP agents#291
toadkicker merged 3 commits intomainfrom
feat/455e-building-on-more

Conversation

@toadkicker
Copy link
Copy Markdown
Contributor

Summary

  • Federation-native dataset search: Two new PAP agents (schema:DatasetAction) for HuggingFace Dataset Search and OpenML Dataset Search — pure TOML, zero new Rust trait code, auto-discovered by FederatedRegistry.query_local_satisfiable()
  • Parallel JoinSet handshake fan-out: All registered zero-disclosure dataset agents run the full 6-phase PAP handshake concurrently; results merged by preference-biased relevance score
  • Long-horizon memex integration: FTS5 prior-art check with 7-day TTL emits RestoredFromMemex immediately while fresh handshakes run in parallel; per-agent and aggregate episodes recorded; PreferenceEngine signals fed back for future ranking
  • DatasetSearchTemplate block renderer: Reads schema:ItemList JSON-LD, shows memex provenance banner, source-agent badges, open-license tint, and receipt provenance footer
  • Intent expansion: 47 tests (up from 32), new schema:DatasetAction rule, free-dictionary coverage, strip-behaviour and ordering-conflict regression tests
  • BlockUpdate migration: Adapted all 6 emit sites to the BlockUpdate wire type from refactor: BlockUpdate typed events + BlockContext provider pattern #289 (reviewed the refactor before applying — it's correct: compiler-enforced separation of backend vs. frontend-only state)

Architecture notes

Adding a new dataset source = add one TOML file. No Rust changes required. Federation self-configuration: publish_agent(did, "pap://shared-registry") makes any Papillon instance auto-discover it via FederatedRegistry.merge_remote().

Test plan

  • cargo test --workspace passes (271 shared-crate tests, all intent tests including 15 new ones)
  • cargo clippy --workspace --all-targets -- -D warnings clean
  • cargo fmt --all -- --check clean
  • cargo check --target wasm32-unknown-unknown clean in apps/papillon/frontend
  • CI smoke-test (trunk WASM build + Playwright) passes
  • CI chrysalis E2E passes

🤖 Generated with Claude Code

Todd Baur and others added 3 commits April 15, 2026 15:44
Adds parallel multi-source ML dataset discovery routed through the full
6-phase PAP handshake. A new `schema:DatasetAction` intent routes dataset
queries to all matching zero-disclosure agents concurrently via JoinSet,
rather than picking a single top-scored agent.

Architecture:
- Two TOML catalog agents (HuggingFace Hub, OpenML) require zero disclosure
  and self-configure via the existing FederatedRegistry — adding a third
  source requires only a new .toml file, no Rust changes
- `canvas_discover_datasets` fans out to all `schema:DatasetAction` agents
  in parallel, merges results sorted by preference-blended relevance score,
  and emits progressive `block_updated` events during the JoinSet fan-out
- Long-horizon memex integration: FTS5 prior-art check (7-day TTL) surfaces
  cached results immediately while fresh handshakes run; per-agent episodes
  feed PreferenceEngine signals so the system learns provider preference over time
- `DatasetState` Leptos context (keyed by block_id) carries discovery phase
  FSM and memex hints to the `DatasetSearchTemplate` block renderer
- Mandate TTL set to 7 days — dataset metadata is durable; the existing
  Active→Degraded→ReadOnly decay UI prompts re-query naturally

New files: dataset_types.rs, dataset_discovery.rs, dataset_template.rs,
           dataset.rs (state), huggingface_datasets.toml, openml_datasets.toml

All 385 tests passing; papillon-shared, papillon backend, and papillon-frontend
WASM targets compile clean.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Closes gaps identified in intent rule coverage:

- Free Dictionary: 3 new tests (define/meaning of/definition of) — rule had
  zero coverage previously; any regression would have gone undetected
- Clean-query stripping: 3 tests verify the third return value of detect_intent()
  (previously almost never checked); confirms "dataset sentiment analysis" strips
  to "sentiment analysis", "define photosynthesis" to "photosynthesis", etc.
- Rule ordering conflicts: 3 tests document first-match-wins behavior for
  ambiguous inputs (weather beats dataset, dataset beats books, dataset beats
  arXiv) — establishes intended semantics and guards against rule reordering
- Missing keywords: 2 tests for "temperature" (weather rule) and "population of"
  (REST Countries rule) which were declared in RULES but never exercised

All 47 intent tests pass; 0 regressions.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
After rebasing on main (BlockUpdate typed events + BlockContext provider
pattern), the 6 emit sites in dataset_discovery.rs used the old CanvasBlock
struct in BlockEvent — now expects BlockUpdate.

Changes:
- Replace CanvasBlock with BlockUpdate at all 6 emit sites, dropping
  linked_block_ids (frontend-only field, absent from BlockUpdate by design)
- Introduce HandshakeConfig struct to reduce run_dataset_handshake from 8
  to 5 args (fixes clippy::too_many_arguments)
- Add HandshakeTaskResult type alias for JoinSet element type
  (fixes clippy::type_complexity)
- Use *= for relevance_score blending (fixes clippy::assign_op_pattern)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

Benchmark Regression Report

PAP Protocol Benchmark Regression Check
========================================
Baseline: .bench-baseline/baseline.json
Threshold: 30%

  ed25519_keypair_generation                19.7 µs  (baseline: 19.7 µs, +0.0%)  [ok]
  did_key_derivation                         1.5 µs  (baseline: 1.5 µs, +0.0%)  [ok]
  mandate_create_sign                       24.0 µs  (baseline: 24.0 µs, -0.2%)  [ok]
  mandate_chain_verify_depth3              125.1 µs  (baseline: 126.0 µs, -0.7%)  [ok]
  sd_jwt_issue_5claims                      28.0 µs  (baseline: 27.9 µs, +0.3%)  [ok]
  sd_jwt_verify_disclose_3of5               44.1 µs  (baseline: 44.5 µs, -1.0%)  [ok]
  session_open_full_lifecycle              111.9 µs  (baseline: 108.6 µs, +3.0%)  [ok]
  receipt_create_cosign                     47.8 µs  (baseline: 49.0 µs, -2.5%)  [ok]
  federation_announce_local                 55.9 µs  (baseline: 55.9 µs, +0.0%)  [ok]

All benchmarks within 30% of baseline.

Threshold: 10% regression vs baseline from main

@toadkicker toadkicker merged commit 34d243f into main Apr 15, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant