Skip to content

feat: index Reader LM 1.5B + BGE-M3 for OSINT pipeline jinaai/reader-lm-1.5b (safetensors, 1 shard, 3.1 GB): HTML→Markdown local model. No Jina API needed. bgz7 → palette → O(1) HTML structure recognition. CompendiumLabs/bge-m3-gguf (GGUF F16, ~1.2 GB): Multilingual embedding model. Replaces DeepNSM for non-English. bgz7 → palette → O(1) semantic similarity. Together: Reader LM reads the web, BGE-M3 embeds it, AriGraph stores it as SPO triplets, AutocompleteCache routes it at 17K tok/sec. https://claude.ai/code/session_01M3at4EuHVvQ8S95mSnKgtK #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AdaWorldAPI merged 3 commits into from
Mar 31, 2026
+88 −491
Merged

feat: index Reader LM 1.5B + BGE-M3 for OSINT pipeline jinaai/reader-lm-1.5b (safetensors, 1 shard, 3.1 GB): HTML→Markdown local model. No Jina API needed. bgz7 → palette → O(1) HTML structure recognition. CompendiumLabs/bge-m3-gguf (GGUF F16, ~1.2 GB): Multilingual embedding model. Replaces DeepNSM for non-English. bgz7 → palette → O(1) semantic similarity. Together: Reader LM reads the web, BGE-M3 embeds it, AriGraph stores it as SPO triplets, AutocompleteCache routes it at 17K tok/sec. https://claude.ai/code/session_01M3at4EuHVvQ8S95mSnKgtK #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
af23e01
remove: axum serve binary from ndarray — REST server belongs in lance…
claude Mar 31, 2026
2dd417a
docs: update CLAUDE.md with session results + architecture rules
claude Mar 31, 2026
dd7fa49
feat: index Reader LM 1.5B + BGE-M3 for OSINT pipeline
claude Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,26 @@ src/
- **880 lib tests passing**, 2 doctest failures out of 302
- **Build currently fails (exit 101)** — needs investigation
- See blackboard for detailed per-module test counts

## Session: Qwen3.5 × Opus 4.5/4.6 Reverse Engineering (2026-03-31)

### New Modules
- `src/hpc/styles/` — 34 cognitive primitives (rte, htd, smad, tcp, irs, mcp, tca, cdt, mct, lsi, pso, cdi, cws, are, tcf, ssr, etd, amp, zcf, hpm, cur, mpc, ssam, idr, spp, icr, sdd, dtmf, hkf). Each is `fn(Base17, NarsTruth) → result`. 49 tests.
- `src/hpc/causal_diff.rs` — CausalEdge64 (u64 packed), scaffold_to_palette3d_layers(), quality scoring (GOOD/BAD/UNCERTAIN), NARS self-reinforcement LoRA, PAL8 serialization (4101 bytes).
- `.cargo/config.toml` — `target-cpu=x86-64-v4` (AVX-512 mandatory).
- `src/simd.rs` — compile-time AVX-512 dispatch via `cfg(target_feature = "avx512f")`.

### Key Data
- 5 Qwen3.5 models indexed: 685 MB bgz7 from 201 GB BF16 safetensors
- GitHub Release `v0.1.0-bgz-data` on AdaWorldAPI/lance-graph: 41 bgz7 files
- 4 diffs: FfnGate dominant (0.6%), v2 reverts v1, K stable at 27B, K shifted at 9B

### Benchmark
- SPO Palette Distance: 611M lookups/sec, 1.8 ns/lookup, 388 KB RAM
- 17K tokens/sec (triple model, 4096 heads, Pearl 2³)

### Architecture Rule
- ndarray = hardware (SIMD, Palette, Base17, SpoDistanceMatrices, read_bgz7_file)
- lance-graph = thinking (NarsTruth, NarsEngine, TripleModel, AutocompleteCache)
- causal-edge = protocol (CausalEdge64, NarsTables, forward/learn)
- p64 = convergence highway (both repos meet here)
Loading
Loading