SOP v5.3.8 - Production Resilience, Multi-Language Alignment, and AI Memory Release

Changelog

v5.3.8

Highlights

• Full ecosystem version alignment: Synchronized all core Go modules and language bindings (Python sop4py, C# Sop, Java, Rust) to release version v5.3.8.
• Zero known vulnerabilities: Pinned Go toolchain 1.26.8 across all modules and Docker containers, resolving 7 standard library CVEs and verifying 0 called vulnerabilities with govulncheck.
• Deterministic container and CI test validation: Resolved direct I/O failover simulation in root/container environments and restored full race-detector test coverage in CI.

Reliability

• Container test runner hardening: Updated Test_EC_Failover_Reinstate_FastForward_Short in infs/integrationtests to utilize the DirectIO simulator and fs.TriggerFailover instead of filesystem chmod manipulations, eliminating permission bypasses in container and root execution contexts.
• Race detector coverage: Verified clean execution across core packages under Go race detection (inmemory, btree, common, fs, cache, cel, encoding, database, jsondb, search).

Performance

• Reproducible cache latency: Verified 62.09 ns/op L1 MRU hit latency and 66.17 us/op under heavy mixed-cache workloads.
• Vector store throughput: Benchmarked end-to-end ACID transaction pipeline in ai/vector at 16.23 ms for batch 50-vector upsert, KNN query, and durable commit.
• CI benchmark stabilization: Calibrated benchmark iteration scaling in performance workflows to prevent runner timeouts.

Developer Experience

• Quickstart zero-dependency demo: Validated examples/quickstart and published container image (ghcr.io/sharedcode/sop-quickstart) providing instant zero-configuration testing.
• Unified versioning: Enhanced scripts/update_version.sh for atomic multi-language version updates.

AI/Data

• Knowledge Base compilation: Verified and generated complete base knowledge base in ai/sop_base_knowledge.json containing 1,257 knowledge items and 1,397 categories.
• Query and Join execution: Verified Adaptive Hash Join planning in AI agent runtime executing in 1.10 ms.

Security

• Go toolchain update: Upgraded all modules and Dockerfiles to Go 1.26.8, addressing CVEs in net/url, html/template, crypto/tls, encoding/asn1, net/http, and golang.org/x/net/idna.
• Dependency hardening: Bumped go-git to v5.19.2 and cel-go to v0.29.0 to eliminate open Dependabot advisories.

CI/CD

• Gated delivery pipeline: Decoupled production promotion into .github/workflows/promote.yml so pending manual approvals do not block or fail continuous delivery status.
• Test execution fix: Explicitly configured package test targets in .github/workflows/go.yml for comprehensive coverage reporting with race detector.

Documentation

• Badge cleanup: Removed retired Go Report Card badge, added dynamically resolved Go version, license, and CI build status badges.
• Architecture and Investor analysis: Added detailed architectural positioning ("Why SOP?") and multi-horizon product roadmap.

Breaking Changes

• None. Full backward compatibility maintained for all storage layouts, transaction protocols, and APIs.

Upgrade Notes

• Go 1.26.8 or later is recommended when building from source. No data migration required for existing stores.

v5.3.7

• Descending iterators for the in-memory B-Tree: AllDesc() and RangeDesc(from, to) walk keys newest-first; RangeDesc seeks straight to the high bound. Both covered by unit tests; quickstart shows a newest-first scan.
• Fix: iteration errors no longer swallowed in the Data Manager item stream (tools/httpserver): a shadowed err inside the paging loop meant store.Next failures never reached the error log or terminated the loop condition.
• Fix: tools/confighub knowledge-base tests now skip on fresh clones instead of failing when the gitignored local tools/config.json is absent.
• Security: jackson-databind 2.21.4 to 2.21.5 in the Java binding (closes the case-insensitive deserialization bypass advisory, the last open Dependabot alert).
• Lint cleanup: removed a stray debug print in the item search path, replaced nil Contexts with context.Background() in the in-memory B-Tree wrapper, finished the typed context key for the vectorized-spaces map, normalized error strings, dropped dead nil checks.

v5.3.6

• Maintenance rebuild: fixed v5.3.5/v5.3.6 release pipeline failures by building the whole knowledge_compiler package and bumping the build image to Go 1.26.4. No library code changes.

v5.3.5

• B-Tree node slot allocation and L1 cache handling optimized for better throughput on hot paths.
• L2 cache eviction policy reworked for standalone mode -- smarter eviction under memory pressure.
• NewBtree/OpenBtree safe multi-open: the same named B-Tree can now be opened multiple times within a single transaction without data races. Stress tests updated to cover this.
• Range-over-func iterators for the in-memory B-Tree: All() and Range(from, to) return iter.Seq2 so callers can for k, v := range b3.Range(102, 104). Range seeks straight to the start key; both are covered by unit tests. Quickstart example and demo GIF updated to use them.
• Security: cleared all 32 Dependabot alerts (7 critical). Go: golang.org/x/crypto 0.52.0, golang.org/x/net 0.55.0, go-git/v5 5.19.1, go-billy/v5 5.9.0, cloudflare/circl 1.6.3. Java binding: jackson-databind 2.19.0 to 2.21.4 (fixes CVE PolymorphicTypeValidator bypass, array subtype allowlist bypass, InetSocketAddress SSRF, case-insensitive deserialization bypass).
• Gated delivery pipeline (.github/workflows/deliver.yml): every push to master runs build, tests, container packaging to GHCR (sop-quickstart), and a staging smoke test. Production promotion (image :stable tag plus GitHub Pages site deploy) sits behind a manual approval on the production environment.
• Quickstart example (examples/quickstart): zero-infrastructure in-memory B-Tree walkthrough (add, find, update, ordered scan). Packaged as a distroless container via Dockerfile.quickstart.
• README demo GIF recorded from the quickstart run; project site landing page added (index.md).
• Root directory cleanup: 26 documentation files moved to docs/, 13 shell scripts moved to scripts/. README and workflow references updated. Root now contains only source code, standard project files (LICENSE, CONTRIBUTING, etc.), and build configs.

SOP V2 build 54 (Upcoming)

• Gate 1 Advanced KB Routing: Major enhancements to specialized focused routing for knowledge base queries.
  • Root Category Navigation: Query omni:<kb> to display all root categories with item counts and subcategory information.
    • Example: omni:sop shows all top-level categories in the SOP knowledge base
    • Provides directory-style exploration without needing to know category names upfront
    • Navigation hints included for deeper exploration (e.g., "Navigate: omni:sop:language")
    • Pagination: 20 categories per page with :page:<number> or /page/<number> syntax
      • omni:sop:page:2 or omni:sop/page/2 - View page 2 of root categories
      • omni:sop:language:page:3 or omni:sop/language/page/3 - View page 3 of subcategories
      • Supports both : and / as separators (matches user's query style)
      • Shows page info: "(Page 2 of 5, showing 21-40 of 87)"
      • Navigation hints: "Previous: omni:sop:page:1 | Next: omni:sop:page:3"
      • LLM filtering suggestion for large sets (>40 categories)
  • :llm <instruction> Meta-Token: Added support for explicit LLM post-processing instructions using :llm suffix (e.g., omni:sop:operations:performance:llm summarize top 3).
    • Clean Query Separation: The :llm meta-token is automatically stripped from the KB search query and treated as post-retrieval guidance.
    • TaskContextClassification Fields: Added CleanQuery and LLMInstruction fields to properly separate user intent from meta-commands.
    • Three-Way Routing: Intelligent decision-making based on result count and :llm presence:
      • :llm present → LLM processes with instruction (highest priority)
      • 1-5 matches → Direct display (no LLM)
      • 6+ matches → Automatic LLM summarization
  • Flexible Hierarchy Support: Full support for any-depth category paths (e.g., omni:kb:cat1:subcat1.1:subsubcat1.1.1:...).
  • Subcategory Navigation: When a category path has no direct items (and no :llm instruction), automatically returns child categories with item counts and descriptions as navigation hints.
  • Enhanced Parsing: New stripLLMInstruction() function ensures consistent meta-token extraction across all query patterns.
  • Architecture Improvements:
    • getSubcategories() function for root and path-level category display
    • buildKBEnrichedQuery() now uses clean queries without meta-tokens for proper LLM context
    • trySpecializedFocusedRouting() handles root navigation, flexible hierarchy, and meta-token parsing
    • Comprehensive test coverage for all routing patterns and hierarchy depths
  • Roadmap - Quoted Text Search: Proposed support for combined category + text queries (e.g., omni:sop:language bindings "java tutorial").
  • Documentation Updates: Updated AI_COPILOT.md, AI_COPILOT_USAGE.md, and IMPLEMENTATION.md with comprehensive routing guides including root category navigation.

SOP V2 build 53 (Upcoming)

• Schema Format Enhancement: Introduced flat schema format for better LLM understanding and correlation with Store Relations.
  • New Fields: Added FlatSchema, KeyFields, and ValueFields to StoreInfo for improved schema representation.
  • Flat Schema: Uses flat format without p...

v5.3.7

Full Changelog: v5.3.6...v5.3.7

v5.3.6

What's Changed

• wired: health endpoint for load balancers by @gerardrecinto in #299

Full Changelog: v5.3.5...v5.3.6

SOP v5.3.5

What changed since v5.3.4

Performance

• B-Tree node slot allocation and L1 cache handling got faster. Measured improvement on hot-path operations.
• L2 cache eviction for standalone mode reworked -- smarter about what to keep under memory pressure.
• NewBtree/OpenBtree can now safely be called multiple times for the same B-Tree within a single transaction. Previously this could cause subtle data races. Stress tests updated.

Security -- all Dependabot alerts cleared

• Go deps: golang.org/x/crypto 0.52.0, golang.org/x/net 0.55.0, go-git/v5 5.19.1, go-billy/v5 5.9.0, cloudflare/circl 1.6.3
• Java binding: jackson-databind 2.19.0 -> 2.21.4
  • PolymorphicTypeValidator bypass via generic type parameters (high)
  • Array subtype allowlist bypass in BasicPolymorphicTypeValidator (high)
  • InetSocketAddress deserialization triggers eager DNS resolution / SSRF (medium)
  • Case-insensitive deserialization bypasses per-property @JsonIgnoreProperties (medium)

32 alerts total were open at the start of this cycle. All 32 are now resolved.

In-memory B-Tree iterators

• All() and Range(from, to) now return iter.Seq2 -- you can do for k, v := range b3.Range(102, 104). Range seeks directly to the start key.

CI/CD

• Gated delivery pipeline (.github/workflows/deliver.yml): build -> tests -> GHCR container (sop-quickstart) -> staging smoke -> manual production promotion.
• Quickstart container (Dockerfile.quickstart): distroless image running the in-memory B-Tree demo.

Developer experience

• Quickstart example at examples/quickstart -- no servers, no config, just go run ./examples/quickstart.
• "What is SOP" plain-language overview added and linked from README and site.
• Demo GIF recorded from the quickstart run.

Repository cleanup

• 26 documentation markdown files moved from root to docs/.
• 13 shell scripts moved from root to scripts/.
• All references updated (README, index.md, release workflow, VS Code tasks, DEV_TOOLS.md).
• Root directory now has source code, standard project files, Dockerfiles, and Go module files. That is it.

---

Full changelog: v5.3.4...v5.3.5

SOP V2: general purpose AI & Knowledge Base support PATCH Update

Upgraded to latest go 1.26.4.

SOP V2: general purpose AI & Knowledge Base support PATCH update

Minor cleanup and CICD fix.

SOP V2: general purpose AI & Knowledge Base support

Summary:

• Support for "general purpose" AI like Claude Code, Gemini & Chat GPT
• Support for first class Knowledge Base, a.k.a. Spaces. Mint: visualize, enrich, group your "thoughts" into nested Categories to
• Support million/billions data set without the 'pigeon hole' weakness, then Vectorize them for "semantic" retrieval
• Support for local Embedders, thanks to Kelindar Search Open source folks!
• Support for rich prefix based routing with zero LLM calls, e.g. custom KB:Category1:Category1.1
• Support for LLM filtering, e.g.: e.g. custom KB:Category1:Category1.1:llm [instruction filter]
• Richer Stores schema view and management
• Stores and Spaces can complement each other, write ACID transactions that manage/query the two databases
• Ask LLM to manage the two databases, no need to learn SQL syntax
• Very rich LLM integration implementation taking full advantage of providers' server side continuation/conversation features, and SOP "memory" feature (MRU, Short Term Memory & Long Term Memory structures).
• Cleaner Release Numbering, targeting latest latest so go toolchain will not get confused.
• ai/memory KnowledgeBase package can easily be reused externally. See http://github.com/grecinto/tensortree
• More robust/optimized SOP B-tree engine:
  • Store Repository Count tracking imposes minimal update of store as opposed to entire Store repository metadata overwrite
  • More efficient B-tree storage use, i.e. - it is now practical to use large SlotLengths like 10,000 per Node. It requires same storage space as 500 SlotLength Node
  • Beta 2/Community Preview Stability

SOP V2: general purpose AI & Knowledge Base support patch update

Few minor refactors/updates to allow easier reuse of ai/memory KnowledgeBase package and dependencies.

SOP V2: general purpose AI & Knowledge Base support

Summary:

• Support for "general purpose" AI like Claude Code, Gemini & Chat GPT
• Support for first class Knowledge Base, a.k.a. Spaces. Mint: visualize, enrich, group your "thoughts" into nested Categories to
• Support million/billions data set without the 'pigeon hole' weakness, then Vectorize them for "semantic" retrieval
• Support for local Embedders, thanks to Kelindar Search Open source folks!
• Support for rich prefix based routing with zero LLM calls, e.g. custom KB:Category1:Category1.1
• Support for LLM filtering, e.g.: e.g. custom KB:Category1:Category1.1:llm [instruction filter]
• Richer Stores schema view and management
• Stores and Spaces can complement each other, write ACID transactions that manage/query the two databases
• Ask LLM to manage the two databases, no need to learn SQL syntax
• Very rich LLM integration implementation taking full advantage of providers' server side continuation/conversation features, and SOP "memory" feature (MRU, Short Term Memory & Long Term Memory structures).
• More robust/optimized SOP B-tree engine:
  • Store Repository Count tracking imposes minimal update of store as opposed to entire Store repository metadata overwrite
  • More efficient B-tree storage use, i.e. - it is now practical to use large SlotLengths like 10,000 per Node. It requires same storage space as 500 SlotLength Node
• Beta 2/Community Preview Stability

SOP V2 Platform Tools Release: AI's scripting engine stabilization

SOP V2 Platform Tools Release: AI's scripting engine stabilization.

• Also, introduced "client RLE sessions", SOP AI engine is now a server that can orchestrate & remember/exchange interactions across many client sessions requests.

NOTE: this will be the last upgrade/enhancement on this form of the SOP AI engine/scripting. The next one will sport "active memory" as we work on introducing a better user experience when it comes to AI copilot's "adaptive universal domain support". SOP AI will sport "adaptive memory system" and will remove need for humans to correct the copilot, it will be able to recognize mistakes/hallucinations (implicitly) and auto-correct itself.