refactor(sdkx/knowledge): remove SemanticProcessor, expose stateless context generators

[lIang70]

Summary

Move all LLM-driven L0/L1 summarization out of sdkx/knowledge and into a pair of pure helper functions. FSStore's storage operations now exclusively persist raw content and update search indexes; callers drive summarization explicitly and publish results back via the existing setters and sidecar writers.

This is a breaking change for any caller using SemanticProcessor, WithSemanticProcessor, SetSemanticProcessor, or WithOnEvict. See migration notes below.

Why

• SemanticProcessor coupled storage to a worker pool, retry policy, queue, and LLM, and emitted side effects (sidecar writes + in-memory mutations) on every AddDocument call. That forced every consumer into the same scheduling/observability model and made it impossible for a higher-level platform to track processing status, recover durably across restarts, or apply its own back-pressure to the LLM.
• The only consumer of sdkx/knowledge today is the flowcraft platform, which already has its own job/queue/persistence stack. Pushing the processor out lets the platform orchestrate L0/L1 with its own status table, retries, and cache invalidation, while keeping sdkx/knowledge focused on storage + retrieval.

What changes

Removed

• SemanticProcessor and all related machinery: workers, queue, retries, processingTask, the knowledge_queue_drops_total metric.
• WithSemanticProcessor, SetSemanticProcessor, WithOnEvict options.
• The implicit Enqueue calls inside FSStore.AddDocument and FSStore.AddDocuments — raw content writes no longer trigger any LLM activity.
• sdkx/knowledge/semantic.go (~430 lines).

Added

• sdkx/knowledge/generator.go with two stateless helpers and exported prompt constants:

GenerateDocumentContext(ctx, llm, content) (DocumentContext, error)
GenerateDatasetContext(ctx, llm, summaries []DocumentSummary) (DatasetContext, error)

// exported templates so callers can compose their own pipelines
AbstractPrompt
OverviewPrompt
DatasetOverviewPrompt
DefaultPromptInputLimit

Both helpers return partial results on error (e.g. abstract is preserved when overview generation fails) so callers can persist what succeeded.

Kept (now caller-facing)

• FSStore.SetDocAbstract / SetDocOverview / SetDatasetAbstract / SetDatasetOverview — in-memory publishers; doc comments updated to describe the caller-driven contract.
• FSStore.WriteSidecar / WriteDatasetFile — durable sidecar writers.
• CachedStore.EvictDataset — cache invalidation hook for callers refreshing layered context out-of-band.

Documentation

• Package doc on types.go now states the side-effect-free contract explicitly.
• FSStore struct doc points users at the generator helpers.

Tests

• Coverage: 73.3% → 83.1%; go test -race ./sdkx/knowledge/... clean.
• New generator_test.go (11 cases): happy path, partial-result error semantics, nil-LLM defense, prompt routing assertions (verifies the exported constants are wired to the right call), dataset rollup distilling the abstract from the generated overview.
• New pipeline_test.go (12 cases): end-to-end caller-driven flow (Generate → Set* → Write sidecar/dataset file → rebuild from disk), concurrent fan-out across documents, regression that AddDocument no longer issues LLM calls, sidecar cleanup on delete, abstractStats bookkeeping on replace/clear, searchAcrossDatasets L0/L1 branches, Abstract/Overview sidecar fallback when not yet indexed.
• cached_store_test.go extended (10 new cases): previously-untested paths for AddDocuments, ListDocuments, DatasetAbstract/DatasetOverview cache + evict, wrong-type cache fallback for dataset-level reads, and inner-error propagation via a new errStore stub.
• go.mod: go.opentelemetry.io/otel/metric drops to an indirect dependency now that the queue-drop counter is gone.

Caller migration

Before:

processor := knowledge.NewSemanticProcessor(llm, nil, knowledge.WithWorkers(3))
store := knowledge.NewFSStore(ws, knowledge.WithSemanticProcessor(processor))
processor.Start(ctx)
defer processor.Stop()

// Implicit: AddDocument -> enqueue -> generate L0/L1 -> sidecar + setter
_ = store.AddDocument(ctx, ds, name, content)

After:

store := knowledge.NewFSStore(ws)

if err := store.AddDocument(ctx, ds, name, content); err != nil {
    return err
}

// Schedule on your own job system / worker pool, then:
go func() {
    docCtx, err := knowledge.GenerateDocumentContext(ctx, llm, content)
    if err != nil {
        // surface to your status table; partial results in docCtx may still be persisted
        return
    }
    store.SetDocAbstract(ds, name, docCtx.Abstract)
    store.SetDocOverview(ds, name, docCtx.Overview)
    _ = store.WriteSidecar(ctx, ds, name, ".abstract", docCtx.Abstract)
    _ = store.WriteSidecar(ctx, ds, name, ".overview", docCtx.Overview)
    cached.EvictDataset(ds) // if wrapped in CachedStore
}()

The dataset-level rollup (previously auto-triggered after each doc) is likewise caller-controlled:

dsCtx, err := knowledge.GenerateDatasetContext(ctx, llm, summaries)
if err == nil {
    store.SetDatasetAbstract(ds, dsCtx.Abstract)
    store.SetDatasetOverview(ds, dsCtx.Overview)
    _ = store.WriteDatasetFile(ctx, ds, ".abstract.md", dsCtx.Abstract)
    _ = store.WriteDatasetFile(ctx, ds, ".overview.md", dsCtx.Overview)
}

Test plan

• go build ./sdk/... ./sdkx/... ./plugin/...
• go test ./sdk/... ./sdkx/...
• go test -race ./sdkx/knowledge/...
• go vet ./sdkx/...
• Coverage report (go tool cover -func): 83.1%, no 0% functions remain in the touched files
• Downstream: bump flowcraft platform to the new sdkx tag, rewire wire_knowledge.go and oapi_handlers.AddDocument to drive GenerateDocumentContext from a platform-owned worker (separate PR on the consumer repo)