Repository navigation
Technical Design 1.en
English · 1 | 中文 · 1 | Technical Design 2
Interactive view: Technical Design view (runtime and storage architecture) — 中文 · English
Read this document before changing the runtime architecture, data storage, interfaces, or technology choices of M1–M3. It turns the frozen Requirements-SPEC and the information semantics of 01–05 into an implementable backend architecture. It covers System Architecture, Data Lifecycle, Technology Baseline, and Data Model, together with M1 intelligence monitoring, M2 evidence retention and normalization, and M3 intelligence enrichment and association; it also fixes the Case–Trajectory–Experience evolution seam used by M4/M5/M7. The M4–M6 runtime continues in Technical Design 2.
Requirements owns what the system needs to do; 01-Data-Sources-and-Processing owns source, evidence, object/relation, retrieval, and data-lifecycle semantics; 02-CTI-Baseline-Reuse-and-Ownership owns the CTI reuse boundary; 03-Normative-Knowledge-and-Policy and 04-AI-Model-Data-and-Agent-Application-Security own specialized semantics; 05-Incident-News-Intelligence owns Incident / News Watch source roles, monitored sources, and event-shaped lifecycle. This document decides which processes handle these objects, which stores they write, which interfaces connect them, and which technology stack the first code version uses.
This is Technical Design 1. Its scope ends at the M1–M3 Data Plane, M3→M4 handoff, and the Case / Trajectory / Experience storage seam. Task Routing, Perception, Investigation State, Agent Runtime, Execution Control Plane, and Decision/QA for M4–M6 are defined in Technical Design 2.
M1–M3 implements continuous acquisition, Evidence/Knowledge, document revisions/indexes, deterministic enrichment, incident candidates/promotion/timelines and the M3→M4 read handoff. Current configuration contains 106 catalog entries, 71 executable SourceDefinitions and 43 scheduled sources; these counts are not a live-health measurement.
CWE, explicit PoC, FIRST/GitHub EPSS, NVD/CVE5/OSV/CSAF applicability, fixed versions, asset CPE joins and exact paper-CVE links are implemented. Material quality gates, retained-evidence reparsing, PDF encoding repair, registered-source expansion and evidence-bound curated incidents are also implemented. Their breadth and quality remain subject to frozen evaluation denominators.
Local Compose pauses background model stages and automatic editorial by default. Collection, evidence retention, lexical indexing, deterministic enrichment and projections continue. Dense/semantic/normative document work requires explicit enablement and configured providers. Startup synchronizes registry/schedule state while preserving cursor/backoff; time-bounded asset sources remain on demand.
The acceptance register distinguishes current code checks, retained benchmark results and open delivery/semantic/asset/incident-quality denominators. Historical 2026-10-01/08 metrics do not describe current runtime health. See the information-quality/report contract for public material and report boundaries.
The SecFusionAgent Data Plane has three classification axes that must not substitute for one another:
- Source taxonomy: eight external source classes answer “who owns which facts”;
- Processing path: four primary paths—Bug Stream, Structured Development Index, Insight Corpus, and Incident Watch—plus the Internet Asset on-demand side path answer “how this source stream is discovered, processed, retained, and promoted”;
- Durable value/state family: Knowledge, Incident, Insight, and Experience answer “what the system accumulates durably.”
One provider may produce several source streams, and one source class may enter different processing paths depending on the object/content. Processing convergence only reuses runtime and retention mechanics; it never collapses source authority or provenance. The Website model “eight source classes → four primary processing paths + asset side path” uses the same semantics as this design.
SourceDefinition.retention_mode is the executable encoding of the processing path; the design does not introduce a second enum:
| Processing path | RetentionMode |
Typical behavior |
|---|---|---|
| Bug Stream | hot_window |
high-frequency current working set with selective promotion |
| Structured Development Index | selective_index |
stable objects/relations are synchronized; content is fixed on demand |
| Insight Corpus | durable_managed |
raw revision, projection, chunks, semantic/index state are durable |
| Incident Watch | incident_signal |
short-TTL signals/candidates; durable Incident after promotion |
| On-demand observation side path | time_bounded |
query-bound observation with observed_at, promoted when needed |
One taxonomy category may use several retention modes; retention mode never determines the category. durable_promoted is a runtime state after promotion, not a sixth source-processing path.
SecFusionAgent designs its data lifecycle by workload. Four classes of intelligence workload need a lasting distinction:
- Bug Stream: high-frequency vulnerability streams that already have a canonical store, such as CVE/NVD/OSV/GHSA/KEV. Local state maintains a working set of recently added, recently updated, frequently accessed, and under-investigation records, and queries the external canonical source for history on demand.
- Incident Intelligence: sudden events such as attacks, theft, data breaches, supply-chain incidents, in-the-wild exploitation, and fund transfers. Initially these carry only news, researcher, on-chain, or telemetry signals, so the system performs hot clustering / timeline watch first and promotes to a durable Incident once sustained analytical value forms.
- Insight Corpus: long-lived knowledge assets such as papers, vendor technical reports, security research, system/model cards, and standards and regulations. Local state holds the revision, projection, semantic result, and retrieval index.
- Investigation Experience: investigation methods, failure modes, and stop/fallback policies extracted from M4/M5 Cases/Trajectories and M7 outcomes; these are the Agent's own evolution assets.
Investigation State spans Bug/Incident investigation and has low throughput, high value, and strong consistency; PostgreSQL durable state holds it.
The runtime consists of Scheduler, Source Worker, Hot Cache, Incident Correlator, Evidence Ingestion, Normalization, Enrichment, and Indexer. PostgreSQL stores durable state, current projections, and long-lived knowledge; the S3-compatible artifact store stores raw JSON/HTML/PDF that must be retained long term; Redis Hot Cache stores the bounded bug working set and short-lived incident signals/clusters; a separate Redis Broker carries Celery task transport.
flowchart LR
subgraph External[External sources]
BugCold[Canonical bug DB\nCVE / NVD / OSV / GHSA / KEV]
IncidentSignal[Incident signals\nnews / on-chain / researcher / telemetry]
Primary[Primary confirmation\nproject / vendor / repo / authority]
Structured[GitHub / package & model platforms]
Managed[Paper / vendor / blog / normative content]
OnDemand[Internet asset / high-cost providers]
end
subgraph Runtime[SecFusionAgent backend]
Scheduler[Scheduler]
Adapters[Source Workers / Adapters]
HotIngress[Hot Bug Ingress + Diff]
Incident[Incident Correlator / Timeline Watch]
Ingress[Durable Evidence Ingestion]
Normalize[Normalization]
Enrich[Enrichment]
Promote[Promotion Gate]
Indexer[Indexer]
Model[LLM Provider Adapter]
API[FastAPI]
end
subgraph Storage[Local state]
Hot[(Redis Hot Working Set\nBug + Incident Candidate)]
PG[(PostgreSQL + pgvector\nDurable Knowledge + Current Views)]
Blob[(S3 / MinIO Artifact Store)]
Broker[(Redis Celery Broker)]
end
Scheduler --> Broker
Broker --> Adapters
BugCold --> Adapters --> HotIngress --> Hot
IncidentSignal --> Adapters --> Incident --> Hot
Primary --> Adapters
Hot --> Normalize
Hot --> Enrich
Incident --> Enrich
Enrich --> Promote
Promote --> Ingress
Structured --> Adapters
Managed --> Adapters
OnDemand --> Adapters
Adapters --> Ingress
Ingress --> Blob
Ingress --> PG
PG --> Normalize
Normalize --> PG
Enrich --> Model
Enrich --> PG
PG --> Indexer
Blob --> Indexer
Indexer --> PG
PG --> API
External sources enter the backend through the access modes frozen by 01, each with a different retention mode:
| Access mode | Typical sources | Local lifecycle | Long-term authority |
|---|---|---|---|
| Canonical bug feed / external query | CVE/NVD/OSV/GHSA/KEV |
hot_window: latest published, latest updated, frequently accessed, and active workflow records enter Redis; evictable after TTL/LFU |
external canonical source |
| Structured index | GitHub, package/model registry, target repo graph |
selective_index: stable IDs/relations stay local; bodies and high-cost objects are snapshotted on demand |
external platform + local relation graph |
| Managed content | vendor advisories, blogs, papers, standards and regulations, Insight Watch |
durable_managed: raw revision, projection, semantic result, and retrieval index are retained long term |
local evidence + original source |
| Incident signal | news flashes, on-chain alerts, researcher disclosures, threat telemetry |
incident_signal: short TTL signal + hot cluster; promotion to Incident is decided after sustained enrichment |
the original sources hold their own |
| On-demand observation | Censys/Shodan/FOFA/ZoomEye, commercial intelligence |
time_bounded: query/filter/result carry an observation time; short TTL by default, promoted when an investigation or answer needs it |
provider observation |
| Promoted bug / incident evidence | revisions that are investigated, cited, form long-lived relations, or enter a benchmark |
durable_promoted: fixes the revision actually consumed and the necessary raw evidence |
local evidence + external revision |
The local value of Bug Stream comes from the current working set; the value of Incident Intelligence comes from event evolution, cross-source corroboration, and the timeline; the value of Insight Corpus comes from long-lived semantic assets; the value of Experience comes from the evolution of the Agent's investigation methods. The four workloads share the source/evidence/provenance contract, while retention, consistency, indexing, and enrichment depth are configured independently.
Hotness is determined by four signals: source change recency, access frequency, workflow activity, and domain priority. Newly published records and new revisions enter the hot set automatically; user queries, M3 enrichment, and M5 investigation raise access/workflow heat; a target AI Infra repo and KEV/PoC/critical signals can raise processing priority. Active investigations, pending promotions, and benchmark pins block eviction.
Admission, processing priority, and eviction are computed separately. Admission decides whether an object enters Redis; processing priority decides whether expensive enrichment is triggered immediately; eviction reclaims capacity according to TTL, LFU/LRU, and workflow protection. The Redis max-memory policy acts as capacity protection, while the business layer continues to hold promotion and eviction semantics.
Bug Stream write path
delta / updated record
-> cheap parse + current projection
-> Redis hot working set
-> change classification
-> selected enrichment
-> promote OR expire/evict
Bug read path
Redis hit
-> external-current projection
Redis miss
-> PostgreSQL durable SecFusion knowledge (if promoted)
-> external canonical source for current record
-> hot admission
A revision change on a high-frequency source goes through a field-level diff first. Changes to structured fields such as CVSS/CWE/KEV/reference update the corresponding projection; only changes to semantic fields such as description and patch explanation trigger semantic extraction / embedding. The processing rule is change type -> invalidate affected derived state, which lowers write amplification.
Promotion turns temporary working state into a long-lived asset. Triggers include: entering an M4/M5 investigation; becoming an M6 answer citation; entering a benchmark/gold set; forming a long-lived target repo / supply-chain / incident relation; a source conflict that must be retained; and an analyst review decision to retain long term. Before a durable conclusion is needed, the provider revision actually consumed is fixed.
Demotion cleans up payloads in long-term storage that can still be reliably re-fetched from outside and no longer have durable relation / benchmark / audit dependencies. Demotion can delete the local raw body and derived cache while keeping the external ID, promotion history, and the necessary provenance pointers. The concrete demotion policy is enabled after data volume is measured.
Authority across the three data tiers is fixed as follows:
External provider
authority for current external canonical record
PostgreSQL durable knowledge / incident / experience
authority for SecFusionAgent 自己确认、推导和长期保存的状态
Redis hot cache
authority for nothing; it is a replaceable working projection
Cache consistency follows the authority tiers: a Redis miss or eviction cannot change a long-term conclusion. When an object has both hot and durable state, the read path returns the hot external-current projection and merges SecFusion-derived knowledge from PostgreSQL; the two keep their own revisions, so an older durable snapshot does not overwrite the latest external state.
Warm adds no new database product and is carried by the materialized current view / current projection in PostgreSQL. It serves high-frequency queries better than raw claims/relations and can be rebuilt incrementally from durable history. The first batch of projections includes:
current_vulnerability_view
current_fix_status
current_affected_versions
current_repo_security_state
current_incident_view
The underlying layer continues to store multi-source claims/relations/timeline events; projections perform incremental view maintenance from KnowledgeChange and incident.changed outbox events. A CVSS or KEV change updates the corresponding fields, while a description/technical analysis change invalidates the semantic projection and embedding. M4/M6 read current projections by default and fall back to the underlying evidence graph when a conflict or a historical state needs explanation.
Current Projection logical identity is fixed as projection_type + subject_id while retaining a deterministic projection_id. PostgreSQL writes may only advance upstream_revision; concurrent rebuilds of the same projection identity first take a transaction-scoped advisory lock derived from the deterministic ID and then perform the conditional upsert, preventing speculative-insert deadlocks across the deterministic PK and logical UNIQUE index. The lock serializes only one identity, does not turn unrelated projections into a global single-writer path, and does not remove database uniqueness invariants.
The concrete monitoring sources, source roles, access priorities, and onboarding probes of Incident / News Watch are held by 05-Incident-News-Intelligence. Technical Design fixes only their behavior once they enter the runtime: signal/telemetry enters short TTL staging first, primary/forensic/authority can take part in promotion and durable enrichment, and source_family/upstream_source removes pseudo-corroboration caused by reprints.
Incident input is bursty, duplicated, uncertain, and continuously revised, so it enters a hot signal buffer first:
SignalItem
-> deterministic anchor extraction
-> IncidentCandidate clustering
-> timeline watch / source follow-up
-> promotion decision
-> durable SecurityIncident
-> M3 enrichment
-> M4 confirmed / conflict / unknown
-> M5 investigation
-> postmortem / insight / experience
SignalItem is stored in a short TTL Redis namespace and contains the source, original/cited source, published/observed time, title/summary, entity hints, URL, content hash, and verifiable anchors. Web3 anchors prefer project, chain, token, address, tx hash, amount, and time window; a general cyber incident prefers organization, product, domain/IP, CVE/GHSA, threat actor, campaign, and time window. The current implementation lets only deterministic strong anchors participate in candidate join; semantic similarity is reserved as later recall assist and must work together with a strong anchor/evidence gate, so that two independent incidents on the same project within a short period are not merged by force.
IncidentCandidate stores cluster state, watch priority, next poll, last material change, source diversity, and unresolved questions. Early in an event, every reprinted news item is not written permanently to PostgreSQL/S3; duplicate reprints are evicted after their TTL. Promotion to a durable Incident happens on the following signals:
- the affected project/vendor or an official channel confirms;
- multiple independent sources corroborate the same verifiable anchor;
- directly verifiable evidence such as an on-chain transaction, a public IOC, or a GitHub patch/advisory;
- the event's impact triggers an M4/M5 investigation, continued fund tracking, or a user query;
- an analyst or benchmark explicitly pins it.
After promotion the system creates a SecurityIncident object and fixes the source revision supporting the current timeline/claim as durable evidence. Later news still enters the hot cluster first, and only items that produce a material change are promoted. Full-text vectorization targets promoted technical reports, postmortems, vendor advisories, and high-value analysis; not every news flash is embedded.
Incident Watch uses adaptive follow-up: watch priority rises on new fund movement, an official response, a continuing exploit, or a change in impact scope; polling is gradually reduced after a long period without material change, and switches to low-frequency revision watch once the event reaches resolved/postmortem state. The concrete cadence is configured from source capability and the M8 budget.
M1 can discover changes at high frequency, so the expensive processing of M3 needs an independent budget. Every hot update passes through deterministic normalization/diff first and then enters the enrichment queue by priority; semantic processing, Internet asset lookups, and large-scale repo traversal have independent concurrency/rate/cost budgets.
Priority signals consist of a target repo/risk domain match, KEV/PoC/critical status, change type, access frequency, active investigation, and SLA. When the queue backlog exceeds budget, low-priority objects stay in the Redis working set with delayed enrichment and source monitoring keeps moving, so LLM/API throughput does not block new vulnerability discovery in reverse.
This strategy decouples monitoring freshness from enrichment depth: M1/M2 guarantee first that a new change is visible, and M3 deepens processing step by step according to value and system capacity. M7 records hot-ingest latency, enrichment wait time, and processor latency separately.
| Runtime | Code owner | State owner | Responsibility |
|---|---|---|---|
api |
apps/api |
no independent business state | exposes the source health / knowledge read API, and later carries the M4–M6 application API |
scheduler |
packages/monitoring |
sources/source_state |
reads source schedules and creates scheduled acquisition tasks |
source adapter |
packages/sources |
no independent durable runtime state | implements API/Git/RSS/Web/query provider protocols and produces IngestEnvelope
|
acquisition runtime |
packages/monitoring/acquisition |
acquisition_runs |
creates runs, parent/query provenance, and terminal state for scheduled/on-demand provider reads |
incident correlator |
packages/intelligence/incident |
hot incident signal/cluster | M2 signal dedup, anchor clustering, IncidentCandidate identity/correlation; follow-up requests go to monitoring/acquisition |
ingestion / normalization worker |
packages/intelligence |
hot projection + durable evidence/knowledge | hot ingest/diff, durable evidence commit, parsing, dedup, and identity resolution |
enrichment worker |
packages/enrichment |
processing result | deterministic, graph, semantic, and provider-backed enrichment |
index worker |
packages/intelligence |
derived index | document chunks, lexical/vector index, metadata projection |
apps/worker starts different Celery queues from the same Python image; a deployment can split I/O collection, semantic processing, and indexing into separate workers by workload, or run them combined in a development environment. Business behavior is still held by the corresponding package.
The Scheduler and Source Worker share one acquisition contract, and M2 routes at the entry point by retention_mode:
- The Scheduler finds due sources in
sources/source_state, creates anAcquisitionRun, and dispatches it to the collection queue. - The Source Worker runs the adapter's
discover/fetch/queryand returns anIngestEnvelopecarrying source/revision/time/hash. - A
hot_windowsource entersHotBugIngress: cheap parse, current projection, and field-level diff are written to Redis, producing aHotNormalizationResult. The cursor advances after the hot record is written successfully and is readable. - An
incident_signalsource entersIncidentSignalStore: a short TTLSignalItemis stored first, then the Incident Correlator updatesIncidentCandidateand watch state from deterministic strong anchors. Semantic similarity can serve as later candidate assist, but the current implementation does not let it trigger a cluster/promotion on its own. - A hot bug enters selective M3 enrichment according to change/domain priority/access/workflow; an IncidentCandidate enters source follow-up/enrichment according to source diversity, material change, and unresolved questions.
- A durable signal triggers the Promotion Gate. Bug promotion fixes the current provider revision; Incident promotion fixes the source revision supporting the timeline/claim.
durable_managed/durable_promotedsources go from acquisition directly into Durable Evidence Ingress. Raw JSON/HTML/PDF is written to the artifact store, and PostgreSQL creates an immutableObservation/EvidenceArtifact. - Durable Normalization writes
Object/ExternalIdentifier/Claim/Relation/EvidenceLinkthrough the source parser plus canonical mapper; Incident promotion also createsSecurityIncident/timeline state. The transaction commits a newknowledge_revisionorincident_revisiontogether with the outbox event. - M3 runs deterministic, graph, semantic, or provider-backed processors on a durable subject. A provider lookup passes through M2 again and chooses the hot/incident/durable path by its own retention mode.
- A durable commit triggers the Indexer/current projection updater; document chunks, lexical/vector indexes, current views, and Experience/Insight retrieval projections are all rebuildable derived state.
The Celery Broker provides at-least-once task delivery, PostgreSQL stores durable run/outbox state, and Redis Hot Cache stores only evictable working state. The idempotency key of a hot record absorbs feed replay, and the idempotency key of a durable Observation/Claim/Relation absorbs promotion and processing replay.
C1–C3 in Requirements map directly onto runtime boundaries:
| Constraint | Technical contract | M1–M3 landing |
|---|---|---|
| C1 evidence traceability | a hot projection/signal traces back to a provider/source revision; every durable claim/relation/timeline event locates its observation, artifact locator, and processing revision | HotBugRecord/SignalItem revision, Observation, EvidenceArtifact, EvidenceLink, ProcessingRun, knowledge/incident revision |
| C2 external content is untrusted | external payloads permanently keep external_untrusted; model processing produces typed candidates only, and source content has no configuration, credential, or execution authority |
HotBugIngress, Incident Correlator, Evidence Ingress, Semantic Processor, ModelProvider boundary |
| C3 bounded execution | acquisition/processing tasks have timeout, attempt, terminal state, and idempotency key |
AcquisitionRun, ProcessingRun, Celery retry policy, source backoff |
A C1 failure appears as a candidate missing a valid evidence locator; a C2 failure appears as external content affecting system config, tool policy, or the credential boundary; a C3 failure appears as a run that cannot reach terminal state. M7 later builds regressions directly around these three failure classes.
The current DESIGN states the first implementation baseline directly. Probes verify high-risk assumptions, and a specific provider is replaced when verification fails; module boundaries and data contracts stay stable.
| Concern | Baseline | Design rationale |
|---|---|---|
| Backend language | Python 3.12+ | the source/API/LLM/CTI ecosystem is concentrated here and matches the current Agent tooling |
| HTTP API | FastAPI | typed async API; shared with the front end by M4–M6 later |
| Schema / validation | Pydantic v2 |
IngestEnvelope, processor output, and the API contract use one type system |
| Relational persistence | PostgreSQL 16+ | unified transaction semantics for source state, knowledge, relations, run provenance, and benchmark metadata |
| ORM / migration | SQLAlchemy 2 + Alembic | explicit schema and migrations, with tests and production sharing the models |
| Vector / lexical retrieval | pgvector + PostgreSQL FTS/pg_trgm
|
the first version keeps durable insight/experience in the same database; VikingDB/OpenSearch/Qdrant providers can follow |
| Experience retrieval | PostgreSQL + pgvector | Experience is versioned policy memory in its own tables, separate from knowledge/evidence; OpenViking serves as a design reference for the evolution pipeline |
| Raw artifact | S3-compatible object storage; MinIO locally/CI | JSON/HTML/PDF is separated from the canonical DB, content hashes are reusable, and production can switch to S3/OSS |
| Background jobs | Celery + dedicated Redis Broker | collection, normalization, enrichment, and indexing use separate queues; the broker uses a separate instance/namespace and a noeviction-class policy to protect task state |
| Hot working set | dedicated Redis Cache | bug working set + incident signal/cluster; the business layer holds admission/promotion/watch, and TTL/LFU/LRU handles capacity protection |
| Scheduling | single-instance scheduler + source-level next_due_at
|
source cadence comes from the registry, avoiding hard-coded cron per source |
| HTTP client | httpx |
async requests, timeouts, redirects, streaming, and adapter tests are easy to control |
| Content parsing | source mapper + HTML/PDF/plain parser adapters; PDF currently uses pypdf, and HTML/plain are part of the executable Managed Content path |
parsing is separated from business semantics and raw artifacts can always be reprocessed |
| LLM integration | own ModelProvider interface + provider adapter; Pydantic/JSON Schema structured output |
the M3 model call is a semantic processor, and prompt/model/provenance stay explicit |
| CTI interoperability | STIX 2.1 / TAXII 2.1 adapters, with OpenCTI/MISP attached | reuses standards and a mature ecosystem while keeping the AI-specific data model |
| Deployment | Docker Compose as the development, CI, and competition deployment baseline | API, worker, scheduler, PostgreSQL, Redis Broker, Redis Hot Cache, and MinIO reproduce in one command |
The first version adopts a lightweight PostgreSQL-owned knowledge kernel + STIX/TAXII interoperability. The PostgreSQL kernel holds the core write path of SecFusionAgent; OpenCTI/MISP attach through CTI sources, exchange endpoints, and capability providers.
This choice follows the data semantics already fixed in 01–04: the system manages traditional CTI, repo/release graphs, paper/research insight, normative requirement/control, Agent/tool/memory relations, and evidence revisions at the same time. Placing these objects together in a relational kernel owned by the project lets the provenance, benchmark revisions, and AI-specific relations of M2/M3 share one transaction boundary. The STIX/TAXII adapters handle exchange with the traditional CTI ecosystem, and mature IOC/parser/correlation capabilities continue to be reused through existing platforms or libraries.
The first version answers relation queries with PostgreSQL relation tables, indexes, and recursive CTEs. If a later graph benchmark shows traversal latency or query expressiveness becoming the bottleneck, a dedicated graph provider is added behind KnowledgeRepository; PostgreSQL continues to hold canonical knowledge.
Retrieval also converges on PostgreSQL in the first version: exact ID/version/package queries go straight to structured tables; document chunks store tsvector/pg_trgm fields and pgvector embeddings; metadata filters use object/source/risk/version fields. Chinese and English semantic retrieval relies mainly on multilingual embeddings, while exact security identifiers and repo/package/version are handled by structured queries.
RetrievalPort hides the underlying implementation. When M7 later finds that lexical recall or data volume needs a separate search engine, providers such as OpenSearch/Qdrant can be added while the evidence, knowledge, and benchmark contracts stay unchanged.
Project packages hold the M1–M3 control flow directly: Scheduler, source routing, evidence commit, dedup, the entity resolution gate, and enrichment routing are deterministic logic; model calls perform unstructured semantic extraction through ModelProvider. Where LangChain/LlamaIndex is used for a local parser/retriever provider, it also obeys these project-level contracts.
The operating conditions of a semantic processor are fixed: input comes from an already persisted Observation/DocumentProjection; the prompt explicitly marks external text as untrusted data; the model's capability is structured generation; output enters a Pydantic candidate first; and a candidate needs an evidence locator and processor provenance before it is written to canonical knowledge. Model API credentials are injected by worker config, and external evidence cannot change provider, tool, or credential policy.
Local state falls into six groups: source/runtime state, Redis hot working set, incident state, durable evidence, canonical knowledge, and derived retrieval / experience. PostgreSQL holds durable metadata, promoted Incidents, canonical state, and current projections; the artifact store holds raw bodies that must be retained long term; Redis hot bug/incident records are evictable working projections and cannot be the sole authority for a long-term claim/timeline.
| Table / object | Key fields | Ownership |
|---|---|---|
sources |
source_id, adapter_type, source_class, authority_scope, access_mode, update_semantics, schedule_policy, retention_mode
|
M1 |
source_state |
cursor, last_attempt_at, last_success_at, next_due_at, backoff_until, rate_limit_state
|
M1 |
acquisition_runs |
trigger, source, query/cursor, attempt, status, parent run/task | M1/M3 |
observations |
source object identity, external revision, published/updated/fetched time, content hash | M2 |
evidence_artifacts |
observation, media type, storage URI, hash, access rights, trust class | M2 |
documents / document_revisions
|
stable document identity, source object, observation, external revision, parser revision, metadata | M2 |
objects |
internal ID, object_type, canonical key, typed properties |
M2 |
external_identifiers |
namespace, value, object ID | M2 |
claims |
subject, predicate, value, qualifier, origin, lifecycle, processing run | M2/M3 |
relations |
source object, relation type, target object, qualifier, origin, lifecycle | M2/M3 |
evidence_links |
claim/relation, observation/artifact, field/page/section/span locator | M2/M3 |
processing_runs |
processor, version, input revisions, model/prompt revision, status | M2/M3 |
insight_candidates |
related claims/relations, change type, evidence maturity, promotion state | M3 |
incidents |
SecurityIncident, lifecycle, watch state, current summary, incident revision | Incident/M4 |
incident_timeline_events |
incident, event time, event type, claim/evidence refs, supersedes | Incident/M4 |
incident_source_links |
signal/evidence source, original/cited source, corroboration dependency | Incident |
current_projections |
vulnerability/fix/repo/incident current state + upstream revision | M2/M3/M4 |
document_chunks |
document/section, text, metadata, embedding, index version | Indexer |
knowledge_changes |
monotonic revision, changed IDs, cause run, commit time | M2/M3 |
outbox_events |
domain event, payload, delivery state | transaction handoff |
investigation_cases |
task signature, target objects, goal, initial knowledge revision, rubric, status | M4/M5 |
investigation_trajectories |
case, step/tool/evidence event, retrieved experience version, outcome, latency/cost | M5/M7 |
experience_candidates |
extracted procedure/failure pattern, scope, supporting trajectory, candidate state | M5/M7 |
experiences / experience_versions
|
scope, trigger, recommended action, failure/stop/fallback, status, version | M5/M7 |
experience_support |
experience version ↔ trajectory / evaluation relation | M7 |
Redis holds an evictable projection of the current vulnerability working set, with keys formed from the provider namespace + external ID. A record stores the current revision, published/updated/fetched time, normalized current fields, content hash, hotness metadata, and processing state. Auxiliary sorted sets / sets maintain updated_at, access frequency, active workflow, and pinned objects.
bug:{source}:{external_id} current hot record
bug:updated sorted by source updated_at
bug:access access score
bug:active active processing/investigation
bug:pinned protected from eviction
incident:signal:{id} short-lived SignalItem
incident:cluster:{id} IncidentCandidate current state
incident:watch sorted by next_poll_at / priority
Redis is not the knowledge/incident source of truth. When a bug record is lost, it is restored from the external canonical source; after an incident signal expires, it is rediscovered from a known source or a later follow-up. When a durable claim/relation/timeline must form, the Promotion Gate fixes first the provider/source revision supporting the conclusion. The Celery broker and the hot cache use separate Redis instances or separate failure domains, so cache eviction does not affect task delivery.
A durable Observation represents what a given source actually provided at a given revision. A hot-window record does not create this object before promotion. The preferred idempotency key is source_id + external_object_id + external_revision; when a source has no revision, a stable locator plus content_hash is used. Replaying the same revision returns the existing observation; a content change on a mutable page produces a new observation.
The EvidenceArtifact body is written to the S3-compatible store with content_hash as the object key. PostgreSQL stores the storage URI and hash. The write order is: upload the artifact, commit observation/artifact metadata, return ObservationAck. A failed database transaction can leave an unreferenced blob, which a periodic janitor cleans up from artifact references; neither an artifact upload failure nor a metadata commit failure acknowledges the source cursor.
Every canonical knowledge transaction creates a monotonically increasing knowledge_revision. Object/Claim/Relation store their creation and supersede/retract revisions; the M7 benchmark records the knowledge revision it used. knowledge_changes and outbox_events are written in the same database transaction, and a background dispatcher then publishes the follow-up normalization/enrichment/index jobs to Celery, so knowledge is not committed while downstream tasks are lost.
Object represents an entity with a stable identity. Canonical object types already instantiated in the generic Knowledge runtime include Vulnerability, Repo, Issue, PullRequest, Commit, Release, Package, Product, SoftwareVersion, InternetAsset, Weakness, ExploitArtifact, ResearchWork, and Document; these types are exercised by deterministic or evidence-gated NVD/GitHub Advisory/OSV/CVE5/CSAF/FIRST EPSS/asset/research paths. NormativeDocument / Requirement / Control and Incident-specific objects remain owned by their domain models. The database carries sparse extension fields through object_type + properties JSONB, and strong identities go into external_identifiers.
Claim stores a scalar or structured literal such as CVSS, known-exploited, attack condition, or applicability. Relation uses the versioned vocabulary to connect objects, for example Vulnerability --affects-package→ Package, Vulnerability --fixed-by→ Commit, and ResearchWork --discusses-vulnerability→ Vulnerability. Edge-local conditions such as version range, time range, role, and scope go into the relation qualifier; a canonical relation entering formal enrichment P/R must also satisfy the subject/target shape frozen by the registry.
Every claim/relation has origin = source_asserted | deterministic_derived | semantic_derived and a lifecycle. accepted in M2/M3 means the write gate has passed; M4 later maintains confirmed facts, conflicts, and unknowns. A source conflict appears as several evidence-backed claims for the same predicate, and M2/M3 preserve the differences.
EvidenceLink is the data implementation of C1. A JSON source stores a JSONPath/field; HTML stores a section plus character span; PDF stores a page plus span; a GitHub object stores an API field / comment / diff locator. DocumentProjection also stores the locator mapping from clean text back to the original artifact, so a citation can still return the original evidence after chunks/embeddings are rebuilt.
document_chunks is derived data. A chunk stores source/object/evidence IDs, risk domain, document revision, chunker revision, embedding model revision, and the vector. Full-text/vector hits provide retrieval candidates, while factual citations still follow EvidenceLink.
The code owners of M1 are packages/sources and packages/monitoring. sources holds adapters and source definitions, and monitoring holds the scheduler, cursor, health, and scheduled runs. M1 handles source acquisition and candidate delivery; M2 decides identity/correlation. An accepted signal is, by lifecycle, a HotNormalizationResult for a hot bug, a SignalAccepted for incident acquisition, and an ObservationAck/NormalizationResult for a durable source; IncidentCandidate is an M2 output.
Static source definitions live in version-controlled config and are synced to the sources table at startup/deployment; credentials are stored only as secret references. Runtime cursor, health, next due, and rate-limit state live in PostgreSQL. Source onboarding is therefore both code-reviewable and observable at runtime.
SourceDefinition:
source_id
adapter_type
source_class
authority_scope
source_role = signal | telemetry | primary | forensic | authority | reference
source_family / upstream_source
access_mode
update_semantics
discovery_method
time_semantics
identity_semantics
auth_ref
rate_limit_policy
access_rights
retention_mode = hot_window | incident_signal | selective_index | durable_managed | time_bounded
schedule_policy
schema_version
Vulnerability business fields enter the M2/M3 knowledge tables. One provider can register several sources: GitHub target repo, the GitHub advisory database, and the GitHub research stream use different schedules/filters while sharing the GitHub adapter capability.
class SourceAdapter(Protocol):
async def discover(self, source: SourceDefinition, state: SourceState) -> DiscoveryBatch: ...
async def fetch(
self, source: SourceDefinition, ref: DiscoveredRef,
*, acquisition_run_id: str, trigger: AcquisitionTrigger,
) -> IngestEnvelope: ...
async def query(
self, source: SourceDefinition, spec: QuerySpec,
*, acquisition_run_id: str, trigger: AcquisitionTrigger,
) -> list[IngestEnvelope]: ...discover serves delta/feed/index; fetch fixes one external object revision; query serves on-demand lookups from M3/M5. CVE Git, GitHub, OSV/NVD/KEV, RSS/Web, arXiv, normative sources, and Internet asset providers each implement an adapter, and provider-specific schemas and errors stay inside the adapter.
The first batch of adapters reuses external standards and official interfaces: CVE uses cvelistV5 Git/delta; NVD/OSV/KEV use official APIs/feeds; GitHub uses API/Git; papers use arXiv/OpenReview/proceedings indexes; STIX/TAXII and MISP/OpenCTI sources use mature clients/parsers. A web page source enters the crawl path only when no structured interface exists.
The Scheduler queries sources with next_due_at <= now() on a fixed tick, creates an AcquisitionRun(trigger=scheduled), and dispatches it to the collection queue. The real cadence is computed in each source's schedule_policy; NVD, GitHub, CVE, arXiv, and vendor blogs are configured independently.
discover returns the current batch and next_cursor. The collection runtime routes by retention_mode: hot_window goes to HotBugIngress, durable_managed to ManagedDocumentService, selective_index to StructuredIndexService, and incident_signal to IncidentSignalIngress. next_cursor is committed after every item covered by the checkpoint is accepted. On a mid-way failure the cursor keeps its old value, the next round replays the whole checkpoint, and the idempotency constraints of each path absorb the items that already succeeded.
AcquisitionRun records success/no_change/rate_limited/auth_failed/schema_changed/fetch_failed. HTTP timeouts, 5xx, and rate limits use adapter-specific bounded retry; authentication failures and schema changes go straight to terminal run state and wait for a config or mapper fix. A failure on one source does not block other source queues.
M1/M7 record provider fetch latency and monitoring latency separately. Competition monitoring latency uses published_at → M2 available_at: for a hot bug the endpoint is HotNormalizationResult.available_at, and for a durable source it is NormalizationResult.committed_at. This localizes the latency of source discovery, network fetch, hot ingest, and durable processing separately.
config/source-inventory.json serves both the product source inventory and the source-category coverage denominator. The portfolio is fixed to eight categories: vulnerability / development / academic / vendor / independent / normative / assets / incidents, and all eight count toward category coverage; the competition threshold is ≥7. assets is acquired through an M3 on-demand enrichment side path but remains a first-class source category. A category counts as supported only when it has a coverage=owned fixed/grouped owner and the runtime adapter factory can construct it. A dynamic resolver, partial ownership, or one successful live probe does not increase the category count.
packages/evaluation/m1_m3.py separates category_coverage from source_delivery_coverage: the latter freezes SourceDeliveryKey(source_id, external_object_id, external_revision | content_hash) for a fixed source/window and records expected, accepted, missed and unexpected keys so dynamic source counts or extra noise cannot inflate coverage. The monitoring-latency sample contract measures records with reliable published_at to M2 available_at/committed_at; a record without reliable publication time is excluded from the latency-value denominator but remains in the denominator of latency_evaluable_coverage. Internal regression reports p50 / p95 / max / ≤6h pass-rate, while the fixed benchmark retains per-sample values; live reachability records environment state only.
packages/intelligence owns M2 and provides three entry points: hot normalization, incident signal staging, and durable evidence. A high-frequency canonical bug feed enters the bounded hot working set; an Incident signal enters a short TTL cluster; Managed Content and already promoted bugs/incidents enter durable Evidence Ingress. The three paths share the source identity/provenance contract, with different retention and write targets.
A source with retention_mode=hot_window hands its IngestEnvelope to HotBugIngress. It validates source/revision/hash, produces the current projection, and writes it to Redis together with the provider revision, source URL, and update time. M2 performs cheap deterministic normalization and a field-level diff on the hot record, producing a HotNormalizationResult; this result satisfies the latency computation for an available monitoring record but does not automatically create a permanent EvidenceArtifact.
When a hot record triggers promotion, the system fixes the current revision from the payload still in Redis first; when the provider supports immutable revision lookup, the revision can also be re-fetched. The durable Observation/EvidenceArtifact is created only after promotion succeeds, and only then can later claims/relations serve as long-term evidence for M4/M6.
An IngestEnvelope with retention_mode=incident_signal enters IncidentSignalStore. M2 stores a short TTL SignalItem, extracts the original/cited source, time, entities, and verifiable anchors, and hands it to the Incident Correlator to update the cluster. Ordinary reprints and weak signals stay in Redis; once a cluster meets the promotion conditions, PromotionService turns the selected source revision into durable evidence and creates or updates the SecurityIncident.
Incident signal staging maintains the context that an event is in progress; technical conclusions and timeline events enter PostgreSQL after their evidence is promoted.
Sources with retention_mode=durable_managed and items converted to durable_promoted by the Promotion Gate enter EvidenceIngress:
acquisition_run_id
trigger = scheduled | on_demand | investigation | promotion
source_id
external_object_id
canonical_url
published_at / updated_at / observed_at
external_revision
media_type
payload / payload_ref
content_hash
request_metadata
idempotency_key
Evidence Ingress validates the source, time, hash, and payload readability. Managed content fixes the body snapshot; a promoted bug fixes the provider revision actually consumed by the conclusion; a promoted incident fixes the primary, forensic, or selected signal revision supporting the timeline/claim.
After the artifact is written to the S3-compatible store, a PostgreSQL transaction creates the Observation/EvidenceArtifact metadata and the downstream outbox event. ObservationAck is returned after commit. Normalization reads the observation as an independent task, so a parser/model failure does not damage the raw evidence and processing can be repeated under the processor revision.
Every source passes through a source-specific parser and then enters the canonical mapper. The parser answers where an external field is; the canonical mapper answers what that field means in SecFusionAgent. The same mapper schema serves the hot and durable paths, with different write targets.
Hot normalization produces the current projection, stable identifiers, changed fields, and priority signals and writes them back to Redis; an ordinary feed record creates no long-term Object/Claim/Relation. Incident signal staging produces anchor/cluster/watch state; durable normalization completes object upsert, external identifier binding, source claim/relation, evidence link, processing run, and KnowledgeChange inside a PostgreSQL transaction, and Incident promotion also maintains the SecurityIncident, timeline events, and the incident revision/outbox. For a snapshot-style provider, a new revision with the same source_id + subject + predicate/relation type supersedes that source's previous current state; differing values from different sources are kept at the same time and left to the current projection to expose as a conflict, rather than being adjudicated silently in M2/M3.
Deterministic mapping covers vulnerability/package/version/CWE/CVSS/reference for CVE/GHSA/NVD/OSV, GitHub repo/issue/PR/commit/release, package/model registry identity, DOI/arXiv work identity, and issuer/jurisdiction/status/version for normative documents. HTML/PDF parsers handle document structure, explicit identifiers, and locator mapping; impact, attack condition, implicit relations, and risk/control mapping go to the M3 semantic processor.
The hot path collapses duplicate updates with the provider namespace + external ID/revision and keeps cross-reference hints. The durable path goes further on cross-source identity: strong identifiers such as CVE/GHSA/DOI/repo/package/commit bind directly to an existing object; an explicit provider cross-reference and a stable normalized key can bind automatically. Similar records without a strong identity stay separate objects and produce candidate_same_as for a later M3/analyst decision.
Object merge keeps every durable observation. NVD, GHSA, vendor advisories, and issues can map to the same vulnerability object while their evidence, times, and source claims continue to exist independently.
M2 produces three stable outputs by retention lifecycle:
HotNormalizationResult
source_id
external_object_id
external_revision
available_at
changed_fields
current_projection_ref
priority_signals
IncidentSignalResult
signal_id
incident_candidate_id
available_at
material_change
source_role
next_watch_at
NormalizationResult
observation_id
knowledge_revision / incident_revision
committed_at
object_ids
claim_ids
relation_ids
timeline_event_ids
candidate_match_ids
processing_run_id
HotNormalizationResult drives Bug Stream priority/enrichment; IncidentSignalResult drives Incident Watch clustering/follow-up; NormalizationResult + KnowledgeChange together with the incident revision/outbox drives durable M3, current projections, and the Indexer. Requirements monitoring latency uses available_at/committed_at of the corresponding lifecycle.
M2 automatically merges identity only through strong identifiers or reproducible deterministic cross-reference. CVE/GHSA/OSV/CNVD/CNNVD, DOI/arXiv, GitHub repository full name, package PURL or ecosystem+name, and commit SHA can act as strong identifiers; name similarity, title similarity, and embedding proximity create candidates only and do not directly merge objects.
Every accepted claim/relation must resolve to a fixed Observation / Artifact and locator. A structured mapper locator must point into the source revision that supports the field/relation; semantic extraction additionally requires the verbatim quote to be a real substring of the fixed chunk. M7 records parser-field accuracy, replay suppression, entity-resolution precision/recall, evidence correctness and conflict preservation separately, so a correct value with an incorrect citation cannot be hidden inside M3 P/R.
packages/enrichment owns M3. Its input can be a HotBugRecord/HotNormalizationResult or an IncidentCandidate/IncidentSignalResult in Redis, or a durable object/evidence in PostgreSQL. Hot bug enrichment writes back the current projection; Incident enrichment updates unresolved questions, source follow-up, and promotion state; when long-term citation is needed it calls PromotionService first and then writes the durable claim/relation/timeline. M3 calls AcquisitionService for all new external material.
The Enrichment Planner creates processor jobs from object type, risk domain, source authority, existing claims, and information gaps.
| Processor | Input | Applicable tasks |
|---|---|---|
| deterministic | structured knowledge | CVSS/CWE/KEV mapping, version compare, taxonomy mapping, fixed graph rules |
| graph / artifact | repo/package/artifact relation | issue–PR–commit–release, dependency, SBOM, container/component propagation |
| semantic | hot/durable source content | attack condition, impact, patch/workaround, TTP, risk/control, implicit relations |
| provider-backed | query spec | NVD/OSV/KEV/GHSA refresh, Internet asset intelligence, commercial/real-time providers |
The Planner uses rules/config to choose the processor combination; the LLM receives the semantic extraction task the Planner has already determined. Every processor produces a ProcessingRun, and once the input revision and processor version are fixed it can be regressed independently. A processor result for a hot subject is written back to the Redis projection by default; after a durable promotion signal appears, the processor waits for the corresponding evidence/object to enter PostgreSQL and then commits the long-term claim/relation.
A provider-backed processor calls AcquisitionService.query(source_id, query_spec, parent_run_id). The IngestEnvelope returned by the Source Adapter enters the hot, incident, or durable M2 path by source retention mode; M3 resumes the parent enrichment job after the corresponding normalized state appears. When a hot bug or incident signal must support a durable investigation/answer, it passes through the Promotion Gate to fix the revision first. NVD lookups, GitHub objects, vendor pages, papers, news/forensic follow-ups, and Internet asset queries share the provenance/dedup contract while keeping their own retention policies.
Internet asset intelligence stores the provider, query, filter/fingerprint, query time, host/service result, and observation time. The mapping from product/version to fingerprint is versioned separately; a possibly affected asset is backed by a concrete provider observation plus mapping revision.
A semantic processor reads the relevant sections from DocumentProjection and sends the source metadata, object candidates, and the schema to extract to the Model Provider together. External body text always sits in a data field; the system instruction, schema, model, and token budget come from code/config.
Model output enters candidate state:
candidate claims
candidate relations
entity candidates
evidence locators
processor/model/prompt revision
The write gate validates the schema, subject/object identity, evidence locator, processing provenance, and semantic duplicates. An entity alignment with high uncertainty stays a candidate; it is promoted to an accepted relation when a strong identifier or a deterministic rule backs it. When a model-generated citation locator cannot be resolved back to the source artifact, that candidate fails outright.
The M3 shared pipeline loads domain processors from 01–04: AI Infra completes issues, fix PRs/commits, affected/fixed versions, and integration along the repo/component/release/runtime graph; Supply Chain computes propagation along package/artifact/image/build/distribution relations; Model/Data links papers, benchmarks, attacks, mitigations, known bypasses, and residual risk; Agent/Application links context, tool, identity, sandbox, network, memory, side effects, and incident paths; Normative Knowledge extracts Requirement/Control/applicability; Research Intelligence produces InsightCandidate.
Domain processors extend object types, claim predicates, and relation types, while evidence, processing provenance, and the write gate share one kernel.
A paper, vendor report, system/model card, or independent research enters M2 as evidence/object first. M3 produces an InsightCandidate according to the promotion criteria in 04, recording the existing claims it supports, refutes, or extends, the evidence maturity, what changed, and the processing revision; source authority continues to be held by the original observation.
A KnowledgeChange is emitted after a knowledge commit. The Indexer builds chunks, metadata, and embeddings for added/changed documents/objects; an old index entry is invalidated by upstream revision. M4–M6 can read canonical knowledge immediately, semantic retrieval reads the corresponding indexed_revision, and the M7 benchmark fixes the knowledge/index revision to stay reproducible.
The formal M3 enrichment vocabulary is owned by packages/intelligence/knowledge/vocabulary.py; the current revision is enrichment-v1. Terms have three scopes: canonical, source_specific, and exploratory. Canonical claim/relation definitions can fix allowed subject/target object types and a benchmarked flag. The Knowledge Writer evaluates term shape against the actual root object: a deterministic/source-asserted unregistered term or shape violation fails the write, while a semantic result with an unknown term or non-canonical shape is retained as exploratory. This prevents free model strings from changing the benchmark without suppressing semantic discovery.
enrichment-v1 fixes twelve dimensions: identity, severity, weakness, product/package, version applicability, fix/remediation, exploit state, exploit likelihood, advisory/reference, asset exposure, research/paper, and incident context. Identity primarily supports M2 normalization diagnostics; internal repository graph edges and open research/incident relations can be canonical yet benchmarked=False. Provider-native github_* / osv_* / kev_* / cnvd_* / cnnvd_* and raw NVD/CVE fields remain available without automatically entering the competition denominator. PoC benchmark semantics use Vulnerability --has-poc→ ExploitArtifact; poc_available is a derived projection. Research/paper v1 benchmarks the stable ResearchWork/Document --discusses-vulnerability→ Vulnerability association, while broader supports/contradicts/evaluates relations stay outside formal P/R until their object model and gold protocol are frozen. Open-text workaround / mitigation / attack_condition remain canonical evidence-backed knowledge, but v1 marks them benchmarked=False until a normalized schema is frozen. Supporting graph edges such as release-contains-commit, asset-runs-product, and asset-version are also canonical/non-benchmarked; the v1 affected-asset gold output is InternetAsset --asset-potentially-affected→ Vulnerability, so inference-support edges are not double-counted in the competition denominator.
Product/version applicability uses affected / not_affected / fixed / under_investigation / unknown. The unified output is not an unscoped claim on Vulnerability; it is a scoped Vulnerability --applicability-status→ Product|Package|SoftwareVersion relation whose qualifier contains at least state and source_semantics, with optional version range / justification. CVE Record 5.x, OSV, NVD CPE configuration, and CSAF/VEX preserve their source-specific range/status semantics and evaluators map them into the common state; GIT ranges require real repository ancestry. Query miss, source miss, and incomparable versions remain unknown; only a structured status, deterministic rule, or VEX justification can produce not_affected.
Current code covers CVSS/CWE plus canonical has-weakness, source-specific CVE5/OSV/NVD-CPE/CSAF-VEX applicability, CISA KEV, canonical PoC relations, GitHub Advisory + FIRST EPSS point-in-time facts, the GitHub development/fix graph, CNVD/CNNVD identity, AssetObservation, full-NVD-configuration CPE applicability joins, exact paper-CVE relations, and evidence-gated document semantics. What is unified is the canonical applicability output state and relation contract, not the source-specific range/VEX/CPE rules into one string comparator. Remaining backlog is mainly broader authorized live-asset, paper/incident strong-anchor, and semantic-provider denominators, including Incident→Vulnerability coverage not yet selected into the current formal closed set; a frozen vocabulary does not imply a sufficiently broad denominator.
The storage/read handoff has passed a real-PostgreSQL integration gate. The semantic handoff additionally requires M3 to describe each canonical dimension with the same vocabulary as resolved, conflict, unknown, or missing; later M4 investigation planning consumes these gaps and does not create a second enrichment ontology.
The minimum M7 gold unit is fixed by EnrichmentFactKey in packages/evaluation/m1_m3.py: stable root object key/type, vocabulary revision, dimension, claim/relation kind, predicate/relation type, normalized value/target, target object type, qualifier keys + normalized qualifier, and temporal scope. Gold itself is validated against the same registry for subject/target shape, benchmarked, and required qualifiers. A TP requires the tuple, entity/version scope, and EvidenceRef to be correct together. Wrong values, wrong entity/version binding, wrong relations, converting unknown into not_affected, and accepted facts without evidence are FP; missing gold is FN. The competition report uses micro precision/recall and internal diagnostics also report dimension-macro P/R and the M2 quality metrics.
A benchmark case fixes world snapshot, source availability, source revision, vocabulary revision, and evaluator revision. Fixed cases cover structured-only enrichment, multi-source conflict, CVE/OSV/NVD/CSAF applicability, KEV/PoC, time-bounded assets, paper relations, Incident strong anchors, alias ambiguity, and wrong-version traps. Open-ended semantic relations are not forced through the same closed-set scorer before they enter the vocabulary.
The five paths below cover high-frequency vulnerabilities, structured platforms, Managed Content, Incident Watch, and on-demand enrichment, and they also serve directly as the first integration tests / probes.
A CVE/GHSA/NVD/OSV/KEV delta updates the Redis hot working set first. Newly published records and revision changes to existing records both enter the hot path; M2 performs cheap normalization and a field-level diff, and M3 continues enrichment only for high-priority objects such as a target repo match, a KEV/PoC/critical signal, or an active query/investigation. Ordinary records are evicted after TTL/LFU, and a historical miss is re-fetched directly from the canonical provider.
When a vulnerability enters the target repo graph, an M4/M5 investigation, an M6 citation, a source conflict, a benchmark, or an analyst review, the Promotion Gate fixes the current provider revision and writes durable evidence / canonical knowledge. GitHub issue–PR–commit–release, affected/fixed versions, and supply-chain relations are then retained long term. Vulnerability feed freshness is thus carried by the bounded working set, and the long-term database accumulates only the new knowledge that SecFusionAgent actually uses or produces.
The GitHub selective_index has expanded from stable target-repo identity and metadata/current claims to first-class Repo / Issue / PullRequest / Commit / Release objects and an explicit reference graph. The M3 graph processor maintains deterministic relations such as issue→PR→commit→release and provides a structured path for deterministic fix-boundary verification; package/container/dependency relations continue to expand with source coverage, while OSV, SBOM, registry, or scanner results verify whether an artifact actually contains an affected version. Long issues, vendor PSIRT material, and patch explanations are stored as managed content and passed to the semantic processor for implicit-condition extraction.
Managed Content now has executable HTML/PDF/plain paths: external content is fixed to a source revision before entering Document/DocumentRevision and DocumentChunk, and each chunk keeps a locator back to the original revision. PostgreSQL FTS provides the lexical index, pgvector columns hold provider-neutral dense state, and the semantic processor requires each proposal to carry a verbatim quote locatable in the chunk or rejects the write. The OpenAI-compatible model/embedding adapter is wired into the indexing worker. Formal Product QA proves that a compatible model provider can execute, but live M3 semantic extraction + dense indexing remains a separate workload/denominator; the current CompetitionReport does not use Product model calls as a substitute for that path. OpenReview, formal proceedings, and additional vendor/security-research sources continue through source onboarding; hybrid query planning, fusion, and reranking are decided by Technical Design 2 and M7 probes.
A breaking source produces a SignalItem first, and the Incident Correlator extracts verifiable anchors and clusters them. The current first batch connects standard RSS breaking sources and extracts CVE/GHSA/transaction/address by rule; weak title similarity takes no part in durable promotion. source_family/upstream_source collapses reprint dependencies, and multi-source corroboration requires two independent sources supporting the same strong anchor. Affected project official channels, security vendor analysis, on-chain evidence, and threat-intel follow-up are onboarded according to the 05 source plan.
When an event meets the promotion conditions, M2 fixes the key source revisions and PostgreSQL creates the SecurityIncident and an append-only timeline. M3 continuously links Vulnerability, Product/Repo, ThreatActor, Address/Transaction, Advisory, Patch/Commit, IOC, and affected Internet asset; M4 consumes the timeline and evidence chain to form confirmed/conflict/unknown. A postmortem after the event ends can enter the Insight Corpus, and the whole investigation trajectory enters Experience Evolution.
M3 creates an on-demand query when a vulnerability/product/version needs affected public assets. The Censys/Shodan/FOFA/ZoomEye adapters write the query/filter, provider response, and observation time into M2; M3 uses version/fingerprint mapping to produce a candidate InternetAsset/ServiceEndpoint relation. The relation keeps the query time and provider observation, so an asset judgement always carries a time boundary.
Knowledge records what is already known about an object; Experience records how to obtain evidence more reliably for a class of investigation task. The bug hot cache can be evicted and large numbers of historical CVEs can stay in the external canonical source; a reusable method formed through real investigation is a long-term asset of SecFusionAgent and, like the Insight Corpus, needs versions, provenance, and evaluation.
InvestigationCase defines one evaluable task and stores task_signature, target objects, goal, initial knowledge revision, constraints, and rubric. A Case can come from an M5 investigation or from an M7 benchmark replay.
InvestigationTrajectory stores one actual execution: which knowledge/experience was read, which sources/tools were called, which evidence was observed, which hypotheses were tried, which failures and recoveries occurred, and the final outcome, latency, tool calls, and cost. A Trajectory uses append-only step/event records; the raw transcript or bulky tool output can enter the artifact store, while PostgreSQL stores the structured events and references.
Experience is a reusable investigation policy extracted from several trajectories. It does not store the facts of one CVE but the applicable conditions and investigation actions:
experience_id / version
name
task_signature
scope
trigger_signals
applicable_conditions
recommended_actions
evidence_expectation
failure_modes
stop_conditions
fallback_actions
supporting_trajectories
validation_summary
success_count / failure_count / partial_count
last_validated_at
status = candidate | validated | active | deprecated
supersedes
Failure experience is a first-class object. For example, "PR merged_at does not directly imply a fixed release" records the failure pattern, the discriminating conditions, and an alternative evidence path; when a later planner retrieves it, the planner can actively avoid an investigation path already proven wrong.
flowchart LR
Case[Investigation Case]
Retrieve[Retrieve Active Experiences]
Plan[M5 Plan / Investigation]
Traj[Trajectory]
Eval[M7 Outcome Evaluation]
Extract[Experience Extraction]
Candidate[Experience Candidate]
Similar[Retrieve Similar Experience]
Update[Create / Merge / Supersede / Reject]
Replay[Historical Case Replay]
Active[Active Experience]
Case --> Retrieve --> Plan --> Traj --> Eval --> Extract --> Candidate
Candidate --> Similar --> Update --> Replay --> Active
Active --> Retrieve
Before planning, M5 retrieves active experiences by task signature, target/risk domain, repo family, and the current evidence gap; a retrieved experience carries its version and applicable conditions and influences action proposals, but cannot override current evidence. The execution records the experience versions actually used, so M7 can measure the success/failure/partial outcome after using a given experience.
After a task ends, the Experience Extractor draws procedures, failure patterns, stop conditions, or fallback candidates from high-information trajectories. The system first retrieves similar experiences by vector/metadata and then decides create / merge / supersede / reject. The update result records the rationale, supporting trajectories, and version relations; success/failure/partial and last_validated_at are aggregated from M7 evaluation and feed recall ranking and deprecation. Trajectory analysis and semantic-gradient-style updates from OpenViking/RFC can serve as later optimizer references; the first version uses explicit candidates plus replay evaluation.
Candidate promotion depends on M7 evaluation, rather than treating a trajectory extraction result directly as active memory. The current Experience Store enforces the candidate → validated → active state machine and requires replay/online support before validated; the concrete multi-case activation threshold is still decided by the M7 benchmark policy. An active experience enters superseded/deprecated when it keeps failing in later cases, when the applicable repo/provider changes structurally, or when new evidence refutes it; activating a new version automatically deprecates the old active version.
M4 holds the current investigation state of a Case and confirmed/conflict/unknown; M5 holds Trajectory and experience retrieval/use; M7 holds outcome/rubric/replay evaluation and is responsible for the Experience validation signal. The Experience Store belongs to the investigation capability, while factual evidence is still held by the M2/M3 knowledge plane.
Experience is a planning hint / policy memory and has no source authority. An investigation action produced by any experience still needs to obtain evidence from the current source again; the final claim continues to satisfy C1. An external web page, tool output, and peer-agent message cannot write an active Experience directly and must pass through the trajectory, evaluation, and versioned update pipeline.
The OpenViking Session commit already treats cases, trajectories, and experiences as long-term memory types and links the full Agent Evolution pipeline when experiences is enabled. Its memory extraction uses LLM Extract → Vector Pre-filter → Similar Memory → LLM Dedup Decision → create/merge/delete → write/vectorize; the Agent Evolution API can also query, by Experience, the Trajectories that actually used it and the outcome distribution. This mechanism serves as a direct design reference for SecFusionAgent Experience Evolution.
SecFusionAgent keeps domain ownership: Case rubric, security evidence provenance, tool trajectory, knowledge revision, benchmark result, and Experience validation are held by this project's data model. OpenViking can later attach as an experience retrieval/memory provider through ExperienceStorePort, but SecFusionAgent defines the Experience lifecycle and evaluation contract.
SecFusionAgent accumulates four asset classes over time: Knowledge stores facts and relations, Incident stores real-world events and their evolution, Insight stores new understanding of risk and defense, and Experience stores how to complete an investigation more reliably. They can reference the same evidence, with separate ownership and lifecycle.
A small set of stable ports is kept between M1–M3, and concrete storage and task frameworks stay in the provider layer.
SourceAdapter.discover / fetch / query
AcquisitionService.schedule / query
HotBugIngress.accept -> HotNormalizationResult
EvidenceIngress.accept -> ObservationAck
KnowledgeRepository.read / commit -> KnowledgeChange
HotBugCache.get / admit / touch / evict / pin
IncidentSignalStore.admit / cluster / watch / expire
IncidentRepository.append_event / update_view
PromotionService.promote
ArtifactStore.put / get
RetrievalPort.search
ExperienceStore.retrieve / append_candidate / activate / supersede
ModelProvider.generate_structured
packages/intelligence holds the EvidenceIngress contract; packages/sources produces envelopes that satisfy it. intelligence holds KnowledgeRepository, and M3 writes canonical knowledge through it. packages/monitoring/acquisition holds AcquisitionService, while packages/sources holds only source definitions, provider contracts, and adapters and does not depend on runtime state in reverse. M3/M5 obtain new external evidence through the acquisition service. The dependency direction stays apps → capability package → lower-level contract/provider; runtime entrypoints depend on the domain, and the domain stays decoupled from apps. The current tests/test_architecture_dependencies.py turns the package dependency direction into an executable invariant.
The system uses at-least-once execution + idempotent state transitions. The Celery Broker Redis and the Hot Cache Redis are separate: the broker carries task transport and avoids eviction, and the cache carries the bounded working set and allows TTL/LFU reclamation. Canonical runs, durable evidence, knowledge, and experience versions stay in PostgreSQL; the scheduler/recovery worker re-dispatches tasks from queued/running timeouts and outbox delivery state.
After a durable Observation is written successfully the raw evidence is replayable; when a hot bug record is lost or evicted it is restored from the external canonical source; an expired incident signal does not delete an already promoted Incident/timeline. Failures in Normalization, Enrichment, and Indexing are recorded in their own ProcessingRun and retried independently. Promotion completes before the durable conclusion, so the final evidence does not depend on an already evicted cache. A source failure is isolated per source, and a persistent failure reaches alerting through M8 observability.
A transactional outbox sits between the database write and the downstream task: the knowledge transaction writes the domain state, KnowledgeChange, and outbox_events together; the dispatcher feeds committed events into Celery, and the consumer uses the event ID / input revision as the idempotency key. If the process crashes after the DB commit, the outbox can still restore downstream processing.
The first version of the code establishes the following responsibilities according to the current repo standard, and creates directories once the corresponding capability starts implementing.
apps/
├── api/ # FastAPI bootstrap
└── worker/ # Celery worker / scheduler bootstrap
packages/
├── sources/
│ ├── registry/ # SourceDefinition + config loader
│ └── adapters/ # GitHub, NVD, OSV, KEV, arXiv, RSS, asset ...
├── monitoring/
│ ├── acquisition/ # AcquisitionService + on-demand run provenance
│ ├── scheduler/ # due-source + incident follow-up scheduling
│ ├── hot_window.py # hot_window collector
│ ├── managed_content.py # durable_managed collector
│ ├── structured_index.py # selective_index collector
│ └── incident_signal.py # incident_signal collector
├── intelligence/
│ ├── incident/ # SignalItem / IncidentCandidate / clustering / timeline
│ ├── hot_cache/ # Redis hot working set + admission/eviction
│ ├── ingestion/ # durable EvidenceIngress
│ ├── promotion/ # Hot Bug promotion
│ ├── knowledge/ # Object/Claim/Relation contract + evidence-backed writer/read
│ ├── normalization/ # parser + canonical mapper + identity/diff
│ ├── structured/ # structured source normalizer, e.g. GitHub Repo
│ ├── documents/ # managed document parser/chunker/revision
│ ├── projections/ # rebuildable current views
│ ├── storage/ # PostgreSQL models + S3-compatible ArtifactStore provider
│ └── retrieval/ # lexical/dense index state + provider contracts; query planning belongs to M4
├── enrichment/
│ ├── processors/ # provider-backed vulnerability enrichment
│ ├── graph/ # GitHub reference graph + deterministic fix boundary
│ ├── semantic/ # evidence-gated managed-document semantic extraction
│ ├── providers/ # OpenAI-compatible model/embedding adapter
│ └── assets/ # time-bounded Internet asset observation orchestration
├── investigation/
│ ├── cases/ # InvestigationCase / context handoff
│ ├── trajectory/ # append-only investigation trajectory
│ └── experience/ # Experience Store / evolution lifecycle
└── shared/ # business-agnostic config/logging/clock foundations
Database migrations belong to the package that owns the semantics of the corresponding tables, but are all released through the Alembic migration history. A source adapter does not depend on monitoring/intelligence; monitoring composes source adapters with M2 ingestion; intelligence/knowledge holds the canonical write contract; enrichment depends on intelligence/knowledge and monitoring/acquisition. This dependency graph stays one-directional, M1/M2/M3 cooperate through contracts, and an architecture test prevents reverse imports.
The first implementation round proceeds in independently acceptable vertical slices:
-
Foundation: Docker Compose starts PostgreSQL/pgvector, Redis Broker, Redis Hot Cache, and MinIO; it establishes the FastAPI/worker bootstrap, config loading, SQLAlchemy/Alembic, and
sources/source_state/acquisition_runs/processing_runs/outbox_events. -
Hot bug slice: implements
cvelistV5/NVD deltas, Redis admission/TTL/LFU, HotNormalizationResult, field-level diff, cursor replay, and external miss fallback; it verifies the high-frequency bug working set. - Promotion / canonical knowledge slice: implements the Promotion Gate, durable Evidence Ingress, Object/ExternalIdentifier/Claim/Relation/EvidenceLink, knowledge revision, and the query API; it verifies hot → durable promotion with a real active bug.
- Enrichment slice: implements the NVD/OSV/KEV/GitHub provider-backed processors, deterministic version mapping, and graph processors, then connects the Semantic Processor; M3 external lookups continue through M2 re-entry.
-
Incident Watch slice: connects two breaking sources, one forensic source, and one primary/authority source, and completes signal TTL, cross-source clusters, adaptive follow-up, promotion, timeline, and
current_incident_view. -
Managed content / retrieval slice: implements HTML/PDF projection, evidence locators, chunk/embedding, and
RetrievalPort, then connects arXiv/vendor/research sources andInsightCandidate. -
Current projection slice: connects the vulnerability/fix/repo/incident current views to
KnowledgeChange/incident.changedand verifies incremental invalidation and the multi-tier read path. - Source expansion: adds target repos, normative sources, and Internet asset providers; each new source class also fixes the source role, retention mode, fixture, contract test, and M7 benchmark case.
- Experience slice: establishes the Case/Trajectory/Experience schema, records trajectories from real AI Infra/Incident investigation cases, extracts positive/negative experience, and performs similarity merge and historical replay; M5/M7 later reuse this lifecycle.
Every slice runs from a real external source to a locally queryable state; the next slice reuses the previous slice's contracts.
Historical checkpoint (2026-09-26): at that point the M3→M4 handoff already had an executable gate, and Foundation, Hot Bug, Promotion/Canonical Knowledge, provider-backed Enrichment, the Incident domain, HTML/PDF/plain Managed Content, Current Projection, the GitHub development graph, lexical/dense indexing, the semantic evidence gate, time-bounded Internet asset observation, S3-compatible ArtifactStore, outbox/Celery replay/idempotency, and the Experience storage seam had code/tests; real PostgreSQL/pgvector/FTS, Redis, Celery, and S3-compatible integration had passed. The MinIO/provider-access, Shodan-credential, semantic+dense live-E2E, OSV GIT fixed sample, and Incident-follow-up gaps listed by that checkpoint are historical progress state; current completion is defined by the 2026-10-16 status and formal benchmark evidence above. Task routing, query construction, retrieval fusion/reranking, candidate assembly, Percept construction, and Investigation State remain owned by M4 / Technical Design 2.
These systems validate the architecture seams and lifecycle design; SecFusionAgent reuses mechanisms and interface ideas, while internal ownership is still decided by Requirements and this document.
| Reference | What it contributes | Corresponding SecFusionAgent design |
|---|---|---|
| Foresight News Timeline | security events are maintained as a continuous timeline rather than treating every news flash as an independent endpoint | product reference for IncidentCandidate / timeline watch |
| CertiK Incident Analysis | exploit events form an independent incident analysis recording mechanism, loss, and follow-up | promoted Incident → forensic enrichment / postmortem |
| OpenCTI Automated Import / Analyst Workbench | unified ingestion for connector/stream/TAXII/RSS and similar; after import, content enters the workbench first and then formal knowledge after analyst review | SourceAdapter, candidate/write gate, durable evidence, and the analyst exception path |
| TheHive / Cortex analyzer workflow | case, observable, analyzer enrichment, and investigation context are layered | M4 Case, M5 enrichment/investigation, evidence write-back |
| OpenViking Session / Memory |
cases/trajectories/experiences, commit extraction, vector pre-filter, similar-memory dedup/merge |
Experience Evolution pipeline |
| OpenViking Agent Evolution | query which Trajectories use an Experience, and the success/failure/partial outcome distribution | M7 tracking of experience effectiveness |
| VikingDB | collection/index, vector/scalar/hybrid retrieval, update/delete, and asynchronous index refresh |
RetrievalPort / future vector provider; the first version still uses pgvector |
| UBASE, ByteDance, 2026 | vector indexing under sustained ingestion, memory/SSD hybrid residency, record-level cache tuning of memory/latency/cost | data temperature, hot/cold tiering, and a reference for future large-scale retrieval optimization |
UBASE and VikingDB target vector data far larger than the competition scale; what this document borrows is workload-aware residency, index/update separation, and the provider seam, not their distributed implementations. OpenViking provides a general memory evolution paradigm; SecFusionAgent adds a security-domain Case rubric, evidence provenance, and M7 replay validation.
Technical Review must confirm that the runtime, storage, and code ownership in this document fully carry Requirements. The following short probes run before formal implementation, and their results verify the technology baseline:
| Probe | What actually runs | Pass/fail criteria |
|---|---|---|
| Hot bug lifecycle | CVE/NVD delta: new/update → Redis → diff/enrich → TTL eviction → miss re-fetch → promotion | working set stays bounded; update latency is correct; promoted evidence does not depend on the cache |
| Structured source E2E | GitHub target repo: discover → normalize → relation → replay | cursor, revision, idempotency, outbox, and source failure semantics hold |
| Incident Watch | pick a real incident: 3+ breaking/forensic/primary sources → cluster → timeline → promotion → follow-up | duplicates do not create false corroboration; material changes enter the timeline; low-value signals can expire |
| Managed content E2E | one PDF plus one dynamic web page: snapshot → projection → evidence locator → semantic extraction → embedding | a citation returns to the original text; parser/model upgrades can be recomputed |
| PostgreSQL knowledge/retrieval | run structured, recursive relation, and vector/metadata queries on a repo/vulnerability/package graph plus document chunks | at the current scale, latency/recall support M3/M4; a separate graph/search provider is introduced only on failure |
| On-demand asset | map one vulnerability/product to an asset provider query, store query/revision/time, then replay | the boundary between a timely observation and long-term knowledge is clear, and a provider failure is recoverable |
| Experience evolution | 5–10 known investigation cases: trajectory → candidate → merge/create → replay → outcome | positive/negative experience is traceable; a wrong experience does not enter active after a single success |
OpenCTI/MISP interoperability serves as an integration test: import/export a set of STIX objects and verify the internal object/relation/evidence mapping. If a probe finds that a mature platform can directly replace a self-built provider, the provider is replaced first; the evidence, knowledge, risk, and benchmark contracts in Requirements stay unchanged.
当前产品与验收
需求与设计
- 需求基线 · EN
- 技术设计 1 · EN
- 技术设计 2 · EN
- 技术设计 2A · Application Backend
- 技术设计 3 · Measurement & Evaluation
- 架构图 1：中文 · EN
- 架构图 2：中文 · EN
- 项目网站
专题
- 01 数据源与处理 · EN
- 02 CTI 基线与复用 · EN
- 03 规范知识与政策 · EN
- 04 模型数据与应用安全 · EN
- 05 事件与新闻情报 · EN
来源