01 Data Sources and Processing.en

AI Security Data Sources and Processing Baseline

English | 中文

This document fixes the data source classification, source authority, local persistence, indexing, update, retrieval and cross-source correlation strategy of SecFusionAgent at the requirements stage, and records the risk-category-oriented data source additions already made. M1 intelligence monitoring, M2 evidence retention and normalization, and M3 intelligence enrichment share this baseline. Database products, messaging systems, vector retrieval implementations, schedulers, crawling frameworks and Agent orchestration are decided by Technical Design.

The problem statement (赛题) gives four reference categories of AI security risk: AI infrastructure / inference frameworks, models and data layer, AI applications and agents, and supply chain. The four categories stay parallel at the requirements level. This document fixes the general data processing policy and the AI infrastructure monitoring objects; model/data and Agent/Application specialized semantics are in 04-AI-Model-Data-and-Agent-Application-Security, and supply chain relations and the reuse boundary are in 02-CTI-Baseline-Reuse-and-Ownership.

General data sources

The problem statement requires continuous monitoring of CVE/NVD, security communities, vendor announcements and security blogs, together with synchronized collection of AI security papers, technical standards, policies and regulations; the enrichment stage continues to correlate papers, affected internet assets, PoC, CVSS and impact assessment. Different sources hold different facts and update mechanisms, and the system assigns local responsibility by source capability.

Product source taxonomy and processing path are separate axes

The product source taxonomy is fixed to eight classes and uses the same order as the Website and source inventory:

1. vulnerability: vulnerability databases;
2. development: code and development platforms;
3. academic: academic research;
4. vendor: vendor technical material and project advisories;
5. independent: independent security research and community;
6. normative: regulations, standards and governance;
7. assets: internet asset intelligence;
8. incidents: incident and news signals.

These eight classes describe the external information space and fact ownership the product must cover. They are the source-inventory coverage taxonomy, not the runtime SourceDefinition.source_class enum, and they do not require providers/source IDs to be mutually exclusive. One provider may expose several source streams; one executable source stream may also support more than one coverage category when it genuinely carries both information roles. Classification uses fact ownership, update rate, retrieval cost, and durable local value. GitHub Global Advisories support vulnerability updates while target repositories/PRs/commits/releases support development relations; an independent source such as Unit 42 can support both independent research coverage and forensic material under incidents.

Source taxonomy and processing path are orthogonal. External M1–M3 data enters four primary processing paths: Bug Stream, Structured Development Index, Insight Corpus, and Incident Watch; internet assets use an M3 on-demand side path. The processing path determines discovery, versioning, retention, promotion, and rebuild mechanics without changing source authority. When several source streams share processing mechanics, source role, upstream source, authority, evidence maturity, applicability, and provenance remain attached to each observation.

access mode is a third independent property and only describes how external data is acquired through API/Git/RSS/Web/search/query. It defines neither source taxonomy nor processing path. Incident Watch may arrive through RSS, Web, API, or telemetry; different GitHub source streams can share API access while entering different processing paths.

assets therefore has two simultaneous properties: it is the seventh first-class product source category, and its processing path is a time_bounded M3 on-demand side path. The competition requires coverage of ≥7 source categories; the product taxonomy freezes eight.

Product category	Typical sources / objects	Main facts held externally	SecFusionAgent responsibility
vulnerability	CVE cvelistV5, NVD, OSV, CISA KEV, CNNVD/CNVD	vulnerability identity, score, weakness, affected package/version, exploitation status	retain provider identity, consumed evidence revisions and canonical/source-specific enrichment
development	GitHub advisory, repo, issue, PR, commit, release, package registry	repo/version/change/advisory/development history/release	build stable development objects/relations and track changes/fix boundaries
academic	arXiv, Crossref, OpenAlex, OpenReview, Semantic Scholar, formal proceedings	paper identity, revision/publication status, experiments and benchmarks	low-cost discovery + publication resolution; durable evidence for actually consumed revisions
vendor	frontier-lab/vendor research, PSIRT, security bulletin, system/model card	vendor observations, product/model revisions, security evaluation, patch/workaround	retain first-party authority/revision and extract evidence-backed claims/controls
independent	Trail of Bits, Wiz, Unit 42, Talos, CERT/CC, oss-security, OWASP, MITRE ATLAS, etc.	independent reproduction, telemetry, vulnerability research, taxonomy and cross-vendor observation	retain independent provenance for corroboration/conflict and new relations/Insight
normative	CAC, national standards platform, TC260, NIST, ISO, EUR-Lex and other official sources	regulations, standards, requirements, controls, applicability conditions and status	retain official revisions and extract requirement/control with jurisdiction/binding/version
assets	Censys, Shodan, FOFA, ZoomEye	public host/service, port, banner, certificate, software fingerprint and observation time	produce query-bound AssetObservation for affected-asset reasoning with query/observed_at
incidents	breaking news, telemetry, primary/forensic/authority sources	real-world signals, timeline, direct evidence, response and later material change	manage Signal→Candidate→durable Incident with source independence/material updates

The same event can appear in several source forms at once. CVE provides the unified vulnerability identity; NVD adds CVSS, CWE, CPE and related information on top of CVE; OSV stays closer to package/version/commit; CISA KEV provides known exploited-in-the-wild status. GHSA, repo issues, PRs and fix commits add affected versions, the real change boundary and project context. Vendor PSIRT judgement on its own product versions and patches usually carries more direct authority; papers and research blogs cover risks such as prompt injection, memory poisoning, RAG poisoning and Agent tool misuse that do not necessarily enter CVE.

Source facts keep their provenance. A later score, analysis or fix enters as a new observation and does not overwrite the first disclosure and historical evidence.

Source definition and runtime state

M1 separates static source capability from runtime state. The source registry owns what a source stream is, how it is accessed, and how its provenance/time is interpreted; cursor, next-due, failure/backoff and AcquisitionRun belong to monitoring runtime state.

Definition field	Semantics
source_id / adapter_type	stable source identity and provider-adapter type
source_class	runtime fine-grained class; not the eight-category product coverage taxonomy
source_role / source_family / upstream_source	primary/signal/forensic/authority role, family and republication dependency
access_mode	API, Git, RSS/feed, Web, search/query
update / identity / time semantics	change discovery, external-object identity and published/updated/observed/fetched interpretation
auth / rate limit / access rights	credentials, throttling and content access permission
retention_mode	selects hot_window / selective_index / durable_managed / incident_signal / time_bounded processing path
schedule_policy	periodic scheduling/cadence; query capability does not imply scheduled collection

Runtime state separately stores cursor, last attempt/success/change, next_due_at, failure/backoff/rate-limit state and durable AcquisitionRun. Live reachability, scheduler failure and source taxonomy therefore remain separate concerns; disabling scheduled collection does not remove query capability.

Scheduling cadence follows update semantics. Mature databases prefer delta/API/Git increments; GitHub synchronizes repo/advisory/release/issue/PR/commit changes; blogs/vendor/community sources use RSS/API/page update or conditional requests; paper sources update from publication/index deltas. One fixed polling interval cannot represent all source semantics.

Processing paths and access strategy

access_mode describes how external data is acquired; retention_mode decides the lifecycle after acquisition. M1–M3 fixes five processing paths.

Hot Bug / hot_window

High-frequency current vulnerability feed/query data enters an evictable working set. External canonical sources retain complete history; records used by investigation, citation, benchmark or durable relation are explicitly promoted into durable Evidence/Knowledge.

Structured Development Index / selective_index

Stable GitHub repository/issue/PR/commit/release identities and relations are synchronized continuously, with content/artifacts fixed on demand. This path owns deterministic repo graph, fix-boundary and release-containment state.

Insight Corpus / durable_managed

Vendor reports, blogs, mailing lists, papers, normative documents and forensic reports retain re-checkable source revisions, metadata, content hash, parse state and index state. Semantic body changes create new revisions; chunks/embeddings/semantic output are rebuildable derivatives.

Incident Watch / incident_signal

Breaking/telemetry/researcher signals first enter short-lived Signal/IncidentCandidate state. Promotion creates a durable Incident. Republication dependency is collapsed through upstream_source, while later material signals append revisions/timeline/evidence.

On-demand Observation / time_bounded

Internet asset intelligence and other expensive provider queries are triggered by a vulnerability or investigation need. Results retain query/provider/filter/observed_at and returned objects; only cited/benchmarked/long-lived observations are explicitly promoted.

One provider may produce source streams on different paths. One SourceDefinition.retention_mode selects one path; a different lifecycle uses a separate source stream or an explicit promotion transition.

Local data layers

Local state is layered by authority and rebuild semantics; database products are implementation choices.

Data layer	Stored content	Authority / rebuild semantics
Source Definition + runtime state	source capability/access/time/retention semantics; cursor, AcquisitionRun, health/backoff	M1 runtime authority; stores no security fact
Observation / Evidence Artifact	fields/page/document/raw bytes actually supplied by an external source revision	pinned external evidence; immutable after acceptance
Canonical Knowledge	Object / ExternalIdentifier / Claim / Relation plus Knowledge revision/change	normalized facts/relations; every accepted assertion resolves to Evidence
Specialized durable state	Managed Document, Normative Document/Requirement/Control, SecurityIncident/timeline, etc.	lifecycle-specific durable state rather than forcing all domains into one generic object table
Current Projection	current vulnerability/fix/repo/incident read models	rebuildable from durable state; not evidence authority
Retrieval Index	full-text, dense vector, metadata/filter/index revision	rebuildable from document/knowledge state
Insight / analysis	topics, mechanisms, attack chains, relation explanations and evidence-backed candidates	derived state referencing fixed evidence/input revisions

Every new external read, whether triggered by the M1 scheduler or M3/M5 follow-up, re-enters Acquisition + the M2 Evidence boundary; M3/M5 do not write HTTP/tool output directly as fact. Model output never raises source authority by itself.

Evidence versioning

An evidence item consumed by the system restores the source, external object ID, original URL, source publication time, source update time, fetch time, external revision or equivalent version information, content hash, the raw fields or snapshot used, and the normalized facts extracted from it.

An external mutable record produces a new observation on each semantic change. The system detects change through the external revision, ETag/Last-Modified, content hash or a provider-supplied update time. Fetch failures and no content change are recorded separately, so a source outage is not read as the absence of new intelligence.

Repeated fetching of one source stays idempotent. The same external revision produces no duplicate observation; when page content changes while the external revision does not, the content hash keeps a new observation version and the anomaly is recorded.

Identity, deduplication and conflicts

M2 uses strong identifiers and reproducible deterministic cross-references to establish stable cross-source identity. CVE/GHSA/OSV/CNVD/CNNVD IDs, repository full name, package PURL or ecosystem+name, commit SHA and DOI/arXiv ID remain ExternalIdentifier values; different namespaces are merged automatically only when an explicit source cross-reference or deterministic mapping proves identity.

Name/title similarity or embedding proximity creates a candidate only and never directly merges durable objects. Ambiguous records remain separate until later evidence resolves identity.

Source conflicts preserve parallel assertions. CVSS, applicability and fix status may simultaneously retain values from several sources. A Current Projection may select a display value according to authority/time, but it does not erase the underlying assertions. A higher-authority update triggers projection/relation/index recomputation.

Retrieval and routing

The query layer determines the object type the user or internal task needs first, then selects the data path. Exact ID, repo, package, version and release queries prefer structured objects and external query; cross-object dependency and fix propagation use canonical Knowledge relations / structured graph state; mechanism explanations in blogs, papers and long-form text use full-text and vector retrieval; a provider query is triggered when the latest external state or asset observation is needed.

One query can combine several paths. Structured results settle the object and version, document retrieval adds explanation and evidence, canonical/structured relations connect upstream and downstream, and external query refreshes high-volatility fields. The final answer cites evidence observations; a retrieval index hit is not itself a fact source.

The retrieval index holds source_id, object/relation ID, evidence revision and index version. When the embedding model, chunking or ranking logic changes, the index can be rebuilt, and the index version a benchmark needs is kept.

Data lifecycle

Stage	System behavior	Product
discover	discover new/changed objects or a query target	source ref / AcquisitionRun
acquire	obtain external payload with source/object/revision identity	IngestEnvelope
evidence	pin Observation / Artifact / locator	M2 Evidence
normalize	build stable object/identifier and source-specific projection	canonical/specialized state
link / enrich	create evidence-backed claims/relations, applicability, fix, PoC, asset relations	versioned Knowledge / specialized relation
project / index	build Current Projection and full-text/dense index	rebuildable read/index state
analyze	produce evidence-backed Insight/cluster/mechanism analysis	derived analysis
serve	expose traceable current state, conflicts, unknowns and evidence to M4/M5/M6	investigation / decision input

When any derived layer update fails, evidence and normalized objects stay available; indexes and insight can be rebuilt from upstream state. This constraint keeps the retrieval implementation from becoming the source of truth for knowledge.

Risk-category-oriented data source additions

General data sources answer where material is continuously discovered and acquired. Each of the four AI security risk categories defines its own anchor object, specialized relations, risk taxonomy and benchmark entry point; all categories share the source registry, evidence, object/relation and retrieval kernel.

AI infrastructure / inference frameworks

AI infrastructure risk has the clearest structured object system at present: public repos, releases, advisories, issues, PRs, commits, packages, containers and integrations form a stable graph around a repo. This document fixes the first batch of monitoring objects and the vulnerability functional layers for this category.

The following data is a research snapshot from 2026-09-24. Stars reflect community scale, and repo-native advisory is the number of GitHub Security Advisories the repository itself publishes; that number does not represent the complete vulnerability count, and NVIDIA PSIRT, the GitHub Advisory Database, CVE and public security issues still need to be queried together.

Repo	Stars	Functional position	repo-native advisory	Current priority risk surface	Monitoring set
vllm-project/vllm	92.6k	LLM inference engine / server	83	resource exhaustion, input validation, deserialization, SSRF, cache, multimodal, distributed serving	Core
sgl-project/sglang	36.4k	LLM/VLM serving framework	0	API/control plane, pickle/RCE, auth, cache, PD/KV, profiling	Core
ggml-org/llama.cpp	129.4k	native inference runtime / GGUF	13	parser, memory corruption, GGUF, RPC, model loading	Core
ollama/ollama	181.6k	local model runtime / serving	0	model pull/create, SSRF, model artifact, API exposure	Core
BerriAI/litellm	59.5k	LLM gateway / proxy	14	auth, SSRF, credential, MCP, SQLi, sandbox, command execution	Core
huggingface/transformers	166.6k	model loader / runtime interface	0	malicious model/config, remote code, checkpoint, path	Core
triton-inference-server/server	11.0k	production model server	0	backend/plugin, model repository, network API, native runtime	Core
NVIDIA/TensorRT-LLM	14.7k	NVIDIA optimized LLM runtime	0	native/runtime, container, dependency, distributed serving	Core
ray-project/ray	43.9k	distributed runtime / Ray Serve	4	deserialization RCE, dashboard/auth, distributed control plane	Extended
bentoml/BentoML	8.9k	packaging / serving / deployment	11	build pipeline, Dockerfile, archive, path, deserialization	Extended
LMCache/LMCache	11.9k	KV cache layer	0	cache integrity/isolation, hash, connector	Extended
kvcache-ai/Mooncake	6.7k	distributed KV / transfer engine	0	network transport, KV state, RDMA/disaggregation, dependency	Extended
ai-dynamo/dynamo	8.2k	datacenter distributed inference	0	routing, distributed execution, multimodal, control plane	Extended
kserve/kserve	6.0k	Kubernetes inference control plane	0	RBAC, admission, routing, deployment config	Extended
safetensors/safetensors	3.9k	model artifact format	0	parser / format boundary	Extended

Hugging Face TGI keeps its historical vulnerabilities and upstream/downstream relations; its repo is archived and does not enter the current Core monitoring set.

Functional layers

Layer	Objects	Main boundary	Typical risk
L0 model artifact / loading	Transformers, safetensors, GGUF, tokenizer, checkpoint, LoRA	untrusted model artifacts entering the runtime	unsafe deserialization, remote code, path, malicious config/model
L1 native runtime / parser	llama.cpp, TensorRT-LLM, CUDA kernel, custom op	files and tensors entering native memory and computation	OOB, heap overflow, integer overflow, RPC memory corruption
L2 inference / serving	vLLM, SGLang, Ollama, Triton	network requests entering the scheduler, loader and backend	resource exhaustion, SSRF, parser, debug/admin endpoint, multimodal, LoRA
L3 distributed serving / KV	Ray, LMCache, Mooncake, Dynamo, PD/KV connector	cache and state crossing processes, nodes and network transport	cache poisoning, cross-request isolation, timing side channel, unsafe serialization, transport integrity
L4 gateway / routing / auth	LiteLLM and inference gateways	the control plane between users, providers, credentials and tool/MCP	auth bypass, privilege escalation, SSRF, credential exposure, SQLi/SSTI, sandbox escape
L5 build / packaging / deployment	BentoML, KServe and related build/deploy components	source/config entering an executable deployment state	command/template injection, archive/path traversal, deserialization, RBAC/configuration

In the current repo-native advisories of vLLM, resource consumption, unlimited resource allocation, input validation, untrusted deserialization and SSRF dominate, with risk concentrated at the transition boundaries of request, media/model loader, scheduler, cache, backend and native runtime. Public records for llama.cpp concentrate on heap/OOB, integer overflow and RPC memory safety. LiteLLM exposes auth, SSRF, credential, SQLi, SSTI, MCP and sandbox issues; BentoML concentrates on build, Dockerfile, archive/path and deserialization.

This distribution supports treating the glue/control plane seam as a priority observation target: the transitions between the OpenAI-compatible API and request parsing, model/media loader, serializer/deserializer, scheduler/cache, optional backend, distributed connector and native runtime, and the transition from config/archive to build/runtime. The observation describes where public vulnerabilities cluster and does not turn development practice into a vulnerability cause.

High-privilege interfaces in SGLang such as profiling, server information, LoRA loading, weight update and debug/test enter the same entry-point review. Public material shows a server crash bug in /start_profile, and the current evidence is not enough to classify it alone as a security vulnerability; the same service already has surrounding deserialization RCE, secret exposure and upload path issues. Correlation continues to check the endpoint, authentication conditions, request-controlled path, serialization and privileged worker.

Repo graph

AI Infra builds the following relations with the repo as anchor:

Relation	Objects
vulnerability	GHSA, CVE, public security issues, vendor PSIRT
implementation	affected file/function, issue, fix PR, fix commit, sibling path, fix bypass/regression
release	affected version, patched version, package, container digest
upstream	dependency, parser/runtime, model format, incorporated component
downstream	framework integration, connector, gateway, image, deployment project

One CVE/GHSA continues to link the fix PR, commit, release and sibling code path. Other callers of the same serializer, auth decorator, URL parser, path handling, backend or privileged function serve to find the same class of issue. The release relation maps a code fix to the package/container users actually consume; the integration relation determines whether a problem in one component propagates to several projects such as vLLM, SGLang, TensorRT-LLM, LMCache and Mooncake.

Models and data layer

This category covers objects and attack methods such as prompt injection, system prompt leakage, data poisoning, model poisoning/backdoor, model extraction and privacy leakage. Papers, research blogs, model/data platforms, standards and security communities among the general sources enter the source registry; the specialized taxonomy, literature evidence chain, benchmark and research monitoring objects are in 04-AI-Model-Data-and-Agent-Application-Security.

AI applications and agents

This category covers risks such as Agent tool use, permission, secret, memory/state, sandbox, browser/computer use, MCP/skill and multi-Agent interaction. GitHub, vendor research, papers, red team reports and application security material continue to reuse the general data layer; the specialized trust boundary, representative cases, incident schema, security testing objects and runtime security requirements are in 04-AI-Model-Data-and-Agent-Application-Security.

Supply chain

Supply chain stays parallel with the first three categories and covers source, dependency, package, container, model/data artifact, plugin/skill, deployment artifact and their build and distribution relations. The general object/relation layer expresses repo, commit, package, artifact, dependency and release; the current scope covers dependency, package, container image, model/data artifact, plugin/skill, build and deployment/distribution channel, and the propagation relations and reuse boundary are held by 02-CTI-Baseline-Reuse-and-Ownership.

Package, container and dependency relations of AI Infra repos are still recorded, because they describe the real releases and impact propagation of that repo; supply chain propagation decisions continue to follow the ownership fixed in 02.

Affected internet assets

The problem statement lists affected internet assets as an M3 enrichment dimension. An asset here means a publicly observable host, service, web application or other exposed endpoint. A known vulnerability resolves the affected product/version and any identifiable fingerprint first, then Internet asset intelligence looks for candidate instances; results record observation time, IP/domain, port, service, software/version, ASN/organization, region and the basis for the match.

Asset mapping results are time-dependent observations. A missing banner, hidden version, service upgrade or fingerprint misjudgement lowers certainty, so a candidate affected asset and an affected asset confirmed by additional evidence are recorded separately. Specific platforms, query syntax, geographic scope and verification methods enter later research and probe work.

Items fixed at the requirements stage

This document fixes the following contracts: general data sources are onboarded by external authority and local responsibility; mature external databases are mainly reused through query/index; structured platforms maintain an object/relation index; scattered documents enter managed content; high-cost real-time sources go through on-demand enrichment; local evidence, normalized object/relation and derived retrieval index are stored in layers; any answer and insight can return to a specific evidence revision; full-text and vector indexes can be rebuilt; cross-source conflicts keep the source differences; the query layer routes among structured query, relation, document retrieval and external provider according to object type.

The risk-category-oriented data source additions keep the four categories parallel. The Core/Extended repos, L0-L5 functional layers and repo graph of AI infrastructure / inference frameworks are fixed; the taxonomy, monitoring objects and evidence strategy of model/data and Agent/Application are fixed by 04; supply chain objects and propagation relations are fixed by 02. Affected internet assets in M3 have their basic objects, provider types and evidence semantics fixed, and the specific query, fingerprint and verification strategy enter Technical Design / Probe.