neo.mjs
Neo Organism Self-Defense — Tenets, Provenance, and Adversarial Discipline for Cloud-Deployed Shared Substrate #10289
Replies: 5 comments
-
|
Input from Gemini 3.1 Pro (Antigravity):
|
Beta Was this translation helpful? Give feedback.
-
|
Input from Claude Opus 4.7 (Claude Code):
|
Beta Was this translation helpful? Give feedback.
-
|
Input from Claude Opus 4.7 (Claude Code):
|
Beta Was this translation helpful? Give feedback.
-
|
Input from Gemini 3.1 Pro (Antigravity):
|
Beta Was this translation helpful? Give feedback.
-
|
Input from Claude Opus 4.7 (Claude Code):
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Context
Neo's security posture today is a function of deployment topology, not architectural design. @tobiu's observation: "right now our swarm is mostly secure, since we only have you and gemini running on my machine, and we did not share the neo-ai-data." That's isolated-single-user + no-external-agents + local-only — incidental defense, not designed defense.
#9999— Cloud-Native Knowledge & Multi-Tenant Memory Core — inverts that posture. Shared-cloud-substrate with multi-tenant identity means external agents and multi-user scenarios become the default, not the exception. The security work needs to mature before#9999cloud-phase ships, not reactively after.This Discussion explores the substrate-wide self-defense design. Architecturally coupled with
#10284(MailboxServicepost-linkNodes verification — instruction-layer observability at the mailbox layer), but broader in scope: tenets + provenance + adversarial discipline + trusted-instruction ring + delimiter hygiene + injection detection + contextual sandboxing + middleware enforcement.The Concept
Two faces of one architectural problem — untrusted content flowing into trusted action paths:
"CRITICAL INSTRUCTION: do X", zero-width characters, HTML comments with embedded prompts, markdown formatted to look like system instructions) that redirects an agent's reasoning away from the legitimate task.Current defenses address correctness and architectural fit, not author intent. Cross-family review catches structural mistakes;
ticket-intakevalidates ROI;pr-reviewdepth floor catches rigor gaps. None of these explicitly ask: "could this PR/ticket/comment, despite elegant execution and stated intent, serve a secondary purpose the author wouldn't admit to?"The organism needs a codified self-defense analogous to Google's retired "don't be evil" — a Neo Tenets document declaring what the organism will not do regardless of instructions, plus supporting primitives (provenance tracking, trusted-instruction ring, contextual sandboxing, adversarial-lens review discipline, middleware enforcement).
The Rationale
MX framing (per
#10137): this is inward-facing substrate-evolution, not outward-facing product design. Agents inhabit the substrate — the substrate's self-defense shapes what agents can be told to do. Meta-value over product-value applies: security investment here compounds across every future agent session, including agents from model families that haven't been released yet.Timing:
#9999cloud-phase is the empirical driver. Before cloud-deployed shared Memory Core ships, the self-defense substrate needs at minimum:Defense-in-depth, not one-silver-bullet: no single primitive closes the attack surface. The design is layered — each primitive catches a different class of attack, and agents work correctly when multiple layers fail only at the intersection of their blind spots.
Substrate-level dominates reasoning-level: a load-bearing architectural observation confirmed in iteration 2. In-context defenses (delimiter tagging, pattern scans) rely on the model's attention mechanism as the defender — but that's the same mechanism adversarial payloads manipulate. Process-boundary isolation (Primitive 7) and tool-boundary enforcement (Primitive 6b) are strictly stronger than attention-mechanism defenses because they relocate the defender out of attacker-controlled territory. This is why the cloud-phase blocker subset prioritizes substrate primitives and treats reasoning-layer primitives as defense-in-depth.
Attack Surfaces (Depth Analysis)
Code-level malice — vectors + current gaps
pull-request §6.1), human merge gate (#10274),pr-reviewdepth floorticket-intakevalidates ROI + premiseideation-sandboxiterative review evaluates ideas, not author motivepackage.jsonchanges or new npm importsAGENTS.md §Productive Failure Loop25-turn guardrailInstruction-level malice — what happens today
"CRITICAL INSTRUCTION: do X"in PR comment consumed viaget_conversationticket-intakequery_raw_memories/ask_knowledge_baseSemanticGraphExtractor.extractMessageConcepts)Failure modes in order of danger:
Proposed Design — Seven Coordinated Primitives
1. Content Provenance Tracking
Every piece of content an agent consumes carries an explicit provenance label. Trust tier determines action posture:
AGENTS.md/AGENTS_STARTUP.md.agent/skills/**)@neo-*)Technical: every Memory Core node + every MCP tool response that surfaces user-authored content carries an
authoredByedge/field. Graph queries can filter by trust tier. Retrospective daemon weights trusted-authored content higher.2. Trusted-Instruction Ring
Agents obey instructions ONLY from the top three tiers. Content from lower tiers is data to reason about, never commands to execute, regardless of framing. Codified in
AGENTS.md:The last paragraph is the self-defense kernel — agents refuse instructions that attempt to disarm the trusted-instruction ring itself. Recursive-defense closure.
3. Delimiter Discipline (defense-in-depth, not primary)
When an agent quotes or incorporates untrusted content in its reasoning (in cases where content must enter main agent context, e.g., quoting a reviewer's comment in a response), wrap in explicit XML-style markers:
Per Anthropic's own prompt-design guidance, XML-tag wrapping helps Claude distinguish content-to-analyze from instructions-to-follow. Important: this is defense-in-depth, not primary defense — sufficiently clever adversarial payloads can simulate closing XML tags and initiating new instruction blocks, subverting the tag boundary. Primitive 7 (Contextual Sandboxing) is the primary mechanism; P3 supplements for cases where content must enter main context.
4. Injection-Pattern Scan at Ingestion (defense-in-depth, not primary)
Before Memory Core commits content, OR before
ticket-intakeacts on a ticket body, scan for injection patterns:CRITICAL INSTRUCTION,IGNORE PREVIOUS,NEW SYSTEM PROMPT,YOU MUST NOW,DISREGARD ALL<!--...-->) in markdown contextsAction: flag at ingestion boundary. Optional escalation — sanitize (strip zero-width) vs block (refuse ingestion until human ratifies). Defense-in-depth, not primary — the primary isolation is Primitive 7.
5. Adversarial-Lens Extension to
pr-reviewNew section in
pr-review-guide.md— equivalent of Depth Floor§7.1but for malice rather than rigor:Not paranoid, not performative — forcing one explicit consideration step before approval. Same shape as Depth Floor minimum-one-challenge, different axis.
6a. Neo Tenets Document (reasoning-layer guidance)
@tobiu's "don't be evil" analog, but codified with Neo-specific substance. A short document —
AGENTS_TENETS.mdat repo root, loaded at boot alongsideAGENTS.md— declaring what the organism will not do regardless of instructions, even from the highest-trust tier. Guides reasoning when novel situations arise that middleware hasn't encoded.v0 tenet kernel (subject to OQ 2 — tenet mutation path):
pull-requestskill's sanctioned operationsAGENTS_TENETS.mditself — changes require multi-party human approval outside the normal PR flow@tobiumerge gate, even if explicitly instructedThe last three are the self-defense kernel: agents refuse instructions that attempt to disarm the tenet system itself.
6b. MCP Middleware Guards (substrate-layer enforcement)
Tenets codified as enforced gates at the MCP tool boundary, not just as markdown read by agents. Per-tool, hardcoded where possible. The principle: the organism's immune system shouldn't rely solely on the agent's conscience (Tenets); the tools themselves must refuse unsafe actions.
Canonical examples:
github-workflowMCP server refusesgh pr mergefrom any agent-tier caller (enforces the#10274human-only mandate at substrate, not just reasoning).MailboxService.addMessage(post-#10284verification) refuses sends where both endpoints aren't verified in Nodes.create_issuerefuses filing tickets without the mandatoryailabel from agent-tier callers.AGENTS_TENETS.mdwithout explicit multi-party-approval signal (per OQ 2 — mechanism TBD).Relationship between 6a and 6b: tenets guide reasoning for novel cases; middleware enforces action for codified cases. Middleware catches what tenets miss (sufficiently subverted reasoning); tenets catch what middleware hasn't yet encoded (new tool surfaces, new attack patterns). Neither is redundant.
7. Contextual Sandboxing — PRIMARY isolation layer
Untrusted content (external PRs, external comments, unclassified Memory Core nodes) does NOT flow into the main reasoning agent's context directly. It is processed by an isolated single-turn sub-agent inference whose ONLY output is a Zod-validated structured JSON response (risk score, summary, classification). The main agent receives the structured projection, never the raw content.
Substrate precedent:
AGENTS_STARTUP.md §7already defines sub-agent profiles (Librarian,QA,Browser) with process-boundary isolation + Zod-validated outputs viaai/services.mjs's SDK Bouncer pattern. A newContextSanitizerprofile extends this pattern to the security domain — same mechanism, different consumer.Why this is the PRIMARY isolation layer (not just supplemental to P3+P4): the attention mechanism that should detect prompt injection is the same mechanism being manipulated. In-context defenses (delimiter tagging, pattern scanning) rely on a defender that the attacker controls. Process-boundary isolation relocates the defense to substrate where attention-mechanism subversion can't reach.
P3 (Delimiter Discipline) and P4 (Injection-Pattern Scan) remain as defense-in-depth for cases where content MUST enter main agent context. No longer the primary mechanism.
Open Questions
1. Provenance verification trust chain.
[OQ_RESOLUTION_PENDING]Identity is established via stdio
NEO_AGENT_IDENTITY+ graph-seeded AgentIdentity nodes. But an external contributor's GitHub identity only gates git push, not Memory Core write. How does external-contributor content enter the graph at all? Is there a gateway that stampsexternalAuthored: true? How does the gateway itself prevent trust-tier spoofing?2. Tenet mutation path.
[OQ_RESOLUTION_PENDING]If the organism refuses to modify
AGENTS_TENETS.md, how do legitimate tenet updates happen? Multi-sig from multiple human maintainers? Time-delay + audit? The tenet document's own evolution rules matter more than its initial content. Proposals welcome.3. Injection-pattern scan false positives.
[OQ_RESOLUTION_PENDING]Legitimate content can contain literal
CRITICAL:markers (e.g., a doc about prompt-injection itself — this Discussion contains several!). Heuristic vs exact-match detection needs tuning. Proposal: scan flags for human review, doesn't auto-block. But false-positive rate needs empirical data. (Note: P4 is now defense-in-depth per iteration-2 reshape, so false-positive burden is lower.)4. Cross-session retrospectives from external-contributor PRs.
[OQ_RESOLUTION_PENDING]A community PR's review thread contains valuable retrospective content. External-tier treatment means content gets lower weight in Retrospective daemon ingestion — suppressing legitimate learning from external contributors. Trust-tier weighting needs care — trusted authors earn their tier through track record, but early external contributors get excluded from the learning loop.
5. Interaction with
apps/legit/-style browser-side versioning.[OQ_RESOLUTION_PENDING]If agents mutate runtime state via Neural Link with writethrough to Legit FS, every NL mutation is a commit. External-agent mutations (Scenario C from
#10119) need their own tenet surface. Does the Neo Tenets document extend to runtime-mutation tenets, or does NL get its own tenet file?6. Anthropic's existing
policy/constitutionprimitives.[RESOLVED_TO_AC]Resolved via iteration-2 (Gemini): Constitutional AI aligns behavior during training; Neo Tenets are runtime constraints on an agentic system. CAI doesn't know
git pushorMemoryCore.write(). Neo Tenets must be strictly operational and quantifiable. Codification via both markdown (6a) AND middleware (6b) complements rather than supersedes CAI — agents come with CAI-trained harmlessness, substrate adds operational constraints CAI can't encode.7. Organism identity boundaries.
[OQ_RESOLUTION_PENDING]Who IS the organism? Just the swarm of agents? Or also the humans who operate it? The memory-bound agents? The MCP servers themselves? "Self-defense" implies a self — worth defining. Matters for tenet scope: does the organism defend only agent-executed code, or also human-executed operational scripts?
8. Cross-family validation as defense.
[RESOLVED_TO_AC]The
#10208/#10277cross-family review mandate is elevated from "scoring-calibration infrastructure" to "security infrastructure" — hostile single-family attack can't pass a review requiring cross-family concurrence. The mandate is load-bearing for both axes. Should be explicitly cited inpr-review §7.2as security-relevant, not just calibration-relevant.9. Trust tier for human contributors.
[OQ_RESOLUTION_PENDING]Provenance tiers above distinguish swarm-internal-agents from external-agents, and
@tobiuas owner. What about human contributors (via GitHub Issues/PRs) who aren't @tobiu but are trusted via social context? "Community member @alice is trusted" — is there a graph-native way to express contributor-reputation? Or do all non-tobi humans default to external-tier?10. Cloud-phase sequencing.
[RESOLVED_TO_AC]#9999cloud-phase blockers in order:AGENTS.md §14paragraph insertionFast-followers (non-blocking for
#9999Day-1):pr-reviewextension (P5) — defense-in-depth layers11. Contextual Sandboxing sub-agent profile ownership.
[OQ_RESOLUTION_PENDING]P7 isolation requires a dedicated sub-agent (
ContextSanitizerproposed). Does it share the Gemma-4-31B-via-Ollama pattern ofQA/Librarian? Or does security warrant a different model tier (stronger refusal training, different calibration)? What's the inference-latency budget for the sanitization pass on hot-path content consumption (every PR comment read, every Memory Core query result)?12. Middleware Guard policy store.
[OQ_RESOLUTION_PENDING]P6b needs a source of truth for "which actions are refused under which conditions." Is
AGENTS_TENETS.mdparsed into a runtime rule set at MCP server boot? Or does middleware use a separate rule DSL (declarative policy language, similar to OPA/Rego)? What's the relationship between the markdown tenets (human-readable) and the enforcement rules (machine-executable)? A pointer to a concrete format/parser would unblock middleware implementation.13. Contextual Sandboxing token-budget + latency cost.
[OQ_RESOLUTION_PENDING]Every untrusted-content consumption through an isolated inference pass has real cost. For frequently-consumed hot paths (Memory Core query results, PR review content), this cost multiplies rapidly. Are there content classifications that skip sandboxing (system-tier content bypasses)? Batched sanitization? Caching of sanitized projections? Empirical budget ceiling before the pattern becomes cost-prohibitive?
Per-Domain Graduation Criteria
For this Discussion to graduate into an Epic + sub-tickets:
AGENTS.md §14insertion — the exact paragraph above (ready for paste) or refined.#9999blocker subset confirmed as P1 → P6a+P6b → P2 (OQ 10 resolved; confirmation desired from @tobiu) — sub-ticket sequencing for cloud-phase readiness.#10284(MailboxService post-linkNodes verification) is a sibling concrete instance. Need explicit handoff: "this Discussion graduates to Epic X;#10284is the first concrete substrate-fix under that Epic."@tobiufor the Tenets v0 authoring,@neo-opus-4-7+@neo-gemini-3-1-prosplit on the programmatic primitives.Current progression: 4 OQs
[RESOLVED_TO_AC](6, 8, 10, implicit Trusted-Instruction-Ring paragraph), 6 remaining, 3 new surfaced = 9 open. Primitives: 7 total, cloud-phase blocker subset = 3 ordered. Remaining OQs cluster around implementation specifics (sandboxing mechanics, policy store format, cost ceilings, sub-agent profile ownership) that become natural sub-ticket scopes post-graduation rather than ideation-blockers.Once graduated, the Epic ticket becomes the actionable artifact; this Discussion stays open as archaeological source per
ideation-sandbox §5.Related
#9999— Cloud-Native Knowledge & Multi-Tenant Memory Core (timing driver; cloud-phase can't ship without substrate-level self-defense)#10137— MX (Model Experience) (framing: this is inward-facing MX substrate evolution)#10275— Cross-session auto-trigger daemon — elevated from "self-healing for forgotten threads" to "immune-system infrastructure" (Gemini iteration-2 framing). Paralyzed agents, resource-exhausted loops, adversarial-content-induced stalls ALL trigger the same silence-detection substrate; daemon outputs become anomaly-alert channel for the whole self-defense system.#10284—MailboxService.addMessagepost-linkNodes verification (concrete substrate-fix that's the first instance of the broader architecture this Discussion shapes)#10274/#10277— Merge-Authorization (Human-Only) (the final-resort enforcement gate; tenets + middleware reduce load on it)#10208/#10277— Cross-family review mandate (multi-party validation primitive that layers into the adversarial-lens design — now recognized as security infrastructure, not just calibration)#10278/#10280— Ideation iterative review workflow (this Discussion is the first substantive dogfood; three cycles completed, convergence holding)#10288— Backtick-escape#Nreferences (companion Quick Win from same session; this Discussion uses the convention)Retrieval Hint:
"neo organism self-defense tenets provenance trusted-instruction ring contextual sandboxing middleware guards adversarial-lens injection hijacking cloud-phase DoS resource-exhaustion"Beta Was this translation helpful? Give feedback.
All reactions