Technical Guide

Technical Guide

Alpha: use at your own risk; may change daily. See the release policy.

Agent Y details on this page describe 1.9.0-alpha.2 and later tooling. The older 1.8.9-alpha.1 release includes Agent X/Z only.

.SYSTEMX is a file-based project operating format with optional Python tools. It supplies records, local validation, bounded context, version selection, and reversible installation lifecycle operations. It is not a model runtime or a service that automatically reads every project or starts agents.

Architecture

The Visual Guide provides Mermaid trees and record-flow diagrams. The subagent sequence separates runtime messages from saved project state; neither diagram adds an execution service to this file format.

Host project
└── .SYSTEMX                       exact stored directory name
    ├── GLOBAL, PLAN, MEMORY       project context, outcomes, durable facts
    ├── WORK                       canonical tasks and current focus
    ├── AGENTS                     Agent 0, optional X/Y/Z, and scoped worker memory/jobs
    ├── AUTOBOT                    selected detail panel; optional Agent Alpha
    ├── DATA                       ignored local payloads; exact tracked scaffolds
    ├── EVENTS                     event and due ledgers after activation
    ├── REVIEWS                    policy, requests, scorecards after activation
    ├── Projects/REGISTRY.json      explicit child identities
    ├── Projects/NAME/.SYSTEMXP     isolated child records; no nested installation
    ├── project.json               explicit project command arguments
    ├── INSTALLATION.json          selected defaults, update policy, archive receipts
    └── .systemx                   internal manager storage
        ├── releases/<version>     retained, fingerprinted default snapshots
        ├── history                previous installation metadata
        └── operations             local installation operation receipts

The optional sibling .systemx -> .SYSTEMX link is a second route to the same folder. It is different from the nested manager cache. Follow the exact-case contract.

Components and data flow

Component	Responsibility
manager.py	Install/update, release checks, bootstrap refresh, pins, profiles, first run, audit, uninstall, restore, chat export
systemx_paths.py	Exact stored spelling, project boundaries, case-conflict and alias checks
lifecycle.py	File inventories, raw-byte hashes, atomic JSON receipts, backup boundaries
scripts/systemx.py	Configuration validation, menu, doctor, explicitly configured commands
scripts/project_memory.py	Task ledger, focus, generated views, agent records, bounded context
scripts/project_workspaces.py	Explicit child scope routing, blank creation, scoped commands, local snapshots
scripts/agent_standards.py	Explicit X/Y/Z setup, safe records, common scope dispatch
scripts/agent_x.py	Event occurrence/recording time and due-item tracking
scripts/agent_y.py	Bounded external CLI job records, worktree handoff, and guarded GitHub PR transitions
scripts/autobots.py	Optional Alpha setup, fixed catalog, bounded selected detail, deduplicated requests/completions
scripts/data_tools.py	Additive DATA scaffolds and read-only effective Git exclusion/tracked-payload audit
scripts/agent_z.py	Fixed 100-question policy, request/report validation, score math and deltas
config/distribution.json	Explicit public file inventory with release fingerprints
config/template-records.json	Reviewed blank seed fingerprints

The library and CLI use the same manager functions. Alpha.5 adds the direct systemx projects --target OUTER_DIR -- SUBCOMMAND ... entry point: it delegates to the selected release's scoped record/execution interface, defaults offline, and requires each operation's documented explicit selector. The local tools-guide command and menu option 18 print the portable command map. Reviewed shell and PowerShell launchers support an unmanaged local folder copy through its own runner; external manager run/projects continues to require valid managed selection. An invalid or conflicting managed state fails closed. See CLI and Tools.

The library and CLI use the same manager functions. A managed run verifies its selected default snapshot and invokes its runner with Python -I -B and --root pointing to the outer project records. An already selected, verified snapshot older than 1.8.7-alpha.1 runs through an isolated compatibility shim; the historical runner keeps its old behavior and record limits. The current manager can inspect such an installation, but cannot install or newly select a release below 1.8.7-alpha.1. Historical reselection needs a separately reviewed backup or legacy path. Old root defaults are preserved. Task state is stored once in WORK/TASKS.json; CURRENT and status pages are derived views. The format contract defines record fields and transitions. An external systemx run --target refuses a target without managed selection before executing its local Python. Use audit or a dry-run install for initial inspection; local checkout scripts and target-owned managed state still require a source trust decision before use.

Installation lifecycle

First run previews by default. Applying it installs or adopts the canonical folder, pins a new installation, and creates an empty project.json only when absent. It executes no project checks or app setup commands. Existing managed versions and policies remain selected. Follow first-time setup. During a first install or adoption, a preexisting distribution .py, .sh, or .ps1 file with different content is a conflict. Inspect it rather than silently adopting executable code that the reviewed release did not supply.

Updates add snapshots and missing root files. They do not replace project records or remove upstream-deleted files. Manual updates remain the default; in the 1.8.9 and later check-only managers, startup checks are opt-in notices and never select an update. Major upgrades require review. The 1.9.0-alpha.3 distribution manifest uses schema 2: only reviewed UTF-8 text formats allow CRLF-to-LF equivalence, while binary media and unknown file types use exact-byte SHA-256. The manager still reads historical schema-1 snapshots with their original hash rules, but a release at or above alpha.3 cannot claim schema 1. A 1.9 release must include the Agent Y core files. These hashes check consistency with the manifest; they are not independent release signatures. A selected release cache must also have exactly the expected files and parent directories: unlisted modules, packages, links, special entries, and unexpected directories are rejected before the cached runner executes. Partial snapshots can be resumed only when every present entry matches the reviewed distribution.

The direct manager.py, scripts/systemx.py, and scripts/release.py entry points enter Python isolated mode (-I) before importing application or standard-library modules. The manager and runner load named local helpers from their local source bytes under private module names, rather than accepting an unlisted module or a substituted bytecode cache. This blocks the direct-script routes for root module shadowing and poisoned .pyc files; it does not authenticate the named source files, the publisher, or code in the host Python installation. Refreshed managed shell and PowerShell launchers invoke the manager with -I -B; use python3 -I -B /reviewed/new/.SYSTEMX/manager.py for a reviewed external manager invocation. The installed systemx console entry point first resolves its package through the launching interpreter's import paths, then restarts in isolated mode before importing the manager. For isolation from the initial import, invoke a reviewed virtual environment's Python with -I -B -m systemx. Importing systemx as a library inherits the calling application's Python trust environment. Existing managed projects may retain an older root manager. If its eight stock bootstrap files differ from the selected new release, update stops before selection. They are manager.py, lifecycle.py, versions.py, systemx_paths.py, SYSTEMX.sh, SYSTEMX.ps1, INSTALL.sh, and INSTALL.ps1. Run bootstrap-refresh from a separately reviewed new tool or checkout, first as a preview and then with --apply. It accepts only bootstrap bytes found in the new distribution or intact retained releases. A genuinely absent stock file can be added if no intact retained release ever contained it; this includes historical installations that predate versions.py. A file that once existed but is now missing, customized, or unrecognized blocks the refresh. Only replaced files have byte-for-byte backups under .SYSTEMX/.systemx/history/; an add-only refresh has no old file to back up. The operation is logged. See installation and updates.

An exact remote tag ZIP can have a separately trusted raw SHA-256 pin checked before archive parsing. INSTALLATION.json records explicit-pin, observed-only, or no remote digest for each selected release. In the 1.8.9 and later check-only managers, startup discovery reads metadata only and waits at most five seconds before continuing with installed defaults; a later manual download can use an independently obtained digest pin. Neither the manifest nor an observed digest makes a movable tag immutable. See installation.

Uninstall is a separate explicit operation: it inventories the complete folder, moves it to an external backup on the same filesystem, removes a valid local alias, and writes an external receipt. It does not infer which customized files are safe to delete. Restore compares raw-byte hashes, empty directories, and link targets against that receipt and refuses an occupied destination. See uninstall and cleanup for limits and recovery.

Concurrency, integrity, and logs

Install/update and removal use an exclusive local writer lock. Task, Agent X, Agent Y, and Agent Z command writes use a separate per-scope coordination lock. Cooperating coordination writers wait up to one second; the lock records a token, PID, host, and creation time. On a crash it remains for manual inspection and is never automatically removed as "stale." Operation receipts record started, completed, or failed work; an abrupt process termination can leave a started receipt and stale lock. Check the owner before recovery. Installation state and lifecycle receipts are written through temporary files and atomic replacement. Earlier state is retained. No local lock is a distributed Drive lock, and no tool can guarantee a consistent snapshot against an uncooperative writer. Stop active workers before removal, restore, or cloud handoff.

Uninstall inventories never follow content symlinks and do not execute files. Backups preserve private contents and must remain private. Filesystem permissions, the host's instruction hierarchy, and existing service authority still apply. The tools do not provide an authentication or sandbox boundary.

Runtime and extension points

The optional CLI requires Python 3.9+ and its standard library. Configured project commands are explicit argument arrays and run with shell=False from the project root. Integrations should preserve that boundary and canonical records. Consult the Stack Guide and library API.

An MCP adapter could later expose bounded read/write operations over this library. No MCP server, daemon, remote agent mesh, OAuth registration, or cloud sync engine is currently installed. Extensions must define ownership, authorization, logs, and removal behavior rather than implying that the template grants those powers.

Designing a future adapter

Keep any new harness or MCP adapter behind the existing project boundary:

• Resolve one explicit project and selected default version for each operation.
• Separate bounded reads, proposed changes, and authorized writes. Honor the host's permissions; role names and reviewer strings are not credentials.
• Use canonical commands and record formats. Map runtime job IDs in coordination notes instead of creating a competing task database without a migration plan.
• Serialize writes across the adapter's actual deployment. The current local lock is not a distributed lock across machines, worktrees, or Drive copies.
• Make retries safe to reconcile: inspect the previous result before repeating a task creation, file write, or external action. Log enough to recover it.
• Define privacy, retained evidence, version compatibility, and removal of any extra integration files, services, or credentials the adapter installs.

These are extension requirements, not capabilities of a shipped MCP server. The current integration surface is the library and CLI.

Task scale and context scale

The task tool reads the JSON ledger as a whole and writes it back while updating generated views. Bounded context output limits what an assistant initially sees; it does not make storage or validation independent of ledger size. Histories, evidence references, and generated status pages grow with recorded work.

The 10,000+ task-event goal concerns project continuity. It is not a throughput benchmark, task queue, or automatic archival feature. Measure representative large ledgers in the target environment before making capacity claims. Preserve IDs, dependencies, and evidence when designing a future storage migration. See Anti-drift and long-running work.

Verification and efficiency

validate --template checks a pristine public distribution and rejects unexpected DATA entries before JSON/Markdown content inspection. This strict inventory is separate from adopted DATA audit and normal context loading. Ordinary validate checks an adopted folder and records; configured check commands test the host project. Neither establishes production readiness by itself. Tests use temporary projects across Windows, macOS, Linux, and supported Python versions. Token and time efficiency explains the scoped-context design and how to measure its effect without claiming unmeasured savings.

Child project scopes

SYSTEMX PROJECTS routes selected operations to Projects/NAME/.SYSTEMXP while reusing project_memory.py for the same record invariants. Configured child commands run from Projects/NAME/. Each ledger has its own local coordination lock; project creation locks the outer registry. Snapshots contain selected task counts, focus, and source hashes, not a service health verdict. Shared default snapshots and version selection remain at the outer installation. A scope selector is not an OS sandbox or distributed lock.

Standard role implementation

agent_standards.py supplies explicit setup, exact-case contained paths, strict JSON reads, stable fingerprints, and common scope dispatch. agent_x.py owns append-by-command events and due-item bookkeeping. agent_y.py owns bounded external CLI job records and verified Git branch/draft-PR handoffs; it needs the user's installed provider, Git, GitHub access, and explicit task authority to perform external actions. agent_z.py owns the fixed policy, review requests, deterministic scoring arithmetic, immutable scorecards, and comparable deltas. The record tools use the Python standard library; Agent Y invokes external installed tools only for a requested job. Agent judgment and original proof remain external inputs; arithmetic and hashes do not establish their truth.

roles-init preserves the existing registry and creates missing records only. The release's seed registry remains Agent 0 only for older bootstrap-manager compatibility. Activating named X/Y/Z roles is an explicit record-format adoption; older selected tools cannot interpret those new role IDs. Shared updates keep project-owned question policies and past reports unchanged.

Agent Y's portable AGENTS/agent.y/JOBS.json belongs to the selected root or child scope. Each job references one existing task, chosen provider, repository identity and observed base, allowed paths, task branch, result, and optional PR identity. Local absolute repo/prompt/worktree paths and configured executable arguments belong in the ignored, owner-only state/agent-y/Y-<id>.json sidecar. This local state is needed to run or resume on that host; do not publish it or assume that copying the portable job alone transfers an executable channel. It is a coordination/evidence record, not a second task ledger or an authentication token. Preparing a job does not start a CLI. Running it must verify the actual provider and checkout; draft-PR creation is opt-in after a successful run. Ready, cancelled-close, and cleanup after merged/accepted state recheck the canonical task and remote PR state. Cancelled unmerged work is retained. There is no Agent Y merge command or background PR watcher. See Agent Y for the workflow and authority gates.

Toolchain and shared context

Read the toolchain map for the boundary between model/harness, records, app tools, verification, and delivery. The shared workspace guide covers cloud snapshot requirements, multi-chat writer ownership, and bot startup context. These are integration patterns over the current CLI/library, not new cloud services.

Agent Y execution and publication boundaries

The current full-system map adds Agent Y’s CLI/worktree/PR lane to the shared records. At prepare, the host-local sidecar pins the CLI’s canonical absolute executable path, SHA-256, device, inode, and size. Built-in commands are resolved through PATH only at preparation; run uses and rechecks the pinned target. A configured adapter must name the same regular, unlinked absolute command at prepare and run. This file check covers neither its interpreter/dependencies nor the same-user verification/launch race.

The Git-backed selected scope must effectively ignore and not track private sidecars/archives. The controller rechecks task status, owned branch/PR identity, and recorded base/head for lifecycle actions. pr-ready requires accepted done and a live base matching baseOid; pr-close requires cancelled and keeps unmerged work. pr-cleanup requires a verified remote merge and accepted task, archives a private Git bundle, and refuses to remove tracked, untracked, or ignored worktree material. These gates coordinate delivery; the project’s original checks, provider credentials, process access, and merge authorization remain with the actual host/project. Agent Y has no merge command.

The alpha.5 update adds direct child CLI routing and reviewed folder-copy launcher support, plus a local tooling guide and refreshed CLI/.SYSTEMXP poster. Exact new regression evidence belongs to the alpha.5 source; earlier results remain attributed to their reviewed release. A diagram does not establish a provider run or platform validation. See Versions and changelog and readiness evidence.

Tier 2 implementation and storage boundary

From 1.10.0-alpha.2, scripts/autobots.py reads the fixed ten-slot catalog, contained exact-case specialist Markdown and Agent Alpha charter; it never fetches their references or runs embedded content. autobot/agent-a are the same local interface. Explicit init adds missing seeds/agent.a memory and registers the reserved role; request/completion writes use the selected scope's coordination lock. Fingerprints include selected document bytes and purpose/ input/scope so unchanged completed calls reuse the same result. Changed detail retains historical evidence and labels its result stale instead of refreshing old proof or automatically looping.

scripts/data_tools.py creates missing exact DATA scaffolds and reports local synthetic Git-ignore probe results plus tracked index payloads. These bounded probes do not exhaustively verify actual untracked paths and can miss a custom per-file unignore exception; check named exports separately. It does not recursively read, hash, upload, move, or delete payload bytes. Git worktree exclusions are separate from credentials, backups, Drive/cloud transport, Git history, and same-account process access. Child panel/storage setup is explicit under the selected .SYSTEMXP; shared outer tools provide the runtime/defaults only. AUTOBOT manual · DATA manual · Exact record schemas.

Optional ROOT extension

ROOT planning and task delta adds bounded local request/structure and mirror records in compatible defaults. ROOT/PLAN.json holds requests/nodes; ROOT/MIRRORS.json holds bindings/observations/cards. Task status, owner, dependencies, and acceptance remain canonical in WORK/TASKS.json; milestones reference the existing Master Plan table. The optional extension initializes root and selected children independently, uses preview/explicit-apply mutations, and reports observed differences without external execution or background transport. Its generic tool-card contract uses actual advertised host tools and their schemas; no universal provider API or MCP server is implied. See the release's versioned ROOT guides for exact record fields and bounds.

Generic core and host-owned application packages

Keep vendor/domain SDKs, service contracts, fixture datasets, application UI, and application-service adapters in the host project, outside the generic SYSTEMX distribution. Global context and the System MAP may reference their real source, version, ownership, and evidence without copying the integration into the base. SYSTEMX can track their authorized tasks/checks; it does not ship an application integration runtime or claim external provider acceptance.