Skip to content

Architectural Quantum Components to be more interoperable & in alignment with Control-Feedback Loop - #196

Merged
JEJodesty merged 63 commits into
mainfrom
dev
Jul 24, 2026
Merged

Architectural Quantum Components to be more interoperable & in alignment with Control-Feedback Loop#196
JEJodesty merged 63 commits into
mainfrom
dev

Conversation

@JEJodesty

@JEJodesty JEJodesty commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Architectural Quantum Components made more interoperable in alignment with the Control-Feedback Loop.

Structure & Plant

  • Split Terraform into infrastructure/ (IPFS transport, MinIO) and plant/ (kind/Helm/Ray), with Plant compose/snapshot CIDs on the BOM
  • Flatten modules into sibling packages; add ContentStore, TransportContext, RayComputePort / RayPlantPort
  • Persist MinIO for Structure lifetime; JobHandle-only scratch; Ray landing under plant_cid

Function & execution

  • Split Process / InfraFunction packages; Process as Composed Function, InfraFunction as Actuator
  • Dispatch Process as a Ray Job on the Plant; Integration staging and Invoice stage CIDs
  • Ports: TransportPort, ComputePort, PlantPort; hybrid Order CIDs, named binds, linkProcess / linkStructure / linkOrder

Network, Node, Service

  • Kubo RPC IPFS client; MeshClient → ContentMesh (order/packaging/named_binds/bootstrap/node_http)
  • Node lifecycle CLI (start|stop|status|ensure) + Makefile targets; configurable host/port
  • Slim Service onto ContentMesh + Factory; in-process execute-order CLI and Make target

Factory / Executor modules

  • Extract Executor, Processor, InfraFunction, Plant, InfraStructure, TF helpers
  • Factory accept / assemble / produce manufacturing cell with stable re-exports

Docs & diagrams

  • BOM, DESIGN, PLANTs, COMPONENTS, STORAGE, MinIO, IPFS, INTEROP, DASHBOARDS, ControlFeedbackLoop
  • README led by CAT Node, AQ/BOM, and Control-Feedback Loop; NodeProductFlow → ControlFeedbackLoop
  • Terminology: Process [Composed Function], InfraFunction [Actuator]; REPLaC = Function Workflow UI
  • code2flow / pyreverse diagram assets

Tests

  • Replace verification_test with provenance suite (test_provenance.py)
  • Unit coverage for ports, ContentStore, lineage helpers, ContentMesh, Node CLI, Factory.accept
  • Document unit vs integration tiers in TEST.md

JEJodesty added 30 commits July 23, 2026 21:23
refactor(terraform): move IPFS/Docker transport resources into
 module.infrastructure and the kind cluster/Helm releases into
 module.plant, wiring plant depends_on infrastructure;
 add moved blocks so existing state migrates without
 destroy/recreate;
 relocate ipfs_connect_peers.sh and ipfs_transport_compose.yaml
 under modules/infrastructure/, and rewrite the peering script to
 discover containers/peers dynamically instead of hardcoding them;
 expose plant_kind_cluster_name, plant_kubeconfig_context, and
 plant_ray_release_name as root outputs;
 update IPFS.md's main.tf path references
feat(executor,factory,service,network): add InfraStructure.compose()
 producing a Plant that reads kind_cluster_name/kubeconfig_context/
 ray_release_name from Terraform outputs, mirroring
 InfraFunction.compose();
 track Plant.rebuilt across deploy()/redeploy() and return a
 snapshot from Structure.reconcile();
 record that snapshot as enhanced_bom['plant'] and
 log['plant_rebuilt'] in Executor.execute(), and persist it via a
 new plant_cid in Service's bom;
 read plant_cid back in MeshClient.flatten_bom();
 pass the Structure directory (not main.tf) as structure_filepath
 via cidDir, since Structure is now multiple module files, updating
 cats_demo.py and verification_test.py to match;
 fix redeploy() to initialize() before destroy() (providers must be
 installed to load schemas for either), and update the module-
 qualified resource addresses (module.plant.*, module.infrastructure.*)
 used by stale-state cleanup;
 fix Terraform provider binaries losing their executable bit under
 this workspace by re-chmod'ing them after init and before every
 apply/destroy/plan
feat(function,process,infrafunction,structure,plant,infrastructure):
 split order.function_cid into process_cid (the Functional Data
 Processors) and infrafunction_cid (Process's orchestrator), and
 order.structure_cid into plant_cid and infrastructure_cid, each
 CID-ed and fetched independently (cats/network/__init__.py);
 add infrafunction_subproc dispatching integrated_subproc as a real
 Ray Job via the Job Submission API against the Plant's Ray
 dashboard, with every write task landing directly in a shared
 MinIO bucket instead of gathering to one node's disk
 (data/input/process.py);
 wire Processor.Integration() to call it with Service's
 RAY_DASHBOARD_ADDRESS/MINIO_* attributes, populated by
 Executor.execute() from Plant.ray_dashboard_address() and
 InfraStructure's new minio_endpoint_host/pod/bucket/access_key/
 secret_key/snapshot() accessors reading matching new Terraform
 outputs;
 record enhanced_bom['structure']/['infrastructure'] and rename the
 BOM's plant_cid to plant_snapshot_cid alongside a new
 infrastructure_snapshot_cid;
 move invoice_cid production (and its order_cid backfill) from
 Service.execute() into Executor.execute(), so the Executor alone
 produces the Invoice CID;
 provision the Ray dashboard on a static kind NodePort, raise the
 chart's default head.resources.limits/requests.memory from 2G to
 3G to stop OOM-kills under InfraFunction's Ray Job dispatch, and
 bump ray_image_tag to match this repo's pinned Python for
 cloudpickle compatibility;
 provision module.infrastructure's MinIO via docker-compose,
 exposing both it and the dashboard as root Terraform outputs;
 pass infrafunction_subproc through cats_demo.py/verification_test.py
feat(node): read CAT_NODE_HOST/CAT_NODE_PORT env vars for the Flask
 bind address, defaulting to 127.0.0.1:5000;
 add _free_stale_port() to kill a leftover node.py still bound to
 that port on startup, leaving unrelated listeners (e.g. macOS's
 AirPlay Receiver) alone, and skip the check under Werkzeug's
 debug-mode reloader worker so it doesn't kill its own monitor
docs: add "How the Architectural Quantum is realized as
 content-addressed CIDs" to DESIGN.md, and BOM.md detailing exactly
 what order.function_cid/structure_cid and bom.plant_snapshot_cid/
 infrastructure_snapshot_cid resolve to, with live example payloads;
 add DASHBOARDS.md (Ray Dashboard, MinIO Console, IPFS WebUI) linked
 from a new README.md Dashboards section;
 cross-link LineageOfProvenance.md to BOM.md's CID breakdown;
 update README.md's Architectural Quantum description to match the
 Process/InfraFunction and Plant/InfraStructure composition now
 implemented, noting the Executor itself produces the Invoice CID
refactor(function): split data/input/process.py into
 data/input/function/process.py and
 data/input/function/infrafunction.py;
 [Process] keep ingress/egress/integration_cache and the pure
 process_0/process_1 transfer functions, now returning the Ray
 Dataset directly with no MinIO/output-path knowledge;
 [InfraFunction] move infrafunction_subproc, its Ray Job dispatch,
 and the entrypoint script delivering Process's Dataset into
 MinIO into their own actuator module;
 [Structure] update infrastructure module's main.tf comment for
 the new infrafunction.py path;
 [Node] update sys.path comment for the split module paths;
 update cats_demo.py, tests/verification_test.py, docs/BOM.md,
 and .gitkeep for the new data/input/function/ layout
chore(gitignore): drop data/input/main.tf, root-level
 bom/invoice/order ephemeral files, data/online/, and
 structure/outputs, data, ipfs_cluster, providers entries no
 longer produced by the current Terraform layout;
 ignore data/input/function/__pycache__ and local_artifacts/
refactor(executor,factory,network,service): [Structure] take
 structure_cid directly on init instead of reconcile() receiving it
 per call;
 [InfraStructure] store structure_cid on init so reconcile() reads
 it off self instead of a passed-in argument;
 [Function] take function_cid directly on init instead of letting
 InfraFunction re-derive it from enhanced_bom;
 [InfraFunction] accept function_cid directly instead of reading
 self.enhanced_bom['order']['function_cid'];
 [Factory] build Structure/Function from order_request['order']'s
 real structure_cid/function_cid and inject both into Executor,
 instead of Executor constructing its own;
 [Executor] accept structure/function as dependencies; backfill
 invoice order_cid from order_request['order_cid'] directly instead
 of re-hashing order_request['order'];
 [Order] thread an explicit order_cid through initBOMjson/
 initBOMcar so the bootstrap Invoice and the final backfilled
 Invoice reference the exact same Order CID, minting a placeholder
 only when no real order_cid exists yet;
 [Service] pass order_request['order_cid'] into initBOMcar and
 construct Factory with the full order_request; drop now-unused
 `copy` import in network/__init__.py
docs(readme,design): [Function] describe mutating Function/
 InfraFunction as producing a new Order whose function_cid/
 structure_cid reflect the update, with a footnote pointing at
 NodeProductFlow.md's not-yet-built registry note;
 [Structure] replace the stale "web app codebase is Content
 Addressed" Plant claim with a description of the REPL UI (see
 cats_demo.py) that submits an Order for the Factory to process;
 [InfraStructure] describe mutating provisioning as producing a new
 Order with an updated structure_cid;
 [Quantum] reword the closing Factory/Executor paragraph in both
 README.md and DESIGN.md to state the Factory composes Function and
 constructs Structure from the Order's CIDs before instantiating
 the Executor with them as dependencies
docs(architecture): [Quantum] add COMPONENTS.md defining the
 Factory, Architectural Quantum, and ephemeral Executor;
 [Function][Structure] add NodeProductFlow.md walking through how
 an Order's Function/Structure CIDs flow through Factory and
 Executor to produce a BOM;
 [Structure] add PLANTs.md mapping the Factory, Architectural
 Quantum, Function, and Structure onto Wikipedia's "Plant"
 disambiguation categories
docs(lineage): [Order] state lineage chains by carrying forward
 each CAT's output data_cid, not a stored bom_cid/order_cid
 backward-pointer; note bom_cid is never written into any
 IPFS-addressed content CATs produce, so the upstream link depends
 on the caller already holding the prior response
docs(examples): point the Demos link at cats_demo.py instead of
 the removed cats_demo.ipynb; update the Test command to
 `uv run pytest -s tests/verification_test.py`
chore(gitignore): add *.zip so downloaded archives (e.g. the
 stray terraform_*.zip) aren't tracked; ignore docs/TODO.md as a
 local-only scratch doc
docs(NodeProductFlow): add a mermaid activity diagram mapping
 Order, Factory, Executor, and BOM to each Architectural Quantum
 component — [Function] InfraFunction (FaaS) and Process (FaaS),
 [Structure] InfraStructure (IaaS) and Plant (SaaS);
 add rendered images/NodeProductFlow.png; reframe the doc as a
 control-feedback loop with the existing step-by-step narrative
 preserved below the diagram
docs(NodeProductFlow): replace flat activity nodes with Factory/
 Executor subgraphs, rename edges, and add dagre layout links;
 add flip-edge-arrow post-render for discoveredVia feedback;
 renumber and expand the control-feedback loop specification;
 regenerate NodeProductFlow.png
docs(NodeProductFlow): remove duplicated mermaid source from the
 spec page; keep the rendered PNG as the diagram reference
feat(process,executor,factory): stage Ray input via integration_cache
 host path; raise on transport failures; record invoice ingress/
 integration_data_cid and rename BOM log CID keys; assert stage
 fields in verification_test
fix(structure): drop orphan Plant dependents without kind cluster;
 recreate missing MinIO compose from drifted shell_script state
docs(readme,bom,design,node): reserve tHOF for integrated_subproc;
 document interim Invoice stage CIDs until Seed; align Process/
 InfraFunction comments and PLANTs wording
docs(images): revert NodeProductFlow.png to the prior diagram
docs(images): refresh NodeProductFlow.png from the experimental
 Mermaid export
fix(ipfs): patch ipfshttpclient HTTPAdapter for urllib3 2.x
 PoolManager (drop obsolete strict=True)
feat(infrastructure): bind structure_minio_data volume;
 tear down with docker-compose down -v on destroy
feat(executor,service): return job_prefix from
 infrafunction_subproc; set MINIO_RESULT_URI and
 log.minio_result_uri for Structure-lifetime scratch
docs(storage,minio,bom): add STORAGE/MinIO guides;
 clarify integration_data_cid and minio_result_uri
feat(network): replace ipfshttpclient with requests-based
 KuboRpcClient behind CatsIPFSClient;
chore(deps): depend on requests instead of ipfshttpclient;
test(network): cover Kubo RPC add/id/error paths
feat(network,node): probe Kubo HTTP API for readiness; heal
 stale repo.lock; shut down owned daemon on node exit;
 warn on MeshClient import if daemon fails;
align Terraform host_ipfs_daemon probe with HTTP API;
docs(IPFS): document API probe, heal, and owned shutdown;
test(network): cover daemon probe, heal, and ownership
JEJodesty and others added 15 commits July 23, 2026 21:23
test: add concise docstrings on unit-test cases for ports,
 ContentStore, lineage helpers, MeshClient, and Node CLI
docs(TEST): document provenance integration vs unit coverage and
 per-file pytest commands;
docs(CONTRIBUTING,ENV,EXAMPLES): point at test_provenance.py
refactor(network): rename MeshClient to ContentMesh; split order,
 packaging, named_binds, bootstrap, and node_http;
feat(network): add feedback/identity seams and CoDTransport stub;
refactor(network): move legacy CoD under network/legacy;
 update package entrypoints to ContentMesh
refactor(executor): extract Executor, Processor, InfraFunction,
 Plant, InfraStructure, and TF helpers into modules;
refactor(factory): Factory.accept/assemble/produce manufacturing
 cell with stable package re-exports
refactor(service): wire contentMesh; delegate intake to
 Factory.accept; nest as-executed structure under Invoice BOM;
refactor(service): drop k8s helper, fake_node, and unused AWS/S3
 fields;
chore(deps): remove boto3 and kubernetes; add code2flow and pylint
 to the dev extra
feat(cli): add cats.execute_order for Order execution without Flask
test(network,factory): retarget MeshClient suites to ContentMesh;
 cover Factory.accept, feedback/identity seams, and
 InfraStructure.snapshot
docs(BOM): document HTTP BOM envelope and structure_as_executed;
docs(readme,DEPS,STORAGE,IPFS,MinIO,...): rename MeshClient to
 ContentMesh and align path/wording updates
feat(makefile): add execute-order, deps-graphviz, and diagrams;
docs(images): commit code2flow/pyreverse PNG assets and generator
refactor(function,executor,network): call Process a Composed
 Function and InfraFunction an Actuator; reserve REPLaC for the
 Function Workflow UI in comments
docs(readme,design,NodeProductFlow,PLANTs,BOM,DEMO,INTEROP):
 rename Process [REPL(aC)] to Process [Composed Function] and
 InfraFunction [FaaS] to InfraFunction [Actuator]; clarify REPLaC UI;
docs(images): refresh NodeProductFlow.png
docs(ControlFeedbackLoop): specify
 the loop as the Node's Minimal FOM / AQ runtime, rename NodeProductFlow.md
docs(readme): lead with CAT Node, AQ/BOM, and a summarized
 Control-Feedback Loop; link ControlFeedbackLoop.md; reorder
 Dashboards before diagramming
Architectural Quantum Components to be more interoperable & in alignment with Control-Feedback Loop
@JEJodesty JEJodesty self-assigned this Jul 24, 2026
@JEJodesty JEJodesty added documentation Improvements or additions to documentation enhancement New feature or request story Agile story testing Testing functionality System functionality labels Jul 24, 2026
@JEJodesty JEJodesty added this to the 2. CATs’ Node milestone Jul 24, 2026
@JEJodesty JEJodesty changed the title Dev Architectural Quantum Components to be more interoperable & in alignment with Control-Feedback Loop Jul 24, 2026
@JEJodesty
JEJodesty merged commit a69adc6 into main Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request functionality System functionality story Agile story testing Testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plant (SaaS) InfraStructure (IaaS) CAT Node Function (FaaS) Factory

1 participant