Data Stores and Pipelines

Data stores, datasets and pipelines

The data side of Auton8: bring data in, describe it, move it through transformations, and query the result.

The chain is:

Data store  →  Dataset  →  Pipeline  →  (query / dashboard)
     ↑
Data connector (scheduled pull from outside)

Data stores

Data → Data Stores holds data that lives inside Auton8. Two kinds:

• Files — uploads behind a folder structure. Any file type; the store keeps the bytes and an index of what is where.
• SQL — tables in a managed Postgres schema (autonate_datastores), created and owned by the platform.

Files stores

Upload, browse by folder, download, copy, and preview. Downloads are served with a sanitised content type rather than echoing whatever the uploader declared — an uploaded HTML file will not execute in a viewer's browser.

Tabular files can be inspected and profiled — column names, inferred types, a sample of rows — which is what makes them usable as a dataset source.

SQL stores

Drop a CSV and Auton8 infers a Postgres column schema from a sample of rows, shows you what it inferred, and lets you correct it before creating the table. That conversion is explicit: you see the proposed types before anything is created.

Each SQL store gets its own database role, so a store cannot read another store's tables.

Data connectors

Data → Data Connectors are outbound integrations that pull data into Auton8 from external systems on a schedule.

A REST connector fetches an admin-supplied URL and lands the result. Because an administrator names the host, outbound requests are constrained by an allow-list — the stored credential is sent to whatever the URL names, so the allow-list is the boundary that matters.

Runs are recorded, so a connector that starts failing is visible rather than silently stale.

Datasets

Data → Datasets describe a shape of data you want to work with, over a source — a files-store file, a SQL table, or a query.

A dataset carries a schema preview: the columns and types Auton8 believes it has. That preview is what downstream pipelines and widgets bind against, so getting it right early saves rework.

Datasets are also queryable directly through the Dataset AQL entity.

Pipelines

Data → Pipelines move and reshape data. A pipeline is a graph of steps, edited on a canvas, each step configured by a schema-driven form rather than free-form JSON.

Built-in steps

The built-in transformer catalogue covers the ordinary reshaping work — selecting and renaming columns, casting types, filtering, aggregating (count, avg, agg, bins), and some analysis (correlation-matrix, anomaly-zscore, anomaly-iqr).

Each built-in publishes a config schema, which is why the node-config drawer renders real fields with validation instead of a text box.

Code transformers

Data → Code Transformers are user-authored steps: JavaScript or Python you write in the browser, in a CodeMirror editor.

They do not run in the web process. They are dispatched over NATS to the executor sidecar and run sandboxed — JavaScript in isolated-vm, Python via Pyodide, with time and memory limits. That isolation is the whole point, and it is treated as security-sensitive code; see External-Services.

Analyzers are the same mechanism pointed at inspection rather than transformation.

Running and history

Pipelines can be run on demand or scheduled. Run history keeps per-step log entries the orchestrator captured, colour-coded by level, so a failure tells you which step and why rather than just "failed".

Permissions

Data stores, connectors, datasets, queries and pipelines each have their own entity kind, with view/create/edit/delete/run actions and instance-level scoping. "Run this pipeline but do not edit it" and "read this one data store" are both expressible.

Worth knowing: a grant to run a code transformer is not a grant to author one. Authoring means writing code that the executor will run, which is a meaningfully larger privilege.