Skip to content

walkerOS v4.0.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 11 May 09:34
· 165 commits to main since this release
v4.0.1
a1ec50e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Changes

Fix events being silently dropped when posted via navigator.sendBeacon. The
browser forces Content-Type: text/plain;charset=UTF-8 for beacon requests even
when the payload is JSON, which previously caused the express middleware to skip
body parsing and the GCP Cloud Functions handler to treat the body as an opaque
string, both falling through to an empty-event push. Express now accepts
text/plain bodies through express.json(), and the Cloud Functions handler
attempts JSON.parse on string bodies before classifying the request.

BigQuery destination: migrate from legacy tabledata.insertAll to the BigQuery
Storage Write API (~2x cheaper at volume, 2 TiB/month free tier), add the
setup() lifecycle for one-shot dataset and table provisioning via
walkeros setup destination.bigquery, and implement pushBatch so the
collector's batch: <ms> mapping setting actually batches into a single
appendRows call.
Breaking changes:

  • The 15-column table schema is now using walkerOS event v4 schema.
    Run walkeros setup destination.bigquery to provision the dataset and table
    with day partitioning on timestamp and clustering on (name, entity, action).

@walkeros/cli: Server bundles now use @vercel/nft to trace dependencies and copy
only files actually used into dist/node_modules/. Pacote remains the install
layer (driven by flow.json's config.bundle.packages field; users do not run npm
install for step packages). The walkerOS.bundle.external annotation field on
package manifests is no longer recognized (deprecation warning if seen). The
flow..config.bundle.external sub-field on flow configs is also no longer
supported (warned and stripped during load). The
flow..config.bundle.traceInclude field is the escape hatch for cases nft
cannot statically trace. Server output is always a directory: dist/{flow.mjs,
package.json, node_modules/}. Default output filename changed from bundle.mjs to
flow.mjs. The runtime image expects /app/flow/flow.mjs. flow.json schema is
unchanged (still v4); only @walkeros/cli bumps. Migration: see
https://walkeros.io/docs/migrate/cli-4x.
@walkeros/server-destination-gcp: removed obsolete walkerOS.bundle.external
annotation from package manifest. nft handles externalization automatically. No
behavior change for consumers.

Bundler honors walkerOS.bundle.external declared in step-package package.json
files. Listed packages are externalized from the ESM bundle and the bundler
always installs them (plus their full transitive deps) into
<outputDir>/node_modules/ via pacote — no npm install shell-out, no manual
deploy step. When externals is empty, output remains a single bundle.mjs
(backward compatible). When non-empty, output is a self-contained directory:
bundle.mjs, package.json, package-lock.json, node_modules/.
The bundler reads npm config (registry, scope tokens) from .npmrc,
parallelizes manifest fetches with retry, atomically stages each package
extraction (no half-populated node_modules/ on failure), and reuses the
closure resolution from the existing collectAllSpecs BFS so peerDependencies
are honored.
Hard-errors when:

  • A package in the install closure declares a pre/install/postinstall script
    (pacote.extract does not run them).
  • A step package names an external in walkerOS.bundle.external but does not
    list it in dependencies or peerDependencies.
  • Two step packages declare the same external and the resolved version does not
    satisfy all consumers' constraints.
    Warns (not errors) when:
  • Bundle output contains unresolved __dirname / __filename references (with
    package attribution by hit count).
  • A step package's walkerOS.bundle.* block contains unknown keys (typo guard).
    New sibling export downloadPackagesWithResolution returns both the package
    paths and the full ResolutionResult. Existing downloadPackages keeps its
    return shape unchanged.

Surface destination init errors in logs at ERROR level. Previously, two layers
swallowed errors silently: the gcp destination's init catch only logged for
isNotFound errors and re-threw everything else without logging; the collector
wrapped destinationInit with tryCatchAsync (no onError), which silently
returned undefined on a thrown error and treated the destination as
not-initialized. Combined effect: a real init failure (e.g., the recent
streamType regression in BigQuery Storage Write API call) showed only
[gcp-bigquery] init in DEBUG logs and nothing else, regardless of log level.
Now: gcp's init catch logs every error at ERROR before re-throwing (with
consistent error: context key), AND the collector logs at ERROR via
logger.scope(destType).error('Destination init threw', { error }) if init
throws or rejects. Failures are never silent. Mocks updated to enforce the new
shapes; tests cover both sync-throw and async-rejection variants.

Fix walkeros bundle failing on Windows when stage 2 import paths contained
backslashes that JS parsed as escape sequences.

Declare @google-cloud/bigquery-storage as a bundle external via
walkerOS.bundle.external. Fixes __dirname is not defined in ES module scope
when bundling a flow that uses BigQuery Storage Write API. The bundler's closure
walker pulls in the transitive gRPC stack (@grpc/grpc-js,
@grpc/proto-loader, protobufjs, google-gax) automatically via
bigquery-storage's own dependencies and peerDependencies, so only the one
entry needs to be declared. Requires @walkeros/cli >= the version shipping the
bundler-externals feature.

Add Pub/Sub sub-destination to the GCP server package. Publishes walkerOS events
to a Pub/Sub topic with optional per-key ordering and dynamic attributes, plus
idempotent topic provisioning via walkeros setup destination.<id>. EU region
default for at-rest storage. Three auth modes: ADC, service account JSON,
pre-configured client.

Adopt the setup lifecycle for one-shot Kafka topic provisioning. Operators can
now run walkeros setup destination.<id> to create topics idempotently with
explicit numPartitions, replicationFactor, and configEntries. Drift on
partition count, replication factor, and config entries emits warnings without
auto-mutating. Optional Confluent Schema Registry binding registers a schema and
(optionally) sets the per-subject compatibility level.
No safe defaults. Kafka topic creation requires cluster-specific decisions.
The boolean form setup: true is rejected with an error listing the required
fields. Only the object form
(setup: { numPartitions, replicationFactor, ... }) is valid. This is the
canonical example of the "no safe default" pattern in the walkerOS
create-destination skill.
When the topic is missing at runtime, push() now logs an actionable error
pointing at walkeros setup destination.<id>.

Add an optional setup lifecycle to destinations, sources, and stores.
Each package may now implement setup?: SetupFn to provision external resources
(BigQuery datasets and tables, Pub/Sub topics and subscriptions, SQLite tables,
webhook registrations, etc.). Setup is triggered only by the new
walkeros setup <kind>.<name> CLI command, never automatically by the runtime,
push, or deploy. Idempotency, ordering, and error semantics are the package's
responsibility; the framework provides the type slot, the CLI invocation, and a
resolveSetup(value, defaults) helper.
LifecycleContext<C, E> is the new shared context type used by both setup and
destroy. DestroyContext remains as a deprecated type alias for one minor
cycle. The Types bundle on Destination, Source, and Store gains a
5th/6th/4th positional slot for setup options; existing aliases compile
unchanged because the slot defaults to unknown.
Config<T>.setup?: boolean | SetupOptions<T> is added across all three kinds
and validated by the corresponding Zod ConfigSchema plus the flow component
schemas in @walkeros/core/schemas/flow.ts.
CLI:

  • walkeros setup <kind>.<name> runs a single component's setup() function.
  • <kind> is source, destination, or store (transformers have no
    provisioning).
  • --config <path> (default ./flow.json), --flow <name> for multi-flow
    configs, plus standard --json / --verbose / --silent.
  • Exit 0 on success or skip; non-zero on failure. Skip narration covers three
    cases: no setup() on the package, config.setup === false, or
    config.setup unset.
  • When the package's setup() returns a non-undefined value, the CLI emits it
    as JSON on stdout for jq piping.

Source lifecycle redesign: factory + eager init + collector-gated on()
Source factories must now be side-effect-free. The collector calls
Instance.init() on each source eagerly after all factories register. require
no longer gates code execution. It gates on(type) delivery (events queue in
Instance.queueOn until the source is started, then replay).
collector.pending.sources has been removed; per-source state lives on
Source.Instance (queueOn) and Source.Config (init, require).
Migration: any source factory with side effects (queue draining, walker command
emission, listener attachment) should move those into the returned Instance's
optional init method. Tests asserting on collector.pending.sources should
read collector.sources[id] and inspect config.init / config.require
instead.
Fixes the elbLayer queue replay clobbering fresh consent/user state,
late-activated sources missing walker run, and inter-source require chains
racing when a non-required source's init fired a state-mutating walker command
before later require-gated sources had been registered.

Add the setup() lifecycle. Run walkeros setup destination.<name> to create
the events table with the canonical walkerOS Event v4 schema and apply pragmas
(journal_mode=WAL, synchronous=NORMAL, foreign_keys=ON,
temp_store=MEMORY). Setup is idempotent and detects drift via
PRAGMA table_info (logs WARN setup.drift, never auto-mutates).
The settings.sqlite.schema: 'auto' | 'manual' setting is deprecated and will
be removed in the next major. Migration: schema: 'auto' to setup: true,
schema: 'manual' to setup: false. The deprecated form still works and emits
a one-time WARN through the destination logger.

Add setup() lifecycle. Operators can now run walkeros setup store.<id> to
create the GCS bucket idempotently with sensible defaults (location EU, STANDARD
storage class, uniform bucket-level access, public access prevention enforced).
Detects drift on subsequent runs without mutating the bucket. Adds a hard-fail
with actionable message at runtime when the bucket does not exist.

Merge definitions into variables. Single concept, single syntax
$var.name(.deep.path)?. Whole-string references preserve native type; inline
interpolation requires scalars. Deep paths and recursive resolution with cycle
detection now supported. Flow.Definitions, Flow.Primitive, and the $def.
reference syntax are removed.

Published Packages

Assets 2
Loading