Skip to content

walkerOS v4.7.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 02 Oct 12:08
· 116 commits to main since this release
v4.7.0
d6685c9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Changes

With a transform and batch, the web API destination now adds each
transformed JSON object or array body to the batch as its value instead of as an
escaped JSON string. Transformed bodies that are not JSON objects or arrays are
still sent as strings.

AWS destinations now deliver in bundled flows without extra setup. Firehose
batches by default, ends records with a newline (newline: false turns it off),
sends mapped data and accepts config.credentials. Rejected records now count
as failed. SNS works with topicArn alone and no longer creates topics at
start. walkeros setup resolves $secret values; partial batch failures log
their causes.

New history setting for single-page apps: every route change (pushState,
replaceState with a new path, back and forward) sends a page view, and tagged
elements are picked up as the new route renders. Off by default. Page views
after the first now carry the previous page's URL as data.referrer.

An invalid entity name such as data-elb="shopping cart" or
data-elb="foo;bar" no longer throws and loses the event: it is ignored like a
missing entity, so other entities and events on the page are tracked as before.
Link ids (data-elblink) containing quotes or brackets no longer throw either.

Server builds served from the build cache now copy the root include folders;
web builds ignore include and log it. An unused package in bundle.packages
warns instead of failing the build. Cache entries are written atomically, so an
interrupt no longer leaves a broken entry, and walkeros cache clear removes
leftovers. Temp dirs are <tmp>/walkeros/<kind>/<id>.

Inside an array, a transformer's own next runs right after it, then the array
continues. many works in every chain field, each match becoming a copy with a
derived event.id. Unknown transformer ids warn and the chain continues, and a
throwing transformer counts in status.failed. walkChain and
extractTransformerNextMap are no longer exported.

walkeros bundle --manifest <url|path> builds every artifact a build manifest
lists and uploads each one with a result. Declare build values in
config.bundle.env; they must be literal strings and supply $env in web
bundles. Step packages, overrides and their dependencies must now come from the
npm registry: git, file and URL specs are rejected.

New ClickHouse server destination. Events are written in bulk as JSONEachRow
inserts into a table you create and own, with a bounded retry made safe by an
insert deduplication token. A column your table is missing fails the batch
loudly instead of dropping the field. The README carries the reference table
DDL.

A destination added at runtime now keeps the defaults its code ships, such as a
default batch, just like a destination configured at startup. Values you pass
in the destination config still win over those defaults, and a require set in
the code's defaults now gates the destination as well.

A destination's on() handler now receives the current state before any event
is sent to it, including with require, walker destination at runtime, and
state passed to walker run. Handlers are awaited up to config.timeout; a
destination whose handler does not respond receives no events until it does. The
exported callDestinationOn now returns a promise.

New collector.next: a transformer chain run once per event before the
destinations; a stop there drops it for all of them. Routes in every chain
field can drop an event with { stop: true }, optionally gated by match. A
route match reads { ingest, event } when the event reaches it. New
getRouteGraph; getNextSteps(spec, root) requires its root.

A bundle's default release is now a content id of config, packages and CLI
version, so repeat builds hit the build cache. Env and secret values are not
part of it, only their names. --release now beats a flow's
collector.release. Rebuilt local path packages and CLI upgrades invalidate
the build caches, and cache clear --builds also clears compiled code.

Simulating the CookieFirst source now records one walker consent per example,
as a browser does. The example trigger no longer replays cf_init after the
initial consent read. Runtime behaviour is unchanged.

Removed the MappingSchema export and the settings.command rule option; the
Mapping type is now an empty interface. The runtime never read command, so a
consent rule only renames the push to walker consent and the consent fields
stay in data, as the updated example shows.

Packages with several exports, such as the GCP and AWS sources and destinations,
now publish a settings schema per export. walkeros validate and MCP
package_get use the schema of the export a step imports. Pub/Sub, BigQuery,
SNS and SQS steps pinned to this release or later are checked instead of
skipped; older pins still report a skip.

The fingerprint settings schema now accepts every mapping value its type allows
for ip, userAgent, site and fields: paths, value configs and fallback
lists such as ["event.source.url", "ingest.origin"]. Configs using them no
longer get settings warnings from walkeros validate.

The canonical flow-complete.json example is rewritten as one tested ecommerce
setup with web, server and warehouse flows and a 15-chapter guide. The new
flowCompleteManifest export from @walkeros/cli/examples names every feature
by a stable id with a JSON Pointer into the file, so docs and tools can
reference it.

getFlowSettings throws a typed FlowCycleError for $flow and $var cycles,
with code (FLOW_CYCLE or VAR_CYCLE) and the chain of names; the message
is unchanged. validateFlowConfig reference warnings now carry the path of the
value that holds the reference, and an unresolvable root contract is an error
at contract.

walkeros flows create and walkeros flows update now read --content from a
file path or URL as well as an inline JSON string. Large flows no longer need to
fit on the command line: pass --content ./flow.json.

The express source passes non-JSON text/plain bodies, such as batched gtag
hits, to ingest.body instead of failing with 400. @walkeros/transformer-ga4
gives each decoded event its own stable id, caps one request at
settings.maxEvents (default 100), validates settings.mapping rules, and its
wiring hint sets paths: ["/g/collect"].

anonymizeIP now handles IPv6 (first 48 bits) and IPv4-mapped addresses;
getHashServer takes an optional HMAC key. The fingerprint transformer works
with just { salt }: it hashes the anonymized IP, reduced user agent and site,
rotated daily. Hashes change once on upgrade; rotate: "none" keeps them
stable. A missing salt warns.

A flow's own log lines now go through the CLI logger: its debug lines need
--verbose, and they go to stderr with --json. Version numbers are no longer
masked as tokens, so walkeros deploy shows the image tag and bundle --stats
shows package versions. Paths under the system temp directory print as
$TMPDIR/....

Manifest builds now survive short network blips: the manifest download, skeleton
downloads and uploads retry up to three times on network errors, timeouts, 5xx
and 429, while other statuses such as an expired link fail at once. Each retry
is logged, and errors name the cause, for example fetch failed (ECONNRESET) or
HTTP 503.

A mapping value runs exactly one producer, in the order loop, map, set,
key, fn, so a fn next to a key is ignored. A loop over a non-list
yields the value fallback. A mapping or policy entry that resolves to
nothing removes the field. The validate transformer never throws and pass mode
keeps such events.

frame_manage returns Tag Mode marks as a flat tags list plus the frame's
note. Tag id, parentId and threadRef come back literal when they have
the app's id shape, so an agent can rebuild the tag tree and pass a tag id to
hub_manage as markId. Other strings and non-identifier keys stay wrapped.

MCP tools validate their input against their input schema when called directly
through createToolHandlers, as they already were over the MCP transport.
Invalid input returns an error result naming each field instead of running on.

Customer information is normalized per Meta's rules before hashing, so
differently cased or padded emails hash the same. The earlier user_data hint
sent email and phone unhashed: rename them to em and ph. Unknown
settings.user_data keys fail validation, unknown mapped keys are dropped with
a warning, and a url without trailing slash works.

Decoded GA4 events and dataLayer events now carry source.url and
source.referrer, so conversion APIs such as Meta and Piwik PRO send them;
Piwik PRO now also sends GA4 events it skipped. Fingerprint hashes of these
events change once. GA4 timestamp is the receive time, and
globals.language/globals.screen moved to user.language/user.screenSize.

New Piwik PRO server destination. A web destination mapping that uses the 13
portable methods produces the same hits on the server. Browser-only commands
such as setUserId are skipped. Events go to the Piwik PRO Tracking API as one
bulk request per push or batch. The identified setting chooses identified or
anonymous tracking, always or based on consent.

walkeros run is removed. Build with walkeros bundle, then start the artifact
with runneros start dist/flow.mjs from the new @walkeros/runner package,
which the walkeros/flow image now runs. The CLI no longer depends on
express, cors or p-limit. New @walkeros/core/node entry with the
Node-only logger, redaction and temp-path helpers.

Simulate and push output, CLI and flow logs, and the MCP flow_simulate and
flow_push results mask the values of secrets a flow references, also inside
JSON strings, numbers and URLs. scrubSecrets takes known, the CLI logger
knownSecrets, and scrubJson replaces maskKnownNumbers. MCP flow_push and
flow_simulate read their config once; a config that cannot be read stops the
run.

New ip and userAgent settings on the Meta, Pinterest, Snapchat, TikTok,
Reddit, X (Twitter), Bing, Criteo and Google Data Manager server destinations.
Requests now send the client IP and user agent in each vendor's field by
default, read from ingest.ip/ingest.userAgent or user.ip/user.userAgent.
A mapped value still wins; set either setting to false to opt out.

Single-page apps that call walker run on route changes no longer start a new
session on every route change. Session detection now runs once per page load;
storage mode still counts each run. Repeated consent updates no longer restart
the session either. Also stops a configured domains list from growing on every
run.

session start now carries source.type: "session" with the page url and
referrer, like browser and dataLayer events, so destinations see the landing
URL with its campaign parameters. Before, it arrived as a collector event
without page context.

user.session is set only in storage mode, like user.device; the window
session id stays in the session start event's data.id. A consent-gated
session start now runs through the source's next and before chains and
mapping, and a custom cb gets { push, command }. A missing Navigation Timing
entry no longer throws.

Simulated click and impression examples of the browser source carry source and
globals again. A page scope from another realm, such as an iframe or a
simulated page, is now recognized. Simulate also exposes Node, Element,
HTMLElement, Document and ShadowRoot, so instanceof checks behave as in
a browser.

walkeros push --simulate prints each step's mapping and vendor calls
(simulations in --json) and reports init or push failures. New flags:
--ingest, --consent (MCP state.consent), --command for command examples,
--simulate collector.default and --mock collector.next.<id>. A destination
that sent nothing says why in skipped. toPrintable is exported.

Simulate records every vendor call through one recorder, observeEnv from
@walkeros/core, and runs offline: BigQuery, Data Manager and the Sheets, GCS
and S3 stores reach the network through env. Client and Env types of
BigQuery, Firehose, PostHog, SQS and Pub/Sub are structural. @walkeros/cli
exports resolveExportName and selectDevExamples and drops findExample and
compareOutput.

Simulate starts only the simulated step, and a simulated transformer continues
through its next. Source simulations list the commands a source issues as
elb calls; --page-url sets the web page. Express flows need no port.
Packages with several exports use per-export examples and mocks, and a
destination without a mock env is refused instead of calling the vendor.

state paths resolve against { event, ingest } and need an event. or
ingest. prefix; a get can write into ingest, and an unresolvable key
warns. Every step field now survives bundling: transformers with only state or
mapping, and code steps with cache or state, were silently dropped before.
buildCacheContext is renamed createMappingRoot.

state gains mapping to shape what a lookup returns or a write stores, merged
into the target. An undeclared state.store, or a file: true store, is now a
validation error in walkeros validate and deploy preflight; at runtime the
entry logs an error or is skipped with a warning. The s3 and gcs stores report
their file mode.

Files that passed walkeros validate may now fail: it checks every flow and
every step's package settings unless you narrow it, and resolves each flow as
the bundler does, so an unresolvable $flow is an error. --path checks the
entry in every flow. Settings findings warn for one minor. Named-export steps
are skipped; --offline skips fetching.

walkeros validate reports a missing or non-JSON input as exit 3, like
--path, --flow or --offline with a non-flow -t. Unknown keys on
destinations and stores warn (UNKNOWN_KEY), as does @walkeros/store-memory.
--path ignores uri and email formats. The Flow JSON Schema adds
config.observe and drops step-level validate; the Usercentrics schema lists
apiVersion and v3EventName.

walkeros validate prints a Scope: line and returns details.scope instead
of details.validatedFlow, details.skipped for checks that did not run and
details.deferred for values known only at runtime. Skips never read as passed.
Every finding has a stable code. --strict fails on warnings or skips (exit
2), and validate() and flow_validate with strict return valid: false;
flow_validate also accepts offline.

An unknown $var or $store now fails walkeros validate, as it fails the
bundle. References resolve per flow, and description text is never read as a
reference. A dangling contract extend, an unknown $contract on a validate
step, or a contract cycle is an error. Contracts bind only on validate steps.
Malformed references and unknown or shadowed routes warn.

walker.js is now one ready-made file: one script tag from static.walkeros.io
pushes every data-elb event to window.dataLayer for GTM. Route changes send page
views, so remove route-change walker run calls. Configuration is gone
(elbConfig, data-elbconfig, createWalkerjs, the dataLayer source, ES5 build); to
keep a configured setup, pin @walkeros/walker.js@4.6.1 instead of @latest.

Piwik PRO and Matomo web destinations: a page view rule with a goal still tracks
the page view, unmapped events are skipped with a warning, and url plus
appId/siteId are required only with loadScript. Piwik PRO adds custom
dimensions and an identified setting for cookieless tracking; Matomo
dimensions no longer leak into later hits.

Web builds now assign the collector to the global named in
config.settings.windowCollector, default walkerOS. Before, walkeros bundle
ignored the setting. It accepts a name or a $var/$env reference; the
resolved name must be a valid, non-reserved JavaScript identifier, or the build
and walkeros validate report an error.

Published Packages

Contributors

latest
Assets 2
Loading