Repository navigation
These release notes are also available on the Arroyo Blog
This release encompasses almost a year of work and over 140 PRs merged, with a focus on scalability, reliability, and performance. This also marks the GA release of Basin Pipelines — Cloudflare's managed Arroyo service — and reflects the hard-won operational lessons of running the engine at scale across thousands of concurrent pipelines and many GB/s of traffic.
And we're already hard at work on the next release, 0.17, which we anticipate will be the last before 1.0.
Arroyo is a community project, and we're happy to welcome a record 9 new contributors in this release:
- @hrushikeshdeshpande made their first contribution in #969
- @oliy made their first contribution in #978
- @garvit-gupta made their first contribution in #985
- @benamartin43 made their first contribution in #981
- @wshirey made their first contribution in #990
- @QnJ1c2kNCg made their first contribution in #1000
- @cluster2600 made their first contribution in #1018
- @aayushn10 made their first contribution in #1074
- @46bit made their first contribution in #1102
Features
Worker leader mode
For Arroyo 0.16 we've added a new way to run the control plane, which we call worker leader mode, that further decouples the data plane (where pipelines run and process data) from the control plane. This improves reliability of highly multi-tenant deployments, and allows us to run massive numbers of concurrent pipelines (today at Cloudflare, this numbers in the thousands).
The new Arroyo control plane/data plane architecture, with worker leaders
In the current architecture (which we call job controller mode) the controller process — the control plane — is responsible for actively managing every pipeline; tracking failures, issuing checkpoints, and preventing split-brain. This last responsibility, in particular, means that control plane downtime causes data plane downtime. The new architecture moves most of these responsibilities into one of the pipeline workers (the worker leader) alongside an object store-based checkpoint linearization protocol, breaking this hard dependency on the controller.
This is a sufficiently deep topic that we previously described it in detail in this post.
In Arroyo 0.16, the existing job controller mode remains the default, and worker leader mode can be enabled by setting
job-controller = "worker"in your cluster config. In 0.17 it will become the default.
- Introduce worker job controller mode by @mwylde in #1035
- Implement new checkpoint protocol by @mwylde in #1051
- Support state machine adoption of existing clusters in leader mode by @mwylde in #1055
- Support checkpoint status API for leader mode by @mwylde in #1063
- Always advance generation when creating a new cluster in scheduling by @mwylde in #1065
- Set finish time on checkpoint details in leader mode when committing by @mwylde in #1069
- Keep leader alive after checkpoint stopping by @mwylde in #1072
- Add support for metrics in leader mode by @mwylde in #1075
- Implement checkpoint cleanup for leader mode by @mwylde in #1076
- Add integration tests for leader mode by @mwylde in #1077
- Add checkpoint_type to /checkpoints response by @mwylde in #1078
- Don't fail in leader mode if job is stopping by @mwylde in #1080
- Fix checkpoint cleaning failures in leader mode that could starve out checkpointing by @mwylde in #1086
- When scheduling a leader cluster times out, check for job failure by @mwylde in #1092
- Speed up recovery in leader mode by bounding attempted leader communication by @mwylde in #1101
- Support checkpoint GC in leader mode for stateful operators by @mwylde in #1112
- Add gRPC connect timeout on LeaderManager::connect by @cmackenzie1 in #1118
- Add support for starting/restarting without state in leader mode by @mwylde in #1122
- Fix race condition that could prevent stopping checkpoint finalization in leader mode by @mwylde in #1124
- Don't fail pipelines when checkpointing fails during restart by @mwylde in #1128
- Allow transitioning from safe restart to force restart in leader mode by @mwylde in #1137
- Show checkpoint type in web UI by @mwylde in #1137
- Add timeouts for leader manager calls by @mwylde in #1144
- Reset worker timeouts after startup succeeds by @46bit in #1183
- Fix checkpoint commit replay after downscaling by @QnJ1c2kNCg in #1195
FileSystem Sink v2
The FileSystem sink is one of the most used in Arroyo (and certainly the most complicated). Despite its (somewhat misleading) name, this is the sink used for writing files to object storage and performing transactional ingestion into data lake formats, including Delta and Apache Iceberg. The original implementation served us well for three years. But in 0.16 we're shipping FileSystem Sink v2, a complete rewrite that offers significant improvements in correctness and performance.
Superficially, writing files to object storage seems like it should be pretty simple. You just upload them, right? But we have some additional requirements that make this a bit harder:
- Events should be able to be ingested with low latency (sub-minute)
- Without writing too many files (which slows down query engines)
- Transactionally (each incoming event should appear exactly once in the output, even if we crash in the middle)
We previously wrote about how we accomplished these goals with the FileSystem sink. This codebase worked, but as we scaled it to thousands of file writes a second, we found several rare correctness bugs. (At scale, any bug in the system will appear for someone.) These sorts of bugs are particularly bad, because they can cause data loss — an absolute no-no for a storage service.
Much of the complexity, and its attendant bugs, came from the ad-hoc system within the operator to achieve parallel writes, necessary to get acceptable levels of throughput. This created effectively a separate timeline that interacted only via message passing with the main consistency system of the engine. Essentially, we'd made a distributed system within the operator and given ourselves all of the challenges of writing correct, transactional distributed systems. Furthermore, this made adding structured error handling (see below) to the sink very challenging.
Fortunately, in the years since the FileSystem sink was originally written, we've added some new tricks to Arroyo's operator framework, including the ability to run work in the background while still operating within Arroyo's consistency model, using Rust's async system. After some experiments, we concluded this was sufficient to implement the concurrency needed in the sink, and we decided to write FileSystem Sink v2.
The design is built around a state machine, tracking each file we are in the process of writing (or committing) to object storage. It looks like this:
State machines are a powerful tool to achieve correctness. In this case, it operates across the various phases of distributed execution and must be able to be saved and restored from checkpoints and recovered from interruption at any phase. The state machine formalism makes it tractable to analyze each of these possible states and transitions and ensure that we achieve correctness for all.
Beyond the performance and reliability improvements, the FileSystem sink also got a new feature: support for single-request uploads. Previously, all writes would be performed as multipart uploads (MPUs). (In fact, the core trick of the sink is to effectively stage in-progress files as parts of an MPU.) However, for small files committed entirely within a single transactional epoch, this is wasteful. We ended up doing three operations against the object store — starting the multipart, adding the single part, then completing it — when one would suffice. Now, the sink can be configured with multipart.minimum_multipart_size — files smaller than that size will be uploaded with a single PUT request.
V2 has been running in production at huge scale for the past six months and is recommended for all users. In 0.16 it's enabled by default. You can roll back to the previous version by setting 'sink.version' = 'v1' in the sink configuration.
- Filesystem Sink v2 by @mwylde in #989
- Add state migration system for operators by @mwylde in #996
- Improve single-file uploads by not sending contents over the commit RPCs by @mwylde in #997
- Address non-idempotent recovery for initiated multiparts by @mwylde in #1027
- Correctly handle errors while validating existing files in multipart recovery by @mwylde in #1125
- Default to filesystem v2 by @mwylde in #1135
- Remove max single part size limit in schema by @mwylde in #1137
- fix(filesystem): classify recovery auth errors as user failures by @QnJ1c2kNCg in #1168
- Preserve storage auth errors during multipart recovery by @QnJ1c2kNCg in #1200
- Tolerate missing multipart outputs for plain-file sinks by @QnJ1c2kNCg in #1218
SQL validation and diagnostics
Arroyo now supports detailed error diagnostics for invalid queries, in both the API and the Web UI. This means instead of getting an unhelpful "No such function COALESE" somewhere in a thousand lines of SQL, we will return the line and column of the error, and highlight it in the editor:
We are also now validating many more possible error conditions at query planning time, rather than run time, allowing you to find problems faster.
- Validate that ConnectionSchema fields are not duplicated by @mwylde in #1120
- Report planner error when inserting into sources or selecting from sinks by @mwylde in #1121
- Add detailed diagnostics for validate_sql API by @mwylde in #1123
- Validate that user-supplied pipeline IDs are valid path components by @mwylde in #1133
Structured errors
Up until now, Arroyo's approach to failure handling in the data plane was pretty simple: let it crash. If anything went wrong in an operator, the operator would issue a panic, which would be caught within the operator framework code, and transmitted back to the controller to trigger job recovery. This may not be uh... Rust best practice, but actually it works quite well. By carefully structuring the async task hierarchy, panics are a convenient mechanism to flow the failure through the task graph, organizing shutdown. It's also very robust, as any unexpected error will also trigger this same mechanism and lead to a clean shutdown.
But there are a few downsides:
- The chain of panics creates a lot of noise in logs, and it can be hard to find the root cause in all of the downstream panics
- We end up relying heavily on specific panic unwinding behavior
- There is no automated way to distinguish different kinds of errors, particularly user-caused errors vs system-caused errors, or within the space of user-caused errors, between data-provoked errors (like a failed CAST) and infrastructural errors, or unexpected panics indicating bugs
The last of which was the main motivation for this work. At Cloudflare we run Arroyo as a managed service, which means that we are responsible for pipelines always operating (and get paged if they ever break). But it turns out there are many ways for a user to break their own pipeline. For example, the user might rotate the key they've configured for writing to their R2 bucket. This is a user-caused error that the user must fix, as we don't have permission to mint a new credential.
On call engineers tend to get cranky when they get woken up for a problem they did not cause and have no way to fix, so we needed to be able to distinguish between these cases.
Now, all error handling within the engine and operator framework relies on structured error handling (in Rust terms, Result). These errors are returned by operators when they cannot continue for some reason; they are handled by the engine machinery, and returned back to the controller to drive reporting and recovery.
Structured errors are visible within the Web UI, returned via the jobs API, and are logged.
For example, if the key for the R2 bucket from the example above is rotated, the FileSystem sink now fails with an error like this one, from GET /v1/pipelines/{pipeline_id}/jobs/{job_id}/errors:
| Error | Domain |
|---|---|
| error in connector: The operation lacked the necessary privileges to complete for path events/2026/11/16/00002-0007.parquet: Error performing PUT … in 84.2ms - Server returned non-2xx status code: 403 Forbidden: … | user |
Because the error is in the user domain and isn't retryable, the job fails immediately instead of restarting in a loop, and it can be routed to the user rather than to an on-call engineer.
- Structured error handling (part 1) by @mwylde in #970
- Structured error handling (part 2) by @mwylde in #980
- Improvements to structured error handling and propagation by @mwylde in #1136
JSON deserialization errors
When ingesting data into a structured Arroyo table (i.e., one with a schema) Arroyo must validate and parse the incoming data into that structure. For example, if you are reading JSON into a table that has a user_id BIGINT NOT NULL field, the records must have a user_id field that is parseable as an integer. If the data is not valid, there are two options: we can drop the event (prioritizing availability), or we can fail the pipeline so the logic or schemas can be repaired (prioritizing durability).
However, we have not made it particularly easy to see why a record failed to deserialize. In this release, we are improving that by introducing detailed JSON deserialization errors.
JSON sources now report exactly why a record couldn't be deserialized: which field failed, and whether it was missing, null when the schema requires a value, or the wrong type. With bad_data = 'fail' (the default), the pipeline fails with that error. With bad_data = 'drop', bad records are dropped and counted in the arroyo_worker_deserialization_errors metric, and each failure is emitted as a structured event.
Because these messages end up in logs and the database, they never include the raw value from the record, only its JSON type and length. For example, for a table with the columns user_id BIGINT NOT NULL and age INT NOT NULL:
| Record | Error |
|---|---|
{"age": 31} |
field 'user_id': required field is missing (expected Int64) |
{"user_id": 42, "age": null} |
field 'age': null value for non-nullable field (expected Int32) |
{"user_id": "john.doe", "age": 31} |
field 'user_id': expected Int64, got string (length: 8) |
Deployments with high volumes of bad data can also stop storing dropped-record errors in the database by setting pipeline.store-deserialization-errors = false.
- Capture and report JSON deserialization errors by @garvit-gupta in #985
- Fix deserialization error counter increment by @garvit-gupta in #999
- Redact raw values from deserialization error logging by @garvit-gupta in #1010
- Add config to disable persisting deserialization errors to DB by @garvit-gupta in #1011
Improvements
Connectors
- Add support for Gzip compression for JSON on the Filesystem connector, v1 and v2 by @QnJ1c2kNCg in #1007
- Local JSON compression and optimization for empty encoders by @QnJ1c2kNCg in #1021
- Output full Iceberg catalog error by @mwylde in #1024
- Improve Iceberg error mappings by @mwylde in #1029
- Improve retry mapping logic for Iceberg errors by @mwylde in #1031
- Make watermark-based file rollover optional, and default to off by @mwylde in #1032
- Populate snapshot summary stats in Iceberg commits by @garvit-gupta in #1038
- Add timeouts to Iceberg commit to prevent unbounded checkpoint stalls by @mwylde in #1056
- Introduce a Postgres TLS config by @QnJ1c2kNCg in #1058
- Preserve full catalog error messages when they contain quotes by @aayushn10 in #1074
- Classify Iceberg rate limits as external failures by @QnJ1c2kNCg in #1141
- Run the Iceberg commit in the background by @mwylde in #1170
Engine and control plane
- Add exponential backoff to state retries by @mwylde in #971
- Retry getting state from remote store by @oliy in #988
- Retry unhandled panics in data flow by @mwylde in #991
- Add ignore_state option to pipeline restart by @garvit-gupta in #992
- Limit number of recovery attempts for task failures by @mwylde in #993
- Add retries on failures to fetch/write checkpoints by @QnJ1c2kNCg in #1000
- Preserve epoch continuity when restarting without state by @garvit-gupta in #1003
- Refactor out common RPC components by @mwylde in #1033
- Add configurable Postgres schema for multi-tenant deployments by @QnJ1c2kNCg in #1042
- Retry transient 409s instead of treating them as user errors by @garvit-gupta in #1043
- Validate target worker_id on WorkerGrpc RPCs via interceptor by @garvit-gupta in #1045
- Increase retry counts for scheduler errors to tolerate transient infrastructure outages by @garvit-gupta in #1048
- Only install pgcrypto if needed by @QnJ1c2kNCg in #1052
- Add optional state_url and tags to pipelines by @QnJ1c2kNCg in #1053
- Redact
Sensitive<T>Debug output to avoid leaking secrets by @QnJ1c2kNCg in #1054 - Introduce env_vars column to the job_configs table by @QnJ1c2kNCg in #1060
- Introduce scheduler_config column to the job_configs table by @QnJ1c2kNCg in #1064
- Expose pipeline_id in job APIs by @QnJ1c2kNCg in #1081
- Expose state_url and tags in pipeline APIs by @QnJ1c2kNCg in #1083
- Move env_vars from the Pipeline API to the Job API by @QnJ1c2kNCg in #1084
- Add organization_id to StartPipelineReq by @QnJ1c2kNCg in #1089
- Support updating env_vars and scheduler_config on pipelines by @cmackenzie1 in #1090
- Return to state-machine recovery when retrying individual states fails by @mwylde in #1091
- Add admin routes for pipelines/jobs scoped by org id by @mwylde in #1100
- Add bytes-based flushing to source collectors by @46bit in #1102
- Allow API CORS behavior to be configured by @mwylde in #1107
- Support restarting without state in leader mode by @mwylde in #1111
- Move to cornucopia queries for restart without state by @mwylde in #1126
- Add bounded retry policies for storage conflicts by @QnJ1c2kNCg in #1188
- Add retries to Arroyo storage reads and deletes by @46bit in #1199
Observability
- Use json-subscriber for faster logging and static logging fields by @oliy in #978
- Add output-format and table-format labels to filesystem sink trace events by @garvit-gupta in #1019
- Attribute worker control-loop RPC failures and log message sizes by @QnJ1c2kNCg in #1098
- Add arroyo-service and pipeline_id to JSON logs by @QnJ1c2kNCg in #1105
- Don't include worker_id in the service name in worker logs by @mwylde in #1110
- Add pipeline_id tag to all pipeline-specific controller logs by @mwylde in #1113
- Export job state metrics for Prometheus by @QnJ1c2kNCg in #1114
- Normalize all job_id logs by @mwylde in #1116
- Log full error chains for analytics and storage errors by @46bit in #1132
Project infrastructure
- Updates to README by @mwylde in #967
- Bump version to 0.16.0-dev to start next release cycle by @mwylde in #968
- Add Semgrep workflow by @hrushikeshdeshpande in #969
- Update to Rust edition 2024 and Rust 1.92 by @mwylde in #987
- Fix tutorial link in README.md by @wshirey in #990
- Clean up Cargo manifests and address dependency alerts by @QnJ1c2kNCg in #1001
- Remove unused variable by @QnJ1c2kNCg in #1006
- Fix various nits and typos by @QnJ1c2kNCg in #1009
- Update Rust toolchain to 1.93 in CI by @QnJ1c2kNCg in #1012
- Partially revert cleanup from #1009 by @QnJ1c2kNCg in #1016
- Fix broken links in README and CONTRIBUTING by @cluster2600 in #1018
- Update Rollup to 4.59.0 by @mwylde in #1028
- Update Vite to 6.4.2 by @mwylde in #1041
- Clean up Cargo.toml by @QnJ1c2kNCg in #1059
- Update CI and Clippy lints to Rust 1.95 by @mwylde in #1070
- Upgrade web UI dependencies to resolve esbuild, js-yaml, and Vite advisories by @QnJ1c2kNCg in #1085
- Update rdkafka to 0.39 to fix common macOS CMake builds by @cmackenzie1 in #1097
- Update Kafka URL for CI by @QnJ1c2kNCg in #1106
- Upgrade js-yaml to resolve Semgrep finding by @mwylde in #1127
- Upgrade postgres-protocol to 0.6.12 to satisfy Semgrep by @QnJ1c2kNCg in #1139
- Upgrade js-yaml to 4.3.2 (from 4.3.1) by @QnJ1c2kNCg in #1152
- Fix udf macro by tagging no_mangle unsafe by @QnJ1c2kNCg in #1157
- Mitigate eMudhra CA compromise in TLS dependencies and images by @QnJ1c2kNCg in #1197
- Upgrade to Rust 1.99.1 by @cmackenzie1 in #1206
- Add back support for building on Rust 1.98 by @mwylde in #1212
- Release udf crates version 0.3 by @mwylde in #1225
Fixes
- Fix local file partitioning file-not-found issue by @benamartin43 in #981
- Fix bytes calculation in checkpoint_state by @QnJ1c2kNCg in #1004
- Propagate error during hash computation by @QnJ1c2kNCg in #1005
- Fix race condition in bounded queue by @mwylde in #1017
- Hold input queues until ready for shutdown by @mwylde in #1020
- Return 404 instead of 400 in PATCH pipeline by @QnJ1c2kNCg in #1057
- Propagate API auth errors instead of panicking by @QnJ1c2kNCg in #1066
- Stop using serde::__private::ser::FlatMapSerializer by @QnJ1c2kNCg in #1071
- Prevent massive error messages from exceeding the gRPC limit by @QnJ1c2kNCg in #1108
- Release job map lock before queue sends by @cmackenzie1 in #1117
- Fix Iceberg truncate transform incorrectly applying bucket by @mwylde in #1134
- Preserve edge schemas when chaining operators by @QnJ1c2kNCg in #1138
- Upgrade DataFusion fork to include fix to substr by @mwylde in #1140
- Prevent single-record batches from
ContextBufferby @46bit in #1143 - fix(worker): drain and restore async UDF calls correctly by @QnJ1c2kNCg in #1150
- Validate filesystem sink data sizes and cap parquet buffer preallocation by @cmackenzie1 in #1184
- Make gRPC receive message limits configurable by @QnJ1c2kNCg in #1196
- Refresh job start time for every execution by @QnJ1c2kNCg in #1198
- Fix an off-by-one error that caused shuffle-by-partition to send to wrong subtasks by @mwylde in #1209
- Handle deleted jobs during controller reconciliation by @QnJ1c2kNCg in #1210
- Increase default gRPC receive limit to 32 MiB by @QnJ1c2kNCg in #1211
