v0.17.0
v0.17.0
Changed
-
Breaking change Recovery system re-worked. Kafka-based recovery
removed. SQLite recovery file format changed; existing recovery DB
files can not be used. See the module docstring for
bytewax.recovery
for how to use the new recovery system. -
Dataflow execution supports rescaling over resumes. You can now
change the number of workers and still get proper execution and
recovery. -
epoch-interval
has been renamed tosnapshot-interval
-
The
list-parts
method ofPartitionedInput
has been changed to
return aList[str]
and should only reflect the available
inputs that a given worker has access to. You no longer need
to return the complete set of partitions for all workers. -
The
next
method ofStatefulSource
andStatelessSource
has
been changed tonext_batch
and should return aList
of elements,
or the empty list if there are no elements to return.
Added
-
Added new cli parameter
backup-interval
, to configure the length of
time to wait before "garbage collecting" older recovery snapshots. -
Added
next_awake
to input classes, which can be used to schedule
when the next call tonext_batch
should occur. Usenext_awake
instead oftime.sleep
. -
Added
bytewax.inputs.batcher_async
to bridge async Python libraries
in Bytewax input sources. -
Added support for linux/aarch64 and linux/armv7 platforms.
Removed
KafkaRecoveryConfig
has been removed as a recovery store.
What's Changed
- Spread operator by @Psykopear in #246
- Start updating CI by @whoahbot in #253
- Correct number of available inputs by @RCdeWit in #255
- Build exception messages only on error by @davidselassie in #254
- Fix links in docs leading to 404 error by @konradsienkowski in #256
- Mark KafkaRecoveryConfig as deprecated by @davidselassie in #258
- Fixes
ymd
deprecation warnings. by @whoahbot in #262 - Batch input take 2 by @Psykopear in #261
- Properly handle when resume_state is falsy by @davidselassie in #267
- AsyncBatcher by @davidselassie in #269
- Next awake by @Psykopear in #266
- Run clippy on pre-commit by @davidselassie in #272
- For some reason test_execution has Windows line endings by @davidselassie in #273
- Various text pre-commit hooks by @davidselassie in #274
- SQLite-only recovery with dynamic partitions to support rescaling by @davidselassie in #271
- Bumps PyO3 to 0.19.2 by @davidselassie in #276
- Calculate primaries only on worker 0 by @davidselassie in #281
- Uses ruff to lint by @davidselassie in #278
- Update CI by @whoahbot in #277
- Update orderbook.py to use level2_batch by @awmatheson in #279
- Removes now-unused pickling infra by @davidselassie in #280
- Fix tiny typo in comment by @rabernat in #283
- File input fixups by @davidselassie in #282
- Fix kafka input error message by @Psykopear in #289
- Error when inconsistent recovery partitions by @davidselassie in #290
- Adding recovery entrypoint script by @miccioest in #291
- Add v0.17 upgrading guide. by @whoahbot in #288
- Prepare for v0.17.0 release. by @whoahbot in #292
New Contributors
Full Changelog: v0.16.2...v0.17.0