Skip to content

v0.17.0

Compare
Choose a tag to compare
@whoahbot whoahbot released this 28 Aug 15:53
· 1239 commits to main since this release
e2df62c

v0.17.0

Changed

  • Breaking change Recovery system re-worked. Kafka-based recovery
    removed. SQLite recovery file format changed; existing recovery DB
    files can not be used. See the module docstring for
    bytewax.recovery for how to use the new recovery system.

  • Dataflow execution supports rescaling over resumes. You can now
    change the number of workers and still get proper execution and
    recovery.

  • epoch-interval has been renamed to snapshot-interval

  • The list-parts method of PartitionedInput has been changed to
    return a List[str] and should only reflect the available
    inputs that a given worker has access to. You no longer need
    to return the complete set of partitions for all workers.

  • The next method of StatefulSource and StatelessSource has
    been changed to next_batch and should return a List of elements,
    or the empty list if there are no elements to return.

Added

  • Added new cli parameter backup-interval, to configure the length of
    time to wait before "garbage collecting" older recovery snapshots.

  • Added next_awake to input classes, which can be used to schedule
    when the next call to next_batch should occur. Use next_awake
    instead of time.sleep.

  • Added bytewax.inputs.batcher_async to bridge async Python libraries
    in Bytewax input sources.

  • Added support for linux/aarch64 and linux/armv7 platforms.

Removed

  • KafkaRecoveryConfig has been removed as a recovery store.

What's Changed

New Contributors

Full Changelog: v0.16.2...v0.17.0