v0.8.0

Check out our release blog.

Major Changes

• Ability to assign event time, as well as filtering and transformations at the source vertex using source transformers
• Idle watermark propagation now does watermark progression in idle edges reducing close-of-book latency
• Message delivery count is now being tracked and is exposed to user-defined code
• Support Redis Streams Source
• Performance improvements
  • Message SerDe has been optimized for better encoding/decoding
  • WAL write/read CPU performance was boosted by 20%, and disk space optimization by close to 40%
• Controller to support high availability
• Kill switch feature - the ability to drop a message on demand when the buffer is full
• UX improvements
  • Ability to do namespace scoped installation
  • Switched to namespace search on the main page
  • Be able to show reduce vertex and edge information
• SASL support for both Kafka source and sink
• Late data processing using allowedLateness

Breaking Changes

• Introduced tags for conditional forwarding, removed keyIn, see the detail at here
• SDK changes
  • Please move to the latest 0.4.x release of each SDK
  • Python
    • Major refactor on server startup, please review the README
    • Added Async Map and Async Sink
    • Map has multi-process support via SO_REUSEPORT
    • Removed Python 3.8 support and added support for 3.11

Pull Requests

v0.7.0...v0.8.0

Contributors

@KeranYang
@vigith
@yhl25
@ashwinidulams
@juliev0
@chromevoid
@dpadhiar
@veds-g
@sandangel
@whynowy
@kohlisid
@xdevxy
@alexanghh

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.8.0/install.yaml