2.0 Roadmap

[mensfeld]

The development of rdkafka-ruby is in progress. Once some neded features are available, we could migrate to it.

Things required:

• Check the config validators quality and the way they operate (Check the quality of rdkafka validations #434)
• Migrate producer (WaterDrop) (2.0 waterdrop#106)
• Normalize the settings (Normalize naming based on kafka recommendations #425)
• drop jruby support for Karafka ecosystem (Drop support for JRuby #433)
• Normalize params and metadata setup
• Release Waterdrop 2.0
• Provide fork. thread and guild/ractor backends
• batch processing
• batch metadata (or a way to calculate those)
• settings normalization (Normalize naming based on kafka recommendations #425)
• Long running loop (Handle EOF partition error properly in Consumer#each rdkafka-ruby#12)
• hooks for before/after poll and other lifecycle moments
• lower dependencies count
• Expose API for managing Karafka from within Puma or Sidekiq process - not going to happen. Stupid idea.
• Check if there's a possibility to improve the Rails integration (more seamless) - there is: done
• 100% code coverage (in progress 85%, 87.51%, 89%, 93%) - rest will be covered with integration specs
• Instrumentation hooks
• WaterDrop + Karafka baseline integration
• Support #seek on the client
• Integration specs
• Updated readme
• Example app update
• Sync testing lib
• rdkafka updated callbacks support for Karafka and WaterDrop - marking as done as no longer needed as I bypassed the issue (Better callbacks support + statistics bug fixes waterdrop#216)
• Benchmarks
• License update (LGPL update + Pro + License token verification #770)
• Long Running Jobs (LRJ) support (part1: Long Running Jobs support for ActiveJob #869)
• Virtual partitions
• Release notes
• Docs
• ActiveJob adapter for jobs
• Update the error callbacks and statistics callback with WaterDrop 2.0.5 style of listeners (introduce consumer and client name #729, Async errors and statistics hooks #732)
• Prepare updated versions of the "Getting started" articles
• Update WaterDrop readme to indicate 2.0 is out
• Remove OpenCollective
• Upgrade notes
• Release blog post

Post-release:

• Remove Karafka git reference from the testing gem Gemfile and uncomment gemspec reference
• Release testing package

[mensfeld]

ref: #428 - closed as by itself is irrelevant.

[mensfeld]

For whoever reads it: I'm working on it. Things look good :)

[nijikon]

I know 💜

[luong-komorebi]

Wish you all the best! Will try to give a hand where possible.

[mensfeld]

Next update: wip branch is pushed.

[mensfeld]

More updates to come in the upcoming weeks. 2.0 works but I still work on many things (mostly quality, stability and docs).

[luong-komorebi]

Do you have a project management board? May share it publicly to invite people to help ?

[mensfeld]

@luong-komorebi just this ^^ There are specs needed (that's what I'm working on) and updated docs (once specs are done). The rest is done. What type of details would you need to help?

[luong-komorebi]

Ahh, I dont need to help. Sounds like things are moving really well. I was just worrying that one man effort may be devastating. Best of luck

[mensfeld]

@luong-komorebi slower than I would want but moving forward :) it's a huge re-write with many performance improvements, that's why it's taking so much time.

[sandipsubedi]

This is exciting! Thank you so much for your hard work here 🥇

[mensfeld]

@sandipsubedi thank you for your kind words. I'm moving forward with each day.

[mensfeld]

Another set of specs pushed. There are 3 main classes that need more coverage and I'm done with specs. Then I will update appropriate instrumentation hooks and we can finally move forward :)

[mensfeld]

Many new things: much much better Rails integration, WaterDrop integration out of the box, fixed error handling, specs, we're getting closer to closing the code development and getting to docs :)

[mensfeld]

aaand ActiveJob Karafka adapter POC is done :) works great!

Performed TestWorker (Job ID: 0f7c403c-dcda-4531-8175-5e0d729aa5ef) from Karafka(default) in 0.25ms

[mensfeld]

Done with unit tests at 95%. The rest will be tackled with the integration specs that I'm working on.

[mensfeld]

Integration specs are done 🎆 . I will be adding more down the road if any corner cases come to my mind. Time to start writing docs and upgrade notes :)

[mensfeld]

Integration specs are finally stable. After 24h hours no random crashes or any other issues... 😅

[mensfeld]

Almost there. Last things for Rails activejob plugin and the hardest part: docs 😅

[mensfeld]

@luong-komorebi

Do you have a project management board? May share it publicly to invite people to help ?

I started a slack channel: https://slack.karafka.io/ :) feel free to join.

[mensfeld]

I have nothing more to do at the moment in the code 💪 Starting to rework / update wiki + upgrade notes + and release notes.

[mensfeld]

I have nothing more to do at the moment in the code

LOL and here I changed like 5000 lines to provide better stability, performance and features 😅

But I'm left with 1.5 pro feature only. Gosh...

[mensfeld]

RC1 will be released today.

[mensfeld]

https://rubygems.org/gems/karafka/versions/2.0.0.rc1

[mensfeld]

Gosh almost 4 years. I think many ppl thought it would not happen:

🚀 🔥 With great pleasure and high hopes I can say, that karafka 2.0.0 has been released. https://rubygems.org/gems/karafka/versions/2.0.0 / https://my.diffend.io/gems/karafka/1.4.13/2.0.0

🎆 closing this in favor of small post-release tasks.