Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Looking for more Collaborators! #628

Open
webmakersteve opened this issue Jun 4, 2019 · 45 comments
Open

Looking for more Collaborators! #628

webmakersteve opened this issue Jun 4, 2019 · 45 comments
Labels
help wanted

Comments

@webmakersteve
Copy link
Contributor

Hi everybody.

First off, I just want to thank the community for all of the support this repository has gotten. A few years ago when I started working on what would become node-rdkafka, I never could have imagined it getting this big! It wouldn't be nearly as big without all of you - your contributions, your bug reports, and the inspiration to keep me working on it.

This project originated on Blizzard's data team. We built out kafka support in node.js because we were using node for message streaming from HTTP to kafka to be processed by our downstream services. A few months after that, we stopped using node for new projects so we started using the library less. I still supported it as best I could, and we had still had existing services that used it that were made better with every release!

About 6 months ago I moved to another team that doesn't use any node.js, and the amount of time I've had to commit on this repository has gone to its lowest ever. This would be more OK, but my free weekend-time has also been at an all time low.

So, this has all been a long way of saying I need help! I'd like to give some of the responsibility for the management of this project to some of you. If you are interested at all please email me at sparente@blizzard.com!

I'm hoping to get more active again as best I can to give this project the support it deserves. There's a lot of work to be done on it to make it even better! If that excites you, send me an email!

Thank you everyone for your continued patience. Its totally undeserved, but appreciated :)
@JaapRood
Copy link
Contributor

I think I speak on behalf of a lot of people to say: 🎉thank you for all your effort so far and in the future❤️! So many people, including myself personally and my team professionally, have benefited from all of your hard work, getting this setup in the first place.

I really hope someone with experience in both C++ and Node is willing to take on a contributor role, as especially for high-throughput cases it's hard to beat binding with the excellent librdkafka. Are there any companies / teams out there (apart from Blizzard) with these skillsets that use node-rdkafka to run any production services?

Unfortunately, that excludes myself, as all my C++ experience is just reading and hacking some other people's code. We're only a 2 person indie team, so lacking the resources to address the issues the project has been facing as of late. It pushed us towards adopting a client written 100% in JS for all new stream processing tasks, but without any of the luxuries of librdkafka's throughput, streams, near-mirrored consumer API's, etc.

@rusty0412
Copy link
Contributor

rusty0412 commented Jul 10, 2019 •
edited
Loading

What kind of help do you need? Do you have a roadmap of things to improve, modernize? Do you know of bottlenecks that need to be optimized? Would you be open to dropping support for unsupported node releases (<8.0)? Would you be open to modernizing tooling (TypeScript, eslint, prettier commit hooks)?

@webmakersteve
Copy link
Contributor Author

@rusty0412

There isn't specifically a road-map. So far there has been very little process involved in what we are upgrading or doing at any given time. Previously it was driven by what I was having to work with at the time (since I was actually using the library), but nowadays I've done very little to actually push the library forward and rely on just merging PRs that come in and providing feedback.

Here is what I'd like to see done in the future:

  1. Drop all support for < 8
  2. Add promise compatible APIs for async functions, or just an easy Promisify wrapper available to wrap all the method calls so they can be used with async await
  3. Upgrade NAN and get rid of all of the deprecations we have. This is a pretty big effort as the async worker handling has been hugely changed.
  4. A lot of C++ refactoring - the code is very old and looks it. There is a lot of stuff that can be consolidated to reduce the huge amount of sprawl.
  5. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time.
  6. First-class typescript support
  7. A lot of CI improvements. Maybe migrate to circle CI? But mainly I want the builder to publish the package on every master build with pre-release tags and auto publish tagged builds.
  8. Better testing. Need to make a mocked version of the thin librdkafka classes and inject those around.
  9. Continue implementing new features of librdkafka. There are ways we can do FD polling for more efficient consumption in streams, etc.

If this sounds interesting to you, let me know! But I can also announce that I've added @iradul, @ankon , and @codeburke as collaborators, so thank you guys for volunteering! I appreciate all the help.

@codeburke
Copy link
Collaborator

Looking forward to collaborating on this project @webmakersteve, @iradul, @ankon!

I'm not going to be much help in the C/C++ world, but willing to help how I can. I can definitely start chipping away on some modernization efforts like dropping <8 support, async/await/Promise support, and improvements to CI. Seems like we should open an Issue for each of these to start discussion about what we are planning on changing and let people provide feedback/PRs.

@iradul
Copy link
Collaborator

iradul commented Jul 25, 2019

But I can also announce that I've added @iradul, @ankon , and @codeburke as collaborators, so thank you guys for volunteering! I appreciate all the help.

Thank you for adding us to the team!

  1. First-class typescript support

Should we consider rewriting the whole project in TypeScript? I think that would bring huge benefits.

Also, it might be helpful if we start using Projects feature to manage and track progress of all the things you'd like to see done in the future.

@IdanAdar
Copy link

IdanAdar commented Jul 25, 2019 •
edited
Loading

@webmakersteve
I think the first thing needed is a more direct line of communication; getting replies is very sporadic and puts the package at high risk. For example right now it's just broken in 2.71, but there's no quick way to reach you - since there is no one else, or very very few, who can merge and kick off new releases - so teams around the world that rely on this package must wait for long period of time. We need to communicate. Lets solve this first.

How about a gitter channel? I've created this: https://gitter.im/node-rdkafka/community
Lets join.

@codeburke
Copy link
Collaborator

codeburke commented Jul 25, 2019 via email

@SoyYoRafa
Copy link

SoyYoRafa commented Jul 26, 2019 •
edited
Loading

Hi,
I am interested in contributing to a TypeScript migration. This project has no need for any dynamic typing behavior and the library needs to be robust. Static type checking will drive robustness since it is a type of test. It will help contributors understand the code layout. TypeScript will allow the project to use more modern syntax than allowed by supported runtimes. It will also guarantee high quality, error-free typing declarations to library users. If it's not TypeScript native, the declarations and API will drift and become inconsistent.

@codeburke
Copy link
Collaborator

codeburke commented Jul 26, 2019 via email

@SoyYoRafa
Copy link

Will that mean that library users will all need to transpile?

No. node-rdkafka's build script would invoke the TypeScript compiler. The TypeScript compiler would then generate javascript files and type declarations. node-rdkafka would publish the javascript files and type declarations files. Library users just run the javascript files, they are unaware the project is written in TypeScript. TypeScript users also run the javascript files but will use the generated type declaration files in their code.

Additionally, you can use syntax newer than es6 in TypeScript and configure the compiler to generate es6-compatible javascript. When each version of node gets deprecated, you just bump up the target in the TypeScript compiler config and you get the native code for that target.

@JaapRood
Copy link
Contributor

JaapRood commented Jul 26, 2019 •
edited
Loading

From my experience to contributing to node-rdkafka, I'd have to underline this mention of @webmakersteve:

  1. Upgrade NAN and get rid of all of the deprecations we have. This is a pretty big effort as the async worker handling has been hugely changed.
  2. A lot of C++ refactoring - the code is very old and looks it. There is a lot of stuff that can be consolidated to reduce the huge amount of sprawl.
  3. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time.

Those are the definitely the points causing a bunch of problems at the moment, the ones that have been causing stability issues thus far and threaten to do so even more in the future. The binding to librdkafka is what sets node-rdkafka apart with a unique value proposition: piggy backing of an very highly optimized native library that achieves order of magnitude higher throughput than the official Kafka client. That's the bit worth trying to save, imo.

It's not that the Javascript side couldn't do with some love, it definitely could. But if what you're looking for is a modern JS code base, Typescript support, Promise support, strong test suite and CI, I wonder if you're time is not better spent on the newer KafkaJS project, which already has all of those things and is at a high rate of further development and progress. It's what we as a tiny 2-person indie team had to move to, and since we don't have much C++/C expertise, it's 100% Node codebase makes our contributions a whole bunch easier and more productive.

That's not to say that I think node-rdkafka shouldn't be kept alive / revived, not at all! Given it's bindings to librdkafka I could see it as a simpler client that's useful if you need some really serious throughput; it could offer levels of performance hard to match with a fully Javascript-implemented client. But then the focus should at least be on that part, that unique strength. Otherwise, all we're doing is fragmenting the already small Node + Kafka ecosystem even further and just duplicating efforts :(

@codeburke
Copy link
Collaborator

codeburke commented Jul 26, 2019 via email

@JaapRood
Copy link
Contributor

@codeburke unfortunately I can't be, through lack of experience with that side of the code base as well:

Unfortunately, that excludes myself, as all my C++ experience is just reading and hacking some other people's code. We're only a 2 person indie team, so lacking the resources to address the issues the project has been facing as of late.

@SoyYoRafa
Copy link

We can expect that users' runtimes will support N-API by January 2020 when Node 8 is deprecated. N-API guarantees API stability and ABI compatibility across major versions of node. So, moving to N-API has the big benefit of being able to compile node-rdkafka in node 10 and run it in future major versions of node (12, 14, etc) without recompiling the shared objects. As a result, I don't think upgrading NAN is worthwhile. Separately, we should use node-addon-api, not N-API directly. N-API is a low level C interface. node-addon-api is the C++, object oriented wrapper on N-API. So, around November/December would be a good time to start working on migrating to node-addon-api.

@iradul
Copy link
Collaborator

iradul commented Jul 28, 2019

Thumbs up for N-API. Looks like it does support Node 8 according to this:
https://nodejs.org/docs/latest/api/n-api.html#n_api_n_api_version_matrix
Also, since librdkafka is a C library and N-API is C API it would make sense to use C API instead of N-API C++ wrapper.

@SoyYoRafa
Copy link

SoyYoRafa commented Jul 28, 2019 •
edited
Loading

@iradul node-rdkafka uses the C++ api of librdkafka. NAN->node-addon-api is a simple mechanical change. NAN->N-API is a rewrite. N-API is marked experimental in node 8. I am not sure if its safe to use before node 8.11.2, when N-APIv3 was backported.

@iradul
Copy link
Collaborator

iradul commented Jul 28, 2019 •
edited
Loading

Mechanical change or not it's still a rewrite unless it can be done in some safe and automated way. I'm sure it would be somewhat easier to rewrite using N-API C++ wrapper but in a long run C API seems better choice because librdkafa is a C library and librdkafka C++ always lags behind it.

@SoyYoRafa
Copy link

SoyYoRafa commented Jul 28, 2019 •
edited
Loading

Regardless of label, what I wanted to communicate was that NAN->node-addon-api is pretty simple. I have already done it. I did it manually but there is a conversion tool that automates a portion of the conversion, https://github.com/nodejs/node-addon-api/tree/master/tools.

Re: using the librdkafka C interface, we would have to build an object oriented wrapper on the librdkafka C API either in C++ or JavaScript/TS. Do you think it's more likely that this project will be ahead of librdkafka's own C++ wrapper or that this project lags the librdkafka C++ wrapper? Also, why not just contribute to librdkafka C++ wrapper?

@IdanAdar
Copy link

IdanAdar commented Jul 28, 2019 •
edited
Loading

May I suggest opening an issue and moving the discussion to there? In this issue we need to find out who wants to collaborate and become an active, or at least more-available maintainer. Without this this package is going nowhere.

I am less proficient than you guys, but I can be highly available and utilize your expertise with the the more complicated PRs. Anyone else willing to chime in?

@webmakersteve It's also up to you to add more maintainers...

@codeburke
Copy link
Collaborator

codeburke commented Jul 28, 2019 via email

@iradul
Copy link
Collaborator

iradul commented Jul 29, 2019

Do you think it's more likely that this project will be ahead of librdkafka's own C++ wrapper or that this project lags the librdkafka C++ wrapper? Also, why not just contribute to librdkafka C++ wrapper?

Yes, for example it happened with headers support. C library got it on Jan 4, 2018 and C++ on Nov 26, 2018.

@davidtrihy
Copy link
Contributor

I'd be happy to lend support where I can, internally in our team we use this and I've got a good bit of experience in C/C++ although not so much in NAN and primarily my development language at the moment is NodeJS, it would be good to get some people on board for the NodeJS side of things and I could assist in the C++ side of things as much as I can.

@iradul iradul mentioned this issue Aug 24, 2019
Closed
@codeburke
Copy link
Collaborator

codeburke commented Sep 12, 2019 via email

@iradul
Copy link
Collaborator

iradul commented Sep 12, 2019

Just wanted to resurrect this thread and see if there was any movement or
plan (coordinated or otherwise)?

It seems publishing to NPM is still a manual task, so I don't know if merging existing PRs would be helpful to anyone. I guess we're still waiting for @webmakersteve to coordinate us, collaborators.

@codeburke
Copy link
Collaborator

codeburke commented Oct 17, 2019 via email

@noderat
Copy link

noderat commented Oct 17, 2019 •
edited
Loading

@iradul Does this mean you also received permissions to publish a new release to npm?

@iradul
Copy link
Collaborator

iradul commented Oct 18, 2019 •
edited
Loading

No, there was something promising in .travis.yml but it turns out it's not working (#700)

@uatuko
Copy link

uatuko commented Oct 23, 2019

I really hope we can have maintainers with access to publish npm packages instead of having to create forks.

@iradul, have you been in contact with @webmakersteve recently? If there has been no response for a while maybe we should consider a fork to back as a community before everyone starts to fork and publish their own versions.

@SoyYoRafa
Copy link

I support a fork at this stage. 2019 has been a lost year for this repository.

@Tapppi
Copy link
Contributor

Tapppi commented Oct 28, 2019

Our company is also currently pondering what to do with our Kafka library usage, basically whether there is a fork the community will back or to migrate to the new KafkaJS project and accept lower performance. For this project it seems best to try and create a mainstream fork at this point.

@uatuko
Copy link

uatuko commented Oct 28, 2019

We've actually started migrating to KafkaJS. The maintainers and owner aren't responsive enough for this to be a production ready package anymore IMHO.

TypeScript has also been mentioned as a strong contender above, which makes me doubt the future sustainability of this package.

It's a shame a package like this goes stale and I'm more than happy to help out/support a fork with a sustainable roadmap. But since I posted my last comment 5 days ago there have been very little interest (and no response from maintainers or the owner). Does than mean only a handful are still interested in this package?

@codeburke
Copy link
Collaborator

codeburke commented Oct 29, 2019 via email

@ramirlm
Copy link

ramirlm commented Nov 6, 2019

Awesome work here! I'll be glad to collaborate!

@iradul
Copy link
Collaborator

iradul commented Nov 12, 2019 •
edited
Loading

@iradul, have you been in contact with @webmakersteve recently?

Yes. He is very busy lately but he'll have some updates soon. Let's be patient guys!

@iradul
Copy link
Collaborator

iradul commented Nov 15, 2019

Good news! We can now publish new versions via Travis. I've successfully published 2.7.4.

@keith-chew
Copy link

keith-chew commented Nov 23, 2019 •
edited
Loading

I have been testing the robustness of node-rdkafka and found some issues, which I reported here:
confluentinc/librdkafka#2630
confluentinc/librdkafka#2631
confluentinc/librdkafka#2634
#705
#715

I have created a fork to support some of the issues above:
https://github.com/keith-chew/node-rdkafka

Hope this will be of use for others as well.

@majusko
Copy link

majusko commented Dec 2, 2019

I created an RxJs wrapper with Typescript using this library. I'm able to help with Typescript + Reactive support for this library and make it modern.

Check it out and if you are interested we can agree on some collaboration where I would merge this project with your amazing node-rdkafka :)

https://github.com/majusko/rx-kafka

@analytik analytik mentioned this issue Feb 26, 2020
Closed
@sam-github sam-github mentioned this issue Mar 10, 2020
Merged
@giulliano-bueno
Copy link

giulliano-bueno commented Jul 2, 2020 •
edited
Loading

Hi, my company is using this library in a few projects and I and my team would like to contribute and maintain it. How could I become a collaborator, please?

@iradul
Copy link
Collaborator

iradul commented Jul 20, 2020

Hi @giulliano-bueno. Great to have you! Only @webmakersteve can add collaborators but you are more than welcome to make contributions. I'll make sure all good PRs get merged.

@anonimitoraf
Copy link

anonimitoraf commented Dec 1, 2020 •
edited
Loading

Hey guys! I'm interested in supporting node ver 14.x.x since it has a bunch of good stuff like optional chaining and whatnot (also because it's an LTS)

I'll try make a start this weekend

@anonimitoraf
Copy link

Weirdly, node-rdkafka as it is, works for node 14.15.0 for me. Can anyone else confirm?

@lxghtless
Copy link

Has anyone started work on this one?

  1. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time.

@edenhill
Copy link

edenhill commented Feb 7, 2021

Our librdkafka.redist nuget packages contains highly compatible librdkafka builds with most dependencies built in, these libs are what we bundle in the confluent-kafka-python, confluent-kafka-go and confluent-kafka-dotnet clients, so I'd advise to use and bundle those.
https://www.nuget.org/packages/librdkafka.redist/

@codeburke
Copy link
Collaborator

codeburke commented Feb 7, 2021 via email

@edenhill
Copy link

edenhill commented Feb 7, 2021 •
edited
Loading

FYI: https://github.com/edenhill/librdkafka/wiki/librdkafka.redist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests