New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
omkafka doesn't work with security.protocol=ssl #1792
Comments
@rgerhards maybe you have an idea. |
Can you post a debug log? |
Also facing the same issue : 1170.205290725:omkafka_default_export queue:Reg/w0: omkafka.c: omkafka: setting custom configuration parameter: ssl.key.location:/home/bller/server_fe-sb-02.key This is right after the : confParam=["security.protocol=ssl", not being recognized as a valid parameter in the configuration file |
@fchiriac are you also installing from package? @alorbach @friedl could it be that the statically linked librdkafka has SSL support not enabled? Looks like this requires a define to be set: https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka |
@fchiriac it would be good if you could post the full debug log - the most interesting information is missing from the excerpt. |
@rgerhards i did try installing rsyslog as a package and by building it from source, the librdkafka was only built from source and configured to have SSL enabled, i will provide a full debug shortly. |
Attached the output of the debug=all parameter within conf param, also the full conf param configuration in our env : confParam=["security.protocol=ssl", Many thanks |
Ah, the error message is different from what you reported! It actually is:
IMHO this makes clear what's going wrong. rsyslog passes the parameter to librdkafka, but librdkafka does not understand it. So librdkafka does not understand it for whatever reason. |
yup, it's definitely an issue with librdkafka (again, guess wrongly configured). The message originates here: I haven't looked extensively at the code, but I guess the root cause is that "ssl" is not found, because it is not present: |
The SSL support is indeed missing in omkafka and imkafka when using packages from our repository. The reason is that the statically linked librdkafka wasn't build with TLS support. The new rsyslog packages should contain TLS support (should be updated soon). |
@alorbach Thank you so much, indeed, i did try to recompile rsyslog latest AFTER i compiled the latest librdkafka and the module now works as expected. Many thanks ! This is now SOLVED for me. |
When I ran into the same issue, I also noticed something triggers a runaway memory consumption issue. It ended up causing:
It's not normal to see rsyslog chew up 9GB of RAM in just a couple of seconds after starting... When run in debug mode, this output happens as an infinite loop with one iteration of output shown here:
I suspect it might be triggered by setting this for the omkafka action (but haven't tested extensively)
I happy to take hints on how to debug this further... My limited understanding is that:
I'm nervous that So perhaps config related issues should somehow not result in trying again? More context on my config
|
I strongly suggest to use a different issue tracker for the different issue. Also, in my experience these type of memory issues usually stem back to librdkafka. I think just recently we had a similar request where librdkafka used up quite a bit of memory. Just my 2cts, possibly Andre can add more (but in a separate tracker, please!). Running rsyslog under valgrind might be useful to pinpoint the culprit. |
This was a packaging issue. Repackaged today. |
Thanks, I will log a new/clean issue if it still results in a problem. I guess an easy way to test is to initially give an invalid value, e.g. |
Sounds like a good plan! (Y) |
@alorbach Any update on omkafka and imkafka SSL support for the official rsyslog packages? |
This should be solved succeed Oct, 4th (see above). Do you still have problems? |
@rgerhards I wasn't sure that was implemented. Also I didn't see any SSL config example in rsyslog documentation http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html |
yeah, we should probably add them -- but TBH I don't know what all needs to be done and what are the correct parameters for librdkafka (we just pass the parameters through to the lib, so rsyslog actually has nothing to do with how kafka TLS works). Would you be up to contribute a sample to the doc? Would be much appreciated... |
@missnebun I guess this is all that is needed?
|
@rgerhards I am more then happy to share with the community my final config once it is finished. From the https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
I'm trying to enable ssl for omkafka with the following settings:
action(broker="<host>:9093" type="omkafka" topic="mytopic" confParam=["security.protocol=ssl", "ssl.ca.location=myca.crt"] template="ls_json")
But omkafka doesn't seem to work with those properties.
OS: CentOS7
rsyslog-kafka.x86_64: v8.29.0-2.el7
rsyslog.x86_64: 8.29.0-2.el7
The text was updated successfully, but these errors were encountered: