Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

omkafka doesn't work with security.protocol=ssl #1792

Closed
fenske opened this issue Sep 25, 2017 · 23 comments
Closed

omkafka doesn't work with security.protocol=ssl #1792

fenske opened this issue Sep 25, 2017 · 23 comments
Assignees

Comments

@fenske
Copy link

fenske commented Sep 25, 2017

I'm trying to enable ssl for omkafka with the following settings:
action(broker="<host>:9093" type="omkafka" topic="mytopic" confParam=["security.protocol=ssl", "ssl.ca.location=myca.crt"] template="ls_json")

But omkafka doesn't seem to work with those properties.

OS: CentOS7
rsyslog-kafka.x86_64: v8.29.0-2.el7
rsyslog.x86_64: 8.29.0-2.el7

@fenske
Copy link
Author

fenske commented Sep 25, 2017

@rgerhards maybe you have an idea.

@alorbach alorbach self-assigned this Oct 2, 2017
@alorbach
Copy link
Member

alorbach commented Oct 2, 2017

Can you post a debug log?
Also add "debug=all" into confParam for verbose librdkafka output.

@fchiriac
Copy link

fchiriac commented Oct 3, 2017

Also facing the same issue :

1170.205290725:omkafka_default_export queue:Reg/w0: omkafka.c: omkafka: setting custom configuration parameter: ssl.key.location:/home/bller/server_fe-sb-02.key
1170.205296873:omkafka_default_export queue:Reg/w0: omkafka.c: omkafka: tryResume returned -1000
1170.205299615:omkafka_default_export queue:Reg/w0: ../action.c: actionDoRetry: omkafka_default_export action->tryResume returned -1000
1170.205302347:omkafka_default_export queue:Reg/w0: ../action.c: actionDoRetry: omkafka_default_export enter loop, iRetries=0, ResumeInRow 1
1170.205307056:omkafka_default_export queue:Reg/w0: omkafka.c: omkafka: setting custom configuration parameter: ssl.key.location:/home/bller/server_fe-sb-02.key

This is right after the :

confParam=["security.protocol=ssl", not being recognized as a valid parameter in the configuration file

@rgerhards
Copy link
Member

@fchiriac are you also installing from package?

@alorbach @friedl could it be that the statically linked librdkafka has SSL support not enabled? Looks like this requires a define to be set: https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka

@rgerhards
Copy link
Member

@fchiriac it would be good if you could post the full debug log - the most interesting information is missing from the excerpt.

@fchiriac
Copy link

fchiriac commented Oct 4, 2017

@rgerhards i did try installing rsyslog as a package and by building it from source, the librdkafka was only built from source and configured to have SSL enabled, i will provide a full debug shortly.

@fchiriac
Copy link

fchiriac commented Oct 4, 2017

@rgerhards

debug.txt

Attached the output of the debug=all parameter within conf param, also the full conf param configuration in our env :

confParam=["security.protocol=ssl",
"debug=all",
"ssl.key.location=/path/to/some.key",
"ssl.key.password=xxxxxx",
"ssl.certificate.location=/path/to/some.cer",
"ssl.ca.location=/path/to/ca.cer"
]

Many thanks

@rgerhards
Copy link
Member

Ah, the error message is different from what you reported! It actually is:

 error in kafka parameter 'security.protocol=ssl': Invalid value for configuration property "security.protocol"

IMHO this makes clear what's going wrong. rsyslog passes the parameter to librdkafka, but librdkafka does not understand it. So librdkafka does not understand it for whatever reason.

@rgerhards
Copy link
Member

yup, it's definitely an issue with librdkafka (again, guess wrongly configured). The message originates here:
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_conf.c#L1040

I haven't looked extensively at the code, but I guess the root cause is that "ssl" is not found, because it is not present:
https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_conf.c#L428

@alorbach
Copy link
Member

alorbach commented Oct 4, 2017

The SSL support is indeed missing in omkafka and imkafka when using packages from our repository. The reason is that the statically linked librdkafka wasn't build with TLS support.

The new rsyslog packages should contain TLS support (should be updated soon).

@fchiriac
Copy link

fchiriac commented Oct 4, 2017

@alorbach Thank you so much, indeed, i did try to recompile rsyslog latest AFTER i compiled the latest librdkafka and the module now works as expected.

Many thanks ! This is now SOLVED for me.

@JPvRiel
Copy link

JPvRiel commented Oct 4, 2017

When I ran into the same issue, I also noticed something triggers a runaway memory consumption issue.

It ended up causing:

Oct  4 11:55:49 <hostname> kernel: [81114.991108] Out of memory: Kill process 10513 (rsyslogd) score 562 or sacrifice child
Oct  4 11:55:49 <hostname> kernel: [81114.991171] Killed process 10513 (rsyslogd) total-vm:19239440kB, anon-rss:9710088kB, file-rss:4kB, shmem-rss:0kB
Oct  4 11:55:49 <hostname> kernel: [81115.756676] oom_reaper: reaped process 10513 (rsyslogd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

It's not normal to see rsyslog chew up 9GB of RAM in just a couple of seconds after starting...

When run in debug mode, this output happens as an infinite loop with one iteration of output shown here:

...
5525.087860012:main Q:Reg/w0  : ../action.c: actionDoRetry: action 1 action->tryResume returned -1000
5525.087861862:main Q:Reg/w0  : ../action.c: actionDoRetry: action 1 enter loop, iRetries=0, ResumeInRow 1
5525.087864602:main Q:Reg/w0  : omkafka.c: omkafka: setting custom configuration parameter: batch.num.messages:1
5525.087867129:main Q:Reg/w0  : omkafka.c: omkafka: setting custom configuration parameter: queue.buffering.max.ms:100
5525.087869596:main Q:Reg/w0  : omkafka.c: omkafka: setting custom configuration parameter: security.protocol:ssl
5525.087872004:main Q:Reg/w0  : omkafka.c: omkafka: tryResume returned -1000
5525.087873702:main Q:Reg/w0  : ../action.c: actionDoRetry: action 1 action->tryResume returned -1000
...

I suspect it might be triggered by setting this for the omkafka action (but haven't tested extensively)

    resubmitOnFailure="on"
    keepFailedMessages="on"

I happy to take hints on how to debug this further...

My limited understanding is that:

  • The main rsyslog queue size is limited, so even if it ran crazy and kept spooling trying to resume on config errors, the queue shouldn't run out of memory and should discard messages/actions past queue limits?
  • I've checked that I haven't messed with the action.reportSuspension setting, so that's still default (but the docs don't mention what the default is).

I'm nervous that action.reportSuspension or action.reportSuspensionContinuation can cause a loop situation if, due to config parameter errors like this, an output causes an error to be logged/emitted and that error is again propagated to the output which generates yet another error, all the while retrying to send prior events which have also failed? Could be a self-inflicted error message spiral...

So perhaps config related issues should somehow not result in trying again?

More context on my config

module(load="impstats" interval="300" log.syslog="on")
...
ruleset(name="output") {
  call fwd_kafka
}
...
module(load="omkafka")
ruleset(name="fwd_kafka")
{
  action(
    type="omkafka"
    broker=["test_kafka:9093"]
    confParam=["batch.num.messages=1", "queue.buffering.max.ms=100", "security.protocol=ssl", "ssl.ca.location=/usr/local/etc/pki/test/test_ca.cert.pem", "ssl.key.location=/usr/local/etc/pki/test/test_syslog_server.key.pem", "ssl.certificate.location=/usr/local/etc/pki/test/test_syslog_server.cert.pem"]
    topic="test_syslog"
    dynatopic="off"
    partitions.auto="on"
    resubmitOnFailure="on"
    keepFailedMessages="on"
    failedMsgFile="/var/lib/rsyslog/kafka_failed.msg"
    template="TmplRFC5424Format"
  )
}

@rgerhards
Copy link
Member

I strongly suggest to use a different issue tracker for the different issue. Also, in my experience these type of memory issues usually stem back to librdkafka. I think just recently we had a similar request where librdkafka used up quite a bit of memory. Just my 2cts, possibly Andre can add more (but in a separate tracker, please!). Running rsyslog under valgrind might be useful to pinpoint the culprit.

@rgerhards
Copy link
Member

This was a packaging issue. Repackaged today.

@JPvRiel
Copy link

JPvRiel commented Oct 4, 2017

I strongly suggest to use a different issue tracker for the different issue.

Thanks, I will log a new/clean issue if it still results in a problem.

I guess an easy way to test is to initially give an invalid value, e.g. security.protocol=lies and see if the memory still goes nuts after I pull the repackaged version in.

@rgerhards
Copy link
Member

Sounds like a good plan! (Y)

@missnebun
Copy link

@alorbach Any update on omkafka and imkafka SSL support for the official rsyslog packages?

@rgerhards
Copy link
Member

rgerhards commented Dec 12, 2017

This should be solved succeed Oct, 4th (see above). Do you still have problems?

@missnebun
Copy link

@rgerhards I wasn't sure that was implemented. Also I didn't see any SSL config example in rsyslog documentation http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html

@rgerhards
Copy link
Member

yeah, we should probably add them -- but TBH I don't know what all needs to be done and what are the correct parameters for librdkafka (we just pass the parameters through to the lib, so rsyslog actually has nothing to do with how kafka TLS works). Would you be up to contribute a sample to the doc? Would be much appreciated...

@rgerhards
Copy link
Member

rgerhards commented Dec 12, 2017

@missnebun I guess this is all that is needed?

confParam=["security.protocol=ssl", "ssl.ca.location=myca.crt" ...]

@missnebun
Copy link

missnebun commented Dec 12, 2017

@rgerhards I am more then happy to share with the community my final config once it is finished.

From the https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka

At minimum:
security.protocol=ssl

# CA certificate file for verifying the broker's certificate.
ssl.ca.location=ca-cert

# Client's certificate
ssl.certificate.location=client_?????_client.pem

# Client's key
ssl.key.location=client_?????_client.key

# Key password, if any.
ssl.key.password=abcdefgh

@lock
Copy link

lock bot commented Dec 26, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Dec 26, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants