Teach elasticsearch_http-output about HTTPS

[alexbrasetvik]

While this change is fairly simple, there is an issue with FTW and/or the flushing that causes it to lock up when more than flush_size events are queued. That issue should probably be fixed before this one is merged.

[tisba]

I could really use this change.

There seems to be some issues (#1262, #430) around that topic. Is there any progress/interest in this feature?

[DSpeichert]

There sure is interest!

[jordansissel]

I am in favor of this change myself and given the popularity and demand for SSL, we should do this.

elasticsearch_http is being deprecated in favor of the elasticsearch plugin when used with protocol => http.

Can we add this to the elasticsearch plugin instead?

I have specific comments other than the move to elasticsearch

• is there agreement on the setting name? Should it be protocol => https ? Or should it be a boolean ssl => true or something?
• If we add ssl support, we must expose settings for specifying, at a minimum, ssl server/ca certs. It would be very nice to have ssl client cert support, too.

[tisba]

I'm pretty new to logstash, if the elasticsearch plugin is the way to go, I'm totally fine with it :)

If the elasticsearch plugin supports multiple protocols and each of them may support SSL/TLS then maybe ssl => true is the superior option.

I'm also absolutely +1 for supporting client certificates.

[colinsurprenant]

+1

we could use the same ssl options as the tcp input ?

[jordansissel]

@colinsurprenant it feels weird having settings that are only valid under certain conditions.

Settings from tcp:

• ssl_enable - don't need here, because we'll have protocol => https
• ssl_cacert - want
• ssl_cert - want (perhaps call this ssl_client_cert? I dunno)
• ssl_key - want
• ssl_passphrase - want
• ssl_verify - do not want. If you don't want security, don't use ssl.

[jordansissel]

Proposed settings:

• protocol => https
• ssl_cacert => ... path ... - path to a file (or directory of files) to load as trusted CA certs
• ssl_cert => ... - path to the client's ssl cert
• ssl_key => ... - path to the client's ssl key

Not sure:

• ssl_passphrase => ... - I'm on the fence about this. I ceases to be a secret passphrase if you write it down in plain text in a logstash config. My preference is to prompt for passphrase on startup. Thoughts?

[colinsurprenant]

or make it possible to get setting from an environment variable?

[jordansissel]

Passing in the passphrase from an env var? I'm open to that, but that still provides a vector for leaking the passphrase (ps can show env vars)

[colinsurprenant]

true. not sure what the best practices are for passing a password to an app other what having a prompt at startup - which in LS case would not be very practical.

[jordansissel]

Related: elastic/elasticsearch-ruby#80

[elasticsearch-release]

Can one of the admins verify this patch?

[wiibaa]

@jordansissel the initial issue of supporting https with elasticsearch output protocol =>http is implemented in logstash-plugins/logstash-output-elasticsearch@bedbba9
so this PR could be closed if you can please move your open question about passphrase enhancement to a dedicated issue

[suyograo]

@alexbrasetvik we 've added https suport to elasticsearch support. https://github.com/logstash-plugins/logstash-output-elasticsearch