TLS support?

[jhaar]

Hi there

I'm wanting to push GELF over the Internet and so want to use graylog-server's TLS support on it's GELF port - but I can't find a client that supports TLS (besides their java graylog-collector)

Would it be possible to add TLS support to pygelf?

[keeprocking]

I've never worked with TLS but I'll consider your request.

[keeprocking]

Still don't have any progress on this issue. Would it be acceptable to send logs over HTTPS? If so, I could add an HTTP handler for this purpose.

[jhaar]

I don't know - I've never used HTTP GELF before. The graylog documentation shows an example

curl -XPOST http://graylog.example.org:12202/gelf -p0 -d '{"short_message":"Hello there", "host":"example.org", "facility":"test", "_foo":"bar"}'

But it seems to be HTTP/1.0 - ie it's all about single TCP transactions per HTTP request. It also doesn't support HTTPS (yeah, surprised me too - given that their other Input channels do - I'm going to mention that to their devs). But that could be got around via stunnel or the like

Being single transaction based would mean a bunch of overhead not seen on your TCP channel option, but might be worth doing - especially if it means you could just use an existing HTTP(S) library instead of having to do all that yourself

(I'm guessing GELF-over-HTTP was really meant as a webservice for logging random application messages - not really intended for massive logging - like the access_log streaming I for one want to do)

[keeprocking]

It also doesn't support HTTPS

It doesn't but you can run graylog behind nginx. Also the documentation says:

Both keep-alive and compression are supported via the common HTTP headers.

[jhaar]

FYI graylog-1.2-rc was just announced and they have introduced GELF-over-HTTPS support - but they say it is one record per HTTP transaction

[keeprocking]

Hey, check out new branch called 'tls'. Have tested it with self-signed certificate today, everything worked like a charm. I haven't updated readme yet, so here are a few instructions:

• in addition to the usual TCP options pass the path of a certificate to the constructor:

  handler = GelfTlsHandler(host='127.0.0.1', port=1234, cert='server.crt')
  

• in order to make the TCP input support TLS, the private key must be in PKCS every GELF record triggers getfqdn? #8 format and the certificate - in the X.509 format: see TLS support over the TCP input Graylog2/graylog2-server#1111. I got some 'Connection reset by peer' errors until I have converted my private key to the right format.

Please let me know if you find any bugs.

[jhaar]

Got it working - but I've caught a couple of things

Your "cert=" call is actually enabling client certificate validation - which isn't used/supported by graylog-server until 1.2 (apparently - just read this yesterday). So it's not really needed in "traditional" mode where you just want the client to connect to the server. Only the "ca*" variables and "cert_reqs" are related to server-only validation

99.9% of graylog-server users would get a cert validation error using pygelf - as typically the server is self-signed. It's always best practice to default to "secure mode" - so your code is fine, but perhaps it would be worth documenting how to disable server cert validation for those who don't care (ie they want to use TLS as a privacy measure instead of a security measure)

I'm not a python coder, so I actually couldn't figure out how to properly make pygelf disable cert validation - I ended up hacking tls.py to change to cert_reqs=ssl.CERT_NONE to do that - but I assume I should have been able to do that from my script instead? (and strace then showed that the "cert=" file was never opened - verifying my above statement about client certs :-)

However, it still works! Awesome :-)

[keeprocking]

Thanks for clarifying. I definitely have misunderstood you.
I made some changes according to your feedback. Now 'cert' is an optional parameter: if you provide it, it will be validated. If not, then just connect in 'secure mode'.

Did I get it right this time? :)

[jhaar]

Well the way I interpret your changes are...

if self.cert is None:
wrapped_socket = ssl.wrap_socket(s)

So that says if you don't call it with a client cert, ssl-open without any parameters - which I assume means it doesn't validate the server cert

wrapped_socket = ssl.wrap_socket(s, ca_certs=self.cert, cert_reqs=ssl.CERT_REQUIRED)

Otherwise this says, validation is required - and the CA will be the client cert

Here's a guess: am I correct in saying you've been testing on localhost? ie the graylog-server uses the same cert as pygelf? Because that's not the way to do it :-)

"ca_certs" normally points to a PEM file containing a list of all the public keys of known CAs (ie not containing private keys). Then "CERT_REQUIRED" is telling pygelf to connect to the TLS service, download the server public key (which is always exposed as part of TLS) and then see if that server cert was signed by a known CA. Your "ca_certs=self.cert" trick should only work if the server cert is the same as the client cert (whether it be self-signed or signed by a CA) - hence my comment about localhost. Sorry it took so long for me to figure that out - should have noticed that sooner

To reiterate: the way TLS works is that clients (eg browsers) don't need to know about the server certs used by the millions of websites out there - they only need to know about the CAs that sign the server certs.

So you should really generate a different client cert for testing - but as I mentioned earlier, graylog-server doesn't support client cert authentication today, so it would never get called anyway (ie there is actually no point of having the "cert" option - as it cannot work as intended against current versions of graylog-server, which don't support client cert verification)

I think it should do the following - call it with ca_certs="/etc/pki/tls/certs/ca-bundle.crt" because with Redhat/CentOs at least that's the default location of the "official" public key list. In fact, we run our own private CA and our graylog-server uses a server cert signed by it - and out of habit we append a copy of our CA public key onto /etc/pki/tls/certs/ca-bundle.crt. So when I changed tls.py to do the above, my script ran perfectly - validating the server cert just fine. If I was using a self-signed server cert, it would have errorred - as expected/desired

I wonder if the ssl package has some variable that references the default ca-bundle.crt file for each OS? I dunno - that would certainly be the best default to use

So I think tls.py needs to support the following

• enable/disable server cert validation (default: enable). Perhaps "validateCert=[01]"?
• enable/disable using client cert (default: disable - but should be usable from graylog-server-1.2+). I think that should be "clientCert" instead of "cert", because if it confuses you - it will also confuse 99.9% of others :-)
• allow override of ca_cert (default: OS specific)

Jason

[jhaar]

Oh, and further information that might be useful to you

CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED

These options are to do with how ssl.py validates the certificate information associated with the certificate if receives from the other end

So in pygelf.py it is to do with server cert validation, but if you were to use ssl.py to create (say) some TLS network service, then you'd use the same vars to validate the client cert. "Normal" web servers would default to the equivalent of CERT_NONE, and as a counter-example, we run Apache web servers that require the browser to have a client cert, so they'd use CERT_REQUIRED or CERT_OPTIONAL (that one is useful as it allows the transaction to complete - but you can then generate an nice HTML error page - compared with CERT_REQUIRED where the user would just get this horrible browser SSL error that means nothing to them)

[keeprocking]

Summing up the above:

• if user doesn't provide cert file, I take the system default cert bundle
• if user doesn't want to validate server cert, I set cert_reqs to CERT_NONE, else - to CERT_OPTIONAL

But now I don't quite understand what client cert is.

[jhaar]

certs are basically a wrapper around (say) the AES encryption component of SSL. They are "public key cryptography" and basically provide a VERY secure channel over which the client and server can negotiate a temporary pre-shared key/password (with a strong sense of privacy and trust) - which they then use for the AES encryption part. Typically with an expiry date of an hour or two - and before it expires they go through the whole public key bit again to negotiate the next pre-shared key ad infinitum

So for 99% of the use of TLS/SSL on the Internet today, it's all about the client (normally a web browser) with no client cert (called a "null" client cert) talking to a HTTPS server which does have a cert (the server cert). The server cert is signed by a CA (self-signed certs are a special case of signing a cert with the cert itself). So the client connects to the server and as the server is (99.9%) configured as the equivalent of CERT_NONE, it doesn't need to validate the client at all - whereas the web browser is set to CERT_REQUIRED and so validates that the CA that signed the server cert (it's part of the server cert) is a known CA to itself. If it's happy, it continues, if it isn't it errors

So all that's need for pygelf to work is for ca_certs to be defined. It doesn't need "cert=" because there's currently no support for client certs within graylog-server (but that doesn't mean it doesn't work - it's actually quite happy to see the client cert - but it's still set to CERT_NONE so doesn't care). However, next release there is support (ie graylog-server will allow CERT_REQUIRED), so you might as well add support for it because - well basically you already have :-)

As far as CERT_OPTIONAL goes - no - you definitely want CERT_REQUIRED when validating server certs (plus CERT_NONE for not validating of course). CERT_OPTIONAL is useful for "optionally" validating client certs - but that's graylog-server's job - not pygelf

Hopefully that makes sense?

[keeprocking]

allow override of ca_cert (default: OS specific)

There's no built-in way to get OS specific CA bundle location. At least, I haven't heard about it. On Debian it would be '/etc/ssl/certs/ca-ceritficates.crt', on CentOS - '/etc/pki/tls/certs/ca-bundle.crt', etc.

Check out the latest version. Now if you want to validate server cert, you have to provide path to CA bundle. If you don't want to validate, simply pass handler's host and port.
Think I'll add client cert support later.

...and I still have a feeling that this is not what you wanted to see.

[jhaar]

You do yourself a disservice - that's perfect!

I think that's as good as it can get. The fact that you can't easily default to CERT_REQUIRED due to the lack of a consistent ca-bundle path convention is entirely understandable. And dropping client cert support is also fine - almost no-one uses them (you just happened to come across me who does ;-)

I think you're good to go. I just tested the following and they all worked as expected

worked: handle = GelfTlsHandler(host=gelf_server, port=12202)

worked: handle = GelfTlsHandler(host=gelf_server, port=12202, validate=False, ca_certs="/tmp/not-our-CA.crt")

failed: handle = GelfTlsHandler(host=gelf_server, port=12202, validate=True, ca_certs="/tmp/not-our-CA.crt")

worked: handle = GelfTlsHandler(host=gelf_server, port=12202, validate=True, ca_certs="/tmp/our-CA.crt")

Thanks for that. I can now use pygelf over the Internet :-)

[keeprocking]

I'm glad I could help you. Thanks for explaining things about TLS. :)
TLS handler is now in master.

[et304383]

@keeprocking I hate to necro this, but I'd like to verify before creating a new issue if the GelfTlsHandler should let me send logs over https to a port proxying to Graylog on localhost.

Log -> https://graylogserver:22201 -> localhost:12201

[bsmithyman]

@et304383 The GelfTlsHandler speaks raw sockets over TLS; i.e., it's a secured version of GelfTcpHandler, not GelfHttpHandler. So, it doesn't speak HTTPS. Same security, different protocol.

However, it looks to me like you should be able to use GelfHttpHandler and specify the right HTTPS endpoint as long as your sockets module has the support compiled in.

[keeprocking]

@et304383 yep, @bsmithyman is right, GelfTlsHandler deals with raw TCP sockets. If I remember correctly, GelfHttpHandler should support HTTPS out of the box. If not - create an issue and I'll take a look.

[et304383]

@keeprocking I'm not seeing https support out of the box. I am trying to call like this:

logger.addHandler(GelfHttpHandler(host='https://graylog.example.com', port=22201))
...
logger.info("A test message")

Output:

Traceback (most recent call last):
  File "index.py", line 48, in <module>
    test()
  File "index.py", line 46, in test
    logger.info("A test message")
  File "/usr/lib/python2.7/logging/__init__.py", line 1167, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
    self.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
    self.callHandlers(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
    self.emit(record)
  File "/usr/local/lib/python2.7/dist-packages/pygelf/handlers.py", line 158, in emit
    connection.request('POST', self.path, data, self.headers)
  File "/usr/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 557, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

[keeprocking]

New issue added (#20), http handler will be taught to speak https soon.