Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Receiving an SSLHandshakeException and nothing being logged #264

Closed
daniel-harrison-cko opened this issue Apr 8, 2019 · 8 comments

Comments

@daniel-harrison-cko
Copy link

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

elk_1  | [2019-04-08T13:45:45,744][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
elk_1  | io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:1000009c:SSL routines:OPENSSL_internal:HTTP_REQUEST
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
elk_1  | Caused by: javax.net.ssl.SSLHandshakeException: error:1000009c:SSL routines:OPENSSL_internal:HTTP_REQUEST
elk_1  |        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:897) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1147) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1101) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1169) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1212) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1211) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]```
@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

version: '3'
services:
  elk:
    image: sebp/elk:latest
    ports:
      - "5601:5601"  # Kibana
      - "9200:9200"  # Elasticsearch REST API
      - "9300:9300"  # Elasticsearch cluster communication
      - "5044:5044" # Logstash
  seq:
    image: datalust/seq:latest
    environment:
      ACCEPT_EULA: "Y"
    ports:
      - "5341:80"
@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

Host machine is running Windows 10. No data is coming through to Kibana, I guess due to this SSLHandeshakeException that occurs every minute or so?

@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

So I managed to get rid of the error message by disabling SSL for logstash. To do this I added the following file to my project:

beats-input.conf

input {
  beats {
    port => 5044
    ssl => false
    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
  }
}

And added a volume to my docker-compose.yml:

volumes:
      - './beats-input.conf:/etc/logstash/conf.d/02-beats-input.conf'

But I still don't see anything coming through into Kibana.

@daniel-harrison-cko daniel-harrison-cko changed the title Receiving an SSLHandshakeException Receiving an SSLHandshakeException and nothing being logged Apr 8, 2019
@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

Now I get a new exception:

elk_1  | [2019-04-08T14:36:18,370][INFO ][org.logstash.beats.BeatsHandler] [local: 172.22.0.3:5044, remote: 172.22.0.1:39416] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 79
elk_1  | [2019-04-08T14:36:18,382][INFO ][org.logstash.beats.BeatsHandler] [local: 172.22.0.3:5044, remote: 172.22.0.1:39416] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 83
elk_1  | [2019-04-08T14:36:18,384][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
elk_1  | io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 79
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
elk_1  | Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 79
elk_1  |        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
elk_1  |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 8, 2019

Where is the filebeat.yml configuration file in this image?

Another question, where is filebeat in this image?

How come I need to install filebeat manually according to the docs? Why doesn't the image have this working configuration by default? :S

@spujadas

This comment has been minimized.

Copy link
Owner

@spujadas spujadas commented Apr 8, 2019

All errors listed above are related to the fact that by default the image is set up to receive logs from Beats using SSL/TLS as per the documentation.

Where is the filebeat.yml configuration file in this image?

There isn't one, but everything needed to set it up is in the doc.

Another question, where is filebeat in this image?

Nowhere (see below).

How come I need to install filebeat manually according to the docs? Why doesn't the image have this working configuration by default? :S

As Filebeat sits with the services that feed logs into ELK, it shouldn't be in the base image (neither should any other log-emitting components btw). You are however welcome to create an extended image with Filebeat in it if your use case so requires.

Closing this, as everything is related to using ELK and Filebeat, not to an issue with the image.

@spujadas spujadas closed this Apr 8, 2019
@daniel-harrison-cko

This comment has been minimized.

Copy link
Author

@daniel-harrison-cko daniel-harrison-cko commented Apr 9, 2019

To anybody who finds this thread, here is how I solved my problem.

So my problem was that by default, the logstash configuration in this image accepts input from filebeat and not HTTP. The logging library I was using was trying to send logs via HTTP.

To enable logstash to accept logs over HTTP you will need to add a new input to the logstash configuration. You do this by mounting a new config file into the /etc/logstash/conf.d/ directory.

Example HTTP setup

I added a new file to my project called logstash-input.local.conf.

input {
  http {
    host => "0.0.0.0"
    port => 31311
  }
}

Then I added the following volume to my docker-compose.yml

volumes:
    - './logstash-input.local.conf:/etc/logstash/conf.d/03-input.conf'

After this, I was able to write logs in my project using Serilog .NET library and see them appearing in Kibana,

If you are new to ELK like me then a good article to understand logstash can be found in the logstash docs "How Logstash Works".

@spujadas

This comment has been minimized.

Copy link
Owner

@spujadas spujadas commented Apr 9, 2019

Thanks, will see if I can add these tips in the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.