Bulk request to ES isnt working - Certificate issue

[gopi-nath]

Hi,
Whenever I start my application, Stagemonitor gets initialized properly, but then when it is supposed to send metrics, it fails.

Logs:
20 Dec 2016 02:12:15,034 WARN [stagemonitor-elasticsearch-reporter] HttpClient:105 - [] - [] - Error sending POST request to url http://localhost:9200/_bulk: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_71]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_71]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_71]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_71]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_71]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_71]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_71]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) ~[?:1.8.0_71]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[?:1.8.0_71]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536) ~[?:1.8.0_71]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[?:1.8.0_71]
at org.stagemonitor.core.util.HttpClient.send(HttpClient.java:96) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ElasticsearchReporter.reportMetrics(ElasticsearchReporter.java:73) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter.report(ScheduledMetrics2Reporter.java:45) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter$1.run(ScheduledMetrics2Reporter.java:101) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_71]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
20 Dec 2016 02:12:15,036 WARN [stagemonitor-elasticsearch-reporter] HttpClient:106 - [] - [] - Error handling error response for POST request to url http://localhost:9200/_bulk: No content to map due to end-of-input
at [Source: UNKNOWN; line: 1, column: 0]
com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
at [Source: UNKNOWN; line: 1, column: 0]
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270) ~[jackson-databind-2.8.4.jar:2.8.4]
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3838) ~[jackson-databind-2.8.4.jar:2.8.4]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3783) ~[jackson-databind-2.8.4.jar:2.8.4]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2321) ~[jackson-databind-2.8.4.jar:2.8.4]
at org.stagemonitor.core.elasticsearch.ElasticsearchClient$BulkErrorReportingResponseHandler.handleResponse(ElasticsearchClient.java:344) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.elasticsearch.ElasticsearchClient$BulkErrorReportingResponseHandler.handleResponse(ElasticsearchClient.java:335) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.util.HttpClient.send(HttpClient.java:103) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ElasticsearchReporter.reportMetrics(ElasticsearchReporter.java:73) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter.report(ScheduledMetrics2Reporter.java:45) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter$1.run(ScheduledMetrics2Reporter.java:101) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_71]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
20 Dec 2016 02:12:15,037 ERROR [stagemonitor-elasticsearch-reporter] ScheduledMetrics2Reporter:103 - [] - [] - RuntimeException thrown from #report. Exception was suppressed.
java.lang.NullPointerException
at org.stagemonitor.core.util.IOUtils.copy(IOUtils.java:24) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.util.IOUtils.readToBytes(IOUtils.java:84) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.util.HttpClient.send(HttpClient.java:108) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ElasticsearchReporter.reportMetrics(ElasticsearchReporter.java:73) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter.report(ScheduledMetrics2Reporter.java:45) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at org.stagemonitor.core.metrics.metrics2.ScheduledMetrics2Reporter$1.run(ScheduledMetrics2Reporter.java:101) ~[stagemonitor-core-0.29.1.jar:0.29.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_71]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_71]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]

Also had another doubt: where do I give credentials for ES as needed for my production ES.

Please advice.
Thanks

[felixbarny]

Seems to be a connection problem. Can you curl your es cluster? How are you securing your cluster?

[gopi-nath]

Yes. curl works fine.
gopinath@mu:~$ curl 'http://localhost:9200'
{
"name" : "Numinus",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.3.5",
"build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
"build_timestamp" : "2016-07-27T10:36:52Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}

Right now, in my localhost there is no security mechanism.

[gopi-nath]

I think this bulk posting issue is solved. Apparently I havent set my ES config properly. I had left with it the default config itself. But not sure how come the port no is 9300(See in the log below)
Then edited the elasticsearch.yml =>

#Set a custom port for HTTP:
http.port: 9200

Also since my localhost is only a single node cluster, i changed my stagemonitor.properties accordingly=>

stagemonitor.elasticsearch.numberOfReplicas=1

Found this after checking the ES logs:

[2016-12-20 17:18:12,735][WARN ][rest.suppressed ] path: /stagemonitor-requests-2016.12.20/requests, params: {index=stagemonitor-requests-2016.12.20, type=
requests}
RemoteTransportException[[Numinus][localhost/127.0.0.1:9300][indices:data/write/index[p]]]; nested: UnavailableShardsException[[stagemonitor-requests-2016.12.20][1]
Not enough active copies to meet write consistency of [QUORUM] (have 1, needed 3). Timeout: [1m], request: [index {[stagemonitor-requests-2016.12.20][requests][AVk
cDoze15w5NhiWk1xC], source[{"id":"8410d6c2-bea5-45bc-98ac-cd66685fe57b"

[gopi-nath]

I have another issue as mentioned earlier. Please look into this. #230

[felixbarny]

Try setting number of replicas to 0. As you only have 1 node you cant replicate your shards to other nodes.

[gopi-nath]

Hi,

I am not able to send data to es5.x

Following is my stagemonitor.properties
stagemonitor.applicationName=name
stagemonitor.instrument.include=com.evive.name
stagemonitor.init.async=true
stagemonitor.active=true
stagemonitor.reporting.interval.elasticsearch=1
stagemonitor.elasticsearch.numberOfReplicas=1
stagemonitor.elasticsearch.numberOfShards=100
stagemonitor.instrument.runtimeAttach=true
stagemonitor.elasticsearch.availabilityCheckPeriodSec=5
stagemonitor.elasticsearch.url=https://user:pass@data3:9202/
stagemonitor.web.collectPageLoadTimesPerRequest=true
stagemonitor.plugins.disabled=JdbcPlugin
stagemonitor.internal.monitoring=true

While starting the application, I get the following logs and then no logs regarding stagemonitor,(no errors or warns).

19 May 2017 10:25:41,815 INFO [main] Application:57 - [] - [] - Started Application in 28.556 seconds (JVM running for 87.871)
19 May 2017 10:25:59,925 INFO [stagemonitor-startup] Stagemonitor:98 - [] - [] - Measurement Session is initialized: [application=Admin] [instance=localhost] [host=mu]
19 May 2017 10:25:59,928 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin WebPlugin
19 May 2017 10:26:00,008 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin RequestMonitorPlugin
19 May 2017 10:26:00,051 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin CorePlugin
19 May 2017 10:26:00,082 INFO [stagemonitor-startup] CorePlugin:557 - [] - [] - Not sending metrics to InfluxDB (url=null, interval=60s)
19 May 2017 10:26:00,083 INFO [stagemonitor-startup] CorePlugin:565 - [] - [] - Sending metrics to Elasticsearch ([https://user:pass@10.13.1.35:9202/]) every 1s
19 May 2017 10:26:00,152 INFO [stagemonitor-startup] Stagemonitor:142 - [] - [] - Initializing plugin JvmPlugin
19 May 2017 10:26:00,238 INFO [stagemonitor-startup] Stagemonitor:134 - [] - [] - Not initializing disabled plugin JdbcPlugin

When checked ES logs, found the following exception:

[2017-05-19T10:24:09,624][WARN ][c.e.p.a.t.Netty4TLSHttpServerTransport] [Stuart] caught exception while handling client http traffic, closing connection [id: 0xbc1ba37f, L:0.0.0.0/0.0.0.0:9202 ! R:/10.14.1.94:38200]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: javax.net.ssl.SSLHandshakeException: error:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:721) ~[?:?]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:954) ~[?:?]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:908) ~[?:?]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:985) ~[?:?]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1028) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1028) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:950) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]
... 15 more

How do i provide Client Certificate SSL support here?

[gopi-nath]

We resolved the certificate issue.

We made a proxy pass in apache. from some.domain.com to es.ip.address
Added the required certificates in the apache config.
Added it in the jre keystore.

[gopi-nath]

Closing this ticket