How to configure to work with elasticsearch

[eamonnfaherty]

Hey all,

Firstly, thanks for this library! I am hoping this will be the answer to my dreams!

I am trying to configure elasticsearch to report errors to my sentry instance.

I have editted my logging.yml to look like:

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

sentry:
type: net.kencochrane.sentry.SentryAppender
dsn: "http://XXX:YYY@mysentryserver.com/9"
threshold: ERROR

And I have added:

rootLogger: INFO, console, file, sentry

I have grabbed the following from the sonatype site

raven-4.0-20130509.125810-7.jar
raven-log4j-4.0-20130509.125829-7.jar
raven-logback-4.0-20130509.125843-7.jar

and have placed them in /usr/share/elasticsearch/lib

I am triggering an error:

[2013-05-16 10:39:40,987][DEBUG][action.bulk ] [Sweetface] [haystack][3] failed to execute bulk item (index) index
[skipped some data]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [attributename]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:506)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:450)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown property [hey]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)

And I still have no sentry events.

Can you please help me?

[ColinHebert]

From my understanding elastic search is doing its own configuration with a yaml file but actually uses log4j.

What you need is a dependency on raven-log4j, and use a configuration similar to what is described in the INSTALL.md file:

https://github.com/kencochrane/raven-java/blob/master/INSTALL.md#using-log4j

[ColinHebert]

By the way you should use maven to manage your dependencies rather than downloading the snapshot manually

[eamonnfaherty]

I installed elasticsearch as a deb file. I do not know how to use mvn with a deb file to be honest.

I have managed to add the jars and I think I have it working now.

The type in my post above was wrong. It should have been what it was in the docs: net.kencochrane.raven.log4j.SentryAppender

I have it up and it appears to be running -- when I run the app manually I get:

May 16, 2013 12:11:42 PM net.kencochrane.raven.DefaultRavenFactory createConnection
INFO: Using an HTTP connection to Sentry.

For anyone interested my conf was:

sentry:
type: net.kencochrane.raven.log4j.SentryAppender
dsn: "http://XXXX:YYY@myserver.com/9"

[ColinHebert]

Great, unless there is something else I'll close this issue.

[eamonnfaherty]

I have set my logging level to debug:

rootLogger: DEBUG, console, file, sentry

but I am not getting an event when the following occurs:

[2013-05-16 12:16:02,106][DEBUG][action.bulk ] [Jack of Hearts] [haystack][3] failed to execute bulk item (index) index {[haystack][modelresult][mymodel.10], source[SOME INVALID JSON]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [expertise_risk_types_id]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:506)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:450)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown property [id]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)
... 11 more

Should I receive an event from this? I thought so.

[eamonnfaherty]

Maybe closed a little early ;)

I still do not have events in sentry.

[ColinHebert]

That's odd, may I see your entire configuration for logging?

The fact that you have a message from raven ("using HTTP...") tells me that the logger is set up properly, so I suspect that there is something else wrong in the configuration.

[eamonnfaherty]

Here it is:

https://gist.github.com/eamonnfaherty/1024d06061ab942138a0

[eamonnfaherty]

I have checked the nginx logs and there hasn't been any sign of incoming requests.

[ColinHebert]

Okay, I'm really confused now. Do you have any other logs? Something that could come from raven?

[eamonnfaherty]

I have grep'd the logs for raven and sentry. I cannot find anything.

Could it be that the debug being raised within elasticsearch is not being escalated to the logger. Do you know of a way I can test the loger sends on an error?

[eamonnfaherty]

Did you mean do I have other sentry events? If so, yes I do.

[ColinHebert]

Other events in sentry sent by elastic-search?

[eamonnfaherty]

I have not seen any others. That's what I was asking. How can I send a test event using the library?

I tried disrupting my elasticsearch but nothing has sent an event yet but I am not sure if it should.

[ColinHebert]

Hum, I think I know what I overlooked. Did you add the dependencies of raven to the lib folder?

Raven depends on commons-codec-1.8 and jackson-core-2.2.0

[ColinHebert]

FWIW, I've tried it without the dependencies. It seems that elasticsearch hides the error that happens when a class isn't found (dependency missing), making it a bit complicated to diagnose.

I've also noticed other issues with the log4j appender which entered in a feedback loop while testing against elastic search. I've fixed that and a new SNAPSHOT of raven-4.x is available.

And now you will need to also add slf4j-api-1.7.5 as a dependency of raven.

[eamonnfaherty]

I have updated to the latest version and I am now getting this:

May 16, 2013 8:39:43 PM net.kencochrane.raven.connection.HttpConnection doSend
SEVERE: Client/server version mismatch: Unsupported protocol version (4)
java.io.IOException: Server returned HTTP response code: 400 for URL: http://myserver.com/api/9/store/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at net.kencochrane.raven.connection.HttpConnection.doSend(HttpConnection.java:105)
at net.kencochrane.raven.connection.AbstractConnection.send(AbstractConnection.java:78)
at net.kencochrane.raven.connection.AsyncConnection$EventSubmitter.run(AsyncConnection.java:139)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

[ColinHebert]

That's better. This means that you are using raven-4.x when your sentry instance uses another protocol (probably v3).

In that case you need to download raven-3.x (it is up to date)

[eamonnfaherty]

I am using Sentry 5.4.5. Surely that should be protocol 4?

[ColinHebert]

No, protocol 4 isn't available in a stable release yet.

[eamonnfaherty]

ahh thanks.

I have switched to version 3 and now when elasticsearch starts it does not output that it is using sentry/raven and the logging still doesn't work.

elasticsearch just halts in the terminal when it runs. The logs show it (elasticsearch) has started but the logging does not work and the terminal doesn't show that sentry is being connected to.

Are the dependencies still the same?

[eamonnfaherty]

I am using

raven-3.0-20130516.191912-9.jar
raven-log4j-3.0-20130516.191940-9.jar

and the deps:

jackson-core-2.2.2-20130512.040915-3.jar
commons-codec-1.8.jar

[ColinHebert]

as for raven-4.x:

• slf4j-api 1.7.5
• commons-codec 1.8
• jackson-core 2.2.0

If you want to get the logs from raven you should also throw slf4j-log4j12 1.7.5 in it.

[eamonnfaherty]

ha ha ha! Be careful what you wish for! I have about 1k events in my sentry instance!

Thank you for all your help. When I did not have slf4j-api 1.7.5 the logging did not work. Once I added it everything worked!

Thank you very much again!

[ColinHebert]

That being said, I'm not sure what is your use case, but most of the time sentry is nice to catch errors and follow the trends, but if you want to store the logs and be able to search through them, there are other (and more appropriate) solutions such as logstash/kibana.

[eamonnfaherty]

I want to report on an error occurring within my django site. It is when data is sent to elasticsearch that is invalid.

Within elasticsearch it is recorded as debug as it does not affect the running of elasticsearch. Unfortunatly the library receiving the response does not process the error back from the api in the way that I would have hoped. I am logging the event now with sentry as it is listening to elasticsearch thanks to your help. I will have a look through this other library now to see how can I contribute these changes.

Thanks again!!