New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to configure to work with elasticsearch #58

Closed
eamonnfaherty opened this Issue May 16, 2013 · 26 comments

Comments

Projects
None yet
2 participants
@eamonnfaherty

eamonnfaherty commented May 16, 2013

Hey all,

Firstly, thanks for this library! I am hoping this will be the answer to my dreams!

I am trying to configure elasticsearch to report errors to my sentry instance.

I have editted my logging.yml to look like:

appender:
console:
type: console
layout:
type: consolePattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_search_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_search_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

sentry:
type: net.kencochrane.sentry.SentryAppender
dsn: "http://XXX:YYY@mysentryserver.com/9"
threshold: ERROR

And I have added:

rootLogger: INFO, console, file, sentry

I have grabbed the following from the sonatype site

raven-4.0-20130509.125810-7.jar
raven-log4j-4.0-20130509.125829-7.jar
raven-logback-4.0-20130509.125843-7.jar

and have placed them in /usr/share/elasticsearch/lib

I am triggering an error:

[2013-05-16 10:39:40,987][DEBUG][action.bulk ] [Sweetface] [haystack][3] failed to execute bulk item (index) index
[skipped some data]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [attributename]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:506)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:450)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown property [hey]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)

And I still have no sentry events.

Can you please help me?

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

From my understanding elastic search is doing its own configuration with a yaml file but actually uses log4j.

What you need is a dependency on raven-log4j, and use a configuration similar to what is described in the INSTALL.md file:

https://github.com/kencochrane/raven-java/blob/master/INSTALL.md#using-log4j

Contributor

ColinHebert commented May 16, 2013

From my understanding elastic search is doing its own configuration with a yaml file but actually uses log4j.

What you need is a dependency on raven-log4j, and use a configuration similar to what is described in the INSTALL.md file:

https://github.com/kencochrane/raven-java/blob/master/INSTALL.md#using-log4j

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

By the way you should use maven to manage your dependencies rather than downloading the snapshot manually

Contributor

ColinHebert commented May 16, 2013

By the way you should use maven to manage your dependencies rather than downloading the snapshot manually

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I installed elasticsearch as a deb file. I do not know how to use mvn with a deb file to be honest.

I have managed to add the jars and I think I have it working now.

The type in my post above was wrong. It should have been what it was in the docs: net.kencochrane.raven.log4j.SentryAppender

I have it up and it appears to be running -- when I run the app manually I get:

May 16, 2013 12:11:42 PM net.kencochrane.raven.DefaultRavenFactory createConnection
INFO: Using an HTTP connection to Sentry.

For anyone interested my conf was:

sentry:
type: net.kencochrane.raven.log4j.SentryAppender
dsn: "http://XXXX:YYY@myserver.com/9"

eamonnfaherty commented May 16, 2013

I installed elasticsearch as a deb file. I do not know how to use mvn with a deb file to be honest.

I have managed to add the jars and I think I have it working now.

The type in my post above was wrong. It should have been what it was in the docs: net.kencochrane.raven.log4j.SentryAppender

I have it up and it appears to be running -- when I run the app manually I get:

May 16, 2013 12:11:42 PM net.kencochrane.raven.DefaultRavenFactory createConnection
INFO: Using an HTTP connection to Sentry.

For anyone interested my conf was:

sentry:
type: net.kencochrane.raven.log4j.SentryAppender
dsn: "http://XXXX:YYY@myserver.com/9"

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

Great, unless there is something else I'll close this issue.

Contributor

ColinHebert commented May 16, 2013

Great, unless there is something else I'll close this issue.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I have set my logging level to debug:

rootLogger: DEBUG, console, file, sentry

but I am not getting an event when the following occurs:

[2013-05-16 12:16:02,106][DEBUG][action.bulk ] [Jack of Hearts] [haystack][3] failed to execute bulk item (index) index {[haystack][modelresult][mymodel.10], source[SOME INVALID JSON]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [expertise_risk_types_id]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:506)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:450)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown property [id]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)
... 11 more

Should I receive an event from this? I thought so.

eamonnfaherty commented May 16, 2013

I have set my logging level to debug:

rootLogger: DEBUG, console, file, sentry

but I am not getting an event when the following occurs:

[2013-05-16 12:16:02,106][DEBUG][action.bulk ] [Jack of Hearts] [haystack][3] failed to execute bulk item (index) index {[haystack][modelresult][mymodel.10], source[SOME INVALID JSON]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [expertise_risk_types_id]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:396)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:515)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:457)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:506)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:450)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:326)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown property [id]
at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:385)
... 11 more

Should I receive an event from this? I thought so.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

Maybe closed a little early ;)

I still do not have events in sentry.

eamonnfaherty commented May 16, 2013

Maybe closed a little early ;)

I still do not have events in sentry.

@ColinHebert ColinHebert reopened this May 16, 2013

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

That's odd, may I see your entire configuration for logging?

The fact that you have a message from raven ("using HTTP...") tells me that the logger is set up properly, so I suspect that there is something else wrong in the configuration.

Contributor

ColinHebert commented May 16, 2013

That's odd, may I see your entire configuration for logging?

The fact that you have a message from raven ("using HTTP...") tells me that the logger is set up properly, so I suspect that there is something else wrong in the configuration.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I have checked the nginx logs and there hasn't been any sign of incoming requests.

eamonnfaherty commented May 16, 2013

I have checked the nginx logs and there hasn't been any sign of incoming requests.

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

Okay, I'm really confused now. Do you have any other logs? Something that could come from raven?

Contributor

ColinHebert commented May 16, 2013

Okay, I'm really confused now. Do you have any other logs? Something that could come from raven?

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I have grep'd the logs for raven and sentry. I cannot find anything.

Could it be that the debug being raised within elasticsearch is not being escalated to the logger. Do you know of a way I can test the loger sends on an error?

eamonnfaherty commented May 16, 2013

I have grep'd the logs for raven and sentry. I cannot find anything.

Could it be that the debug being raised within elasticsearch is not being escalated to the logger. Do you know of a way I can test the loger sends on an error?

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

Did you mean do I have other sentry events? If so, yes I do.

eamonnfaherty commented May 16, 2013

Did you mean do I have other sentry events? If so, yes I do.

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

Other events in sentry sent by elastic-search?

Contributor

ColinHebert commented May 16, 2013

Other events in sentry sent by elastic-search?

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I have not seen any others. That's what I was asking. How can I send a test event using the library?

I tried disrupting my elasticsearch but nothing has sent an event yet but I am not sure if it should.

eamonnfaherty commented May 16, 2013

I have not seen any others. That's what I was asking. How can I send a test event using the library?

I tried disrupting my elasticsearch but nothing has sent an event yet but I am not sure if it should.

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

Hum, I think I know what I overlooked. Did you add the dependencies of raven to the lib folder?

Raven depends on commons-codec-1.8 and jackson-core-2.2.0

Contributor

ColinHebert commented May 16, 2013

Hum, I think I know what I overlooked. Did you add the dependencies of raven to the lib folder?

Raven depends on commons-codec-1.8 and jackson-core-2.2.0

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

FWIW, I've tried it without the dependencies. It seems that elasticsearch hides the error that happens when a class isn't found (dependency missing), making it a bit complicated to diagnose.

I've also noticed other issues with the log4j appender which entered in a feedback loop while testing against elastic search. I've fixed that and a new SNAPSHOT of raven-4.x is available.

And now you will need to also add slf4j-api-1.7.5 as a dependency of raven.

Contributor

ColinHebert commented May 16, 2013

FWIW, I've tried it without the dependencies. It seems that elasticsearch hides the error that happens when a class isn't found (dependency missing), making it a bit complicated to diagnose.

I've also noticed other issues with the log4j appender which entered in a feedback loop while testing against elastic search. I've fixed that and a new SNAPSHOT of raven-4.x is available.

And now you will need to also add slf4j-api-1.7.5 as a dependency of raven.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I have updated to the latest version and I am now getting this:

May 16, 2013 8:39:43 PM net.kencochrane.raven.connection.HttpConnection doSend
SEVERE: Client/server version mismatch: Unsupported protocol version (4)
java.io.IOException: Server returned HTTP response code: 400 for URL: http://myserver.com/api/9/store/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at net.kencochrane.raven.connection.HttpConnection.doSend(HttpConnection.java:105)
at net.kencochrane.raven.connection.AbstractConnection.send(AbstractConnection.java:78)
at net.kencochrane.raven.connection.AsyncConnection$EventSubmitter.run(AsyncConnection.java:139)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

eamonnfaherty commented May 16, 2013

I have updated to the latest version and I am now getting this:

May 16, 2013 8:39:43 PM net.kencochrane.raven.connection.HttpConnection doSend
SEVERE: Client/server version mismatch: Unsupported protocol version (4)
java.io.IOException: Server returned HTTP response code: 400 for URL: http://myserver.com/api/9/store/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625)
at net.kencochrane.raven.connection.HttpConnection.doSend(HttpConnection.java:105)
at net.kencochrane.raven.connection.AbstractConnection.send(AbstractConnection.java:78)
at net.kencochrane.raven.connection.AsyncConnection$EventSubmitter.run(AsyncConnection.java:139)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

That's better. This means that you are using raven-4.x when your sentry instance uses another protocol (probably v3).

In that case you need to download raven-3.x (it is up to date)

Contributor

ColinHebert commented May 16, 2013

That's better. This means that you are using raven-4.x when your sentry instance uses another protocol (probably v3).

In that case you need to download raven-3.x (it is up to date)

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I am using Sentry 5.4.5. Surely that should be protocol 4?

eamonnfaherty commented May 16, 2013

I am using Sentry 5.4.5. Surely that should be protocol 4?

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

No, protocol 4 isn't available in a stable release yet.

Contributor

ColinHebert commented May 16, 2013

No, protocol 4 isn't available in a stable release yet.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

ahh thanks.

I have switched to version 3 and now when elasticsearch starts it does not output that it is using sentry/raven and the logging still doesn't work.

elasticsearch just halts in the terminal when it runs. The logs show it (elasticsearch) has started but the logging does not work and the terminal doesn't show that sentry is being connected to.

Are the dependencies still the same?

eamonnfaherty commented May 16, 2013

ahh thanks.

I have switched to version 3 and now when elasticsearch starts it does not output that it is using sentry/raven and the logging still doesn't work.

elasticsearch just halts in the terminal when it runs. The logs show it (elasticsearch) has started but the logging does not work and the terminal doesn't show that sentry is being connected to.

Are the dependencies still the same?

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I am using

raven-3.0-20130516.191912-9.jar
raven-log4j-3.0-20130516.191940-9.jar

and the deps:

jackson-core-2.2.2-20130512.040915-3.jar
commons-codec-1.8.jar

eamonnfaherty commented May 16, 2013

I am using

raven-3.0-20130516.191912-9.jar
raven-log4j-3.0-20130516.191940-9.jar

and the deps:

jackson-core-2.2.2-20130512.040915-3.jar
commons-codec-1.8.jar

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

as for raven-4.x:

  • slf4j-api 1.7.5
  • commons-codec 1.8
  • jackson-core 2.2.0

If you want to get the logs from raven you should also throw slf4j-log4j12 1.7.5 in it.

Contributor

ColinHebert commented May 16, 2013

as for raven-4.x:

  • slf4j-api 1.7.5
  • commons-codec 1.8
  • jackson-core 2.2.0

If you want to get the logs from raven you should also throw slf4j-log4j12 1.7.5 in it.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

ha ha ha! Be careful what you wish for! I have about 1k events in my sentry instance!

Thank you for all your help. When I did not have slf4j-api 1.7.5 the logging did not work. Once I added it everything worked!

Thank you very much again!

eamonnfaherty commented May 16, 2013

ha ha ha! Be careful what you wish for! I have about 1k events in my sentry instance!

Thank you for all your help. When I did not have slf4j-api 1.7.5 the logging did not work. Once I added it everything worked!

Thank you very much again!

@ColinHebert

This comment has been minimized.

Show comment
Hide comment
@ColinHebert

ColinHebert May 16, 2013

Contributor

That being said, I'm not sure what is your use case, but most of the time sentry is nice to catch errors and follow the trends, but if you want to store the logs and be able to search through them, there are other (and more appropriate) solutions such as logstash/kibana.

Contributor

ColinHebert commented May 16, 2013

That being said, I'm not sure what is your use case, but most of the time sentry is nice to catch errors and follow the trends, but if you want to store the logs and be able to search through them, there are other (and more appropriate) solutions such as logstash/kibana.

@eamonnfaherty

This comment has been minimized.

Show comment
Hide comment
@eamonnfaherty

eamonnfaherty May 16, 2013

I want to report on an error occurring within my django site. It is when data is sent to elasticsearch that is invalid.

Within elasticsearch it is recorded as debug as it does not affect the running of elasticsearch. Unfortunatly the library receiving the response does not process the error back from the api in the way that I would have hoped. I am logging the event now with sentry as it is listening to elasticsearch thanks to your help. I will have a look through this other library now to see how can I contribute these changes.

Thanks again!!

eamonnfaherty commented May 16, 2013

I want to report on an error occurring within my django site. It is when data is sent to elasticsearch that is invalid.

Within elasticsearch it is recorded as debug as it does not affect the running of elasticsearch. Unfortunatly the library receiving the response does not process the error back from the api in the way that I would have hoped. I am logging the event now with sentry as it is listening to elasticsearch thanks to your help. I will have a look through this other library now to see how can I contribute these changes.

Thanks again!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment