Wrong order on Papertrail, is it because of UDP protocol being used? #194

Closed
edpichler opened this Issue May 26, 2016 · 17 comments

Comments

Projects
None yet
4 participants
@edpichler

The logs appears in wrong order on Papertrail. Is this a limitation? Or it's o configuration problem?

Logs in wrong order are almost useless for erros investigation, unfortunately. Is it possible to logspout container show it in right order?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez May 26, 2016

Member

This seems to be a limitation of papertrail - at least according to this discussion entry. Not much we can do here if they throw out the timestamp in the syslog message.

Member

josegonzalez commented May 26, 2016

This seems to be a limitation of papertrail - at least according to this discussion entry. Not much we can do here if they throw out the timestamp in the syslog message.

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler May 26, 2016

Thank you @josegonzalez . So I will talk to them.

Thank you @josegonzalez . So I will talk to them.

@johlym

This comment has been minimized.

Show comment
Hide comment
@johlym

johlym May 27, 2016

Hi all,

Johnathan from Papertrail, here. @edpichler sent us an email and mentioned this issue so I wanted to add some insight.

The timestamp of when the log message was created isn't ultimately thrown out. The timestamp a user sees in PT is when we received the message. Original timestamps (when the message was created) are still viewable in the archives. Whatever order the messages are seen in Papertrail is the order we received them. We see a lot of out-of-order message receipts when users are transmitting via UDP at high volumes or the sender (be it their app, a syslog daemon, etc) is doing something funky.

I'm more than happy to answer any questions y'all have.

johlym commented May 27, 2016

Hi all,

Johnathan from Papertrail, here. @edpichler sent us an email and mentioned this issue so I wanted to add some insight.

The timestamp of when the log message was created isn't ultimately thrown out. The timestamp a user sees in PT is when we received the message. Original timestamps (when the message was created) are still viewable in the archives. Whatever order the messages are seen in Papertrail is the order we received them. We see a lot of out-of-order message receipts when users are transmitting via UDP at high volumes or the sender (be it their app, a syslog daemon, etc) is doing something funky.

I'm more than happy to answer any questions y'all have.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez May 27, 2016

Member

@johlym that makes sense on my end. Is there a way for the user to default to using the original timestamp of the log message? If not, thats fine - its by design, so shrug - but just curious so I know ho to answer these questions in the future.

Thanks for the clarification!

Member

josegonzalez commented May 27, 2016

@johlym that makes sense on my end. Is there a way for the user to default to using the original timestamp of the log message? If not, thats fine - its by design, so shrug - but just curious so I know ho to answer these questions in the future.

Thanks for the clarification!

@johlym

This comment has been minimized.

Show comment
Hide comment
@johlym

johlym May 27, 2016

Hey @josegonzalez,

Short answer is no but if they're really curious, their archived logs will have the original creation stamp as well as the receipt stamp with second resolution. For the vast majority of folks, the stamps will be the same.

johlym commented May 27, 2016

Hey @josegonzalez,

Short answer is no but if they're really curious, their archived logs will have the original creation stamp as well as the receipt stamp with second resolution. For the vast majority of folks, the stamps will be the same.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez May 27, 2016

Member

Gotcha, makes sense to me. UDP is UDP, can't do much about it.

Member

josegonzalez commented May 27, 2016

Gotcha, makes sense to me. UDP is UDP, can't do much about it.

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler May 31, 2016

So @josegonzalez the problem is because UDP?

I just configured my app to use the Logback framework, directly, without this giderlab/logspout, and Papertrail is showing correctly.

all_systems_ _papertrail

I conclude that the giderlab/logspout docker image has a limitation (or a bug), so my issue should be considered, no? This is a blocker issue, unfortunately I can't continue to use this image, I will migrate.

Thank you @johlym for your support.

So @josegonzalez the problem is because UDP?

I just configured my app to use the Logback framework, directly, without this giderlab/logspout, and Papertrail is showing correctly.

all_systems_ _papertrail

I conclude that the giderlab/logspout docker image has a limitation (or a bug), so my issue should be considered, no? This is a blocker issue, unfortunately I can't continue to use this image, I will migrate.

Thank you @johlym for your support.

@johlym

This comment has been minimized.

Show comment
Hide comment
@johlym

johlym Jun 1, 2016

@edpichler What was the transmission medium? UDP? TCP?

johlym commented Jun 1, 2016

@edpichler What was the transmission medium? UDP? TCP?

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler Jun 1, 2016

@johlym I don't know. May @josegonzalez can answer.

@johlym I don't know. May @josegonzalez can answer.

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 1, 2016

Member

The example that ships to papertrail uses the Syslog adapter, which sends UDP messages.

Member

josegonzalez commented Jun 1, 2016

The example that ships to papertrail uses the Syslog adapter, which sends UDP messages.

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler Jun 2, 2016

@josegonzalez my issue will not be considered as a limitation to be added on roadmap?

@josegonzalez my issue will not be considered as a limitation to be added on roadmap?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 2, 2016

Member

@edpichler The only "fix" here is for you to use a different adapter. The logback framework ships via TCP I believe, hence why you don't have the same "bug". It's a limitation of the protocol, not of logspout, and I don't think I can fix that.

Member

josegonzalez commented Jun 2, 2016

@edpichler The only "fix" here is for you to use a different adapter. The logback framework ships via TCP I believe, hence why you don't have the same "bug". It's a limitation of the protocol, not of logspout, and I don't think I can fix that.

@mattatcha

This comment has been minimized.

Show comment
Hide comment
@mattatcha

mattatcha Jun 2, 2016

Member

Wouldn't this be as simple as using syslog+tcp:// instead of syslog://?

Member

mattatcha commented Jun 2, 2016

Wouldn't this be as simple as using syslog+tcp:// instead of syslog://?

@josegonzalez

This comment has been minimized.

Show comment
Hide comment
@josegonzalez

josegonzalez Jun 2, 2016

Member

Yeah that's what I'm getting at.

Member

josegonzalez commented Jun 2, 2016

Yeah that's what I'm getting at.

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler Jun 3, 2016

@mattaitchison I changed it like your example and logs stop working. Nothing is being send to papertrail.

@mattaitchison I changed it like your example and logs stop working. Nothing is being send to papertrail.

@mattatcha

This comment has been minimized.

Show comment
Hide comment
@mattatcha

mattatcha Jun 3, 2016

Member

@edpichler Do you have tcp enabled in papertrail?

Member

mattatcha commented Jun 3, 2016

@edpichler Do you have tcp enabled in papertrail?

@edpichler

This comment has been minimized.

Show comment
Hide comment
@edpichler

edpichler Jun 3, 2016

@mattaitchison yes. It's working with you?

@mattaitchison yes. It's working with you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment