Skip to content
This repository has been archived by the owner on Apr 27, 2021. It is now read-only.

Improve logging. #104

Merged
merged 12 commits into from Nov 17, 2017
Merged

Improve logging. #104

merged 12 commits into from Nov 17, 2017

Conversation

olessia
Copy link

@olessia olessia commented Nov 7, 2017

  • Remove event_timestamp spammy logging.
  • Fix kafka.max.pull.hrs confusing message
  • Add a wrapper log class that echoes to stdout so Splunk picks up the logs
    See staging output in Splunk

Closes https://github.com/Shopify/camus-project/issues/11 with the caveat that only some hand-picked important Camus longs will appear in Splunk.

datwright and others added 7 commits October 27, 2017 10:47
…stamp that's malformed or missing.

This will significantly reduce our map tasks' log size.
This reverts commit 3b15ed7.
* Add deduplicator class.
Create a non-scheduled deduplication task: it will need to be updated manually in Azkaban with the desired params.
@dterror-zz
Copy link

adds dedup too, intentional?

@olessia olessia changed the title Remove more logging Improve logging. Nov 9, 2017
@dterror-zz
Copy link

So you're mapping debug -> info and error -> warn, intentional? We'll get all of the debug entries for these classes, are they not adding a lot of noise? You ran it in staging, so I'm half-assuming all answers will be all good, just asking anyway :)

@olessia
Copy link
Author

olessia commented Nov 10, 2017

Ugh, thanks for catching that. And I think I will remove the debug printout. When I tested in staging I expected the debug to be there, but for the wrong reason. Will update the PR

@olessia
Copy link
Author

olessia commented Nov 10, 2017

To be honest, this is not a pretty solution.
I'm going with what Hadoop book says:

Each task child process produces a logfile using log4j (called syslog), a file for data sent to standard out (stdout), and a file for standard error (stderr).

I've spent quite a bit of time trying to figure out how to redirect log4j, but to no result 😞
This is the next best thing from sprinkling printlns all over the place.

@dterror-zz
Copy link

I figured you'd done that. Next best thing is good enough.

@olessia olessia merged commit 6296106 into master Nov 17, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants