Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too much data being sent to cloudwatch logs #396

Closed
ankurd1 opened this issue Mar 16, 2018 · 5 comments
Closed

Too much data being sent to cloudwatch logs #396

ankurd1 opened this issue Mar 16, 2018 · 5 comments

Comments

@ankurd1
Copy link

ankurd1 commented Mar 16, 2018

We recently upgraded to Elastic-stack 2.3.5 and seems like the amount of data being sent to cloudwatch logs drastically increased with this update. We run 30ish machines (4 agents per machine) and we have accumulated a few gigs of logs over the last few days.

We never check these logs (didn't even know they existed). Is there a way to opt-out of this logging? I would like to avoid paying cloudwatch charges.

@lox
Copy link
Contributor

lox commented Mar 16, 2018

That's no good! What's in the logs? Perhaps we can filter it down.

@ankurd1
Copy link
Author

ankurd1 commented Mar 16, 2018

The biggest ones are:

  • /buildkite/docker-daemon and /var/log/docker: Both contain logs at level=debug and lots of iptables stuff like time="2018-03-16T23:16:17.075813447Z" level=debug msg="/sbin/iptables, [-t filter -C DOCKER-ISOLATION -i br-bd5efbc42f31 -o br-a383a705445a -j DROP]"
  • The buildkite lambda functions are also logging a significant amount of data. Stuff like 2018-03-16T23:52:03.986Z 06d2ca04-2975-11e8-a95c-bd66763be9bf Skipping agent, doesn't match queue filter deploy

We'd prefer if there was a config option in the stack template to turn down the log level or select which one of these we want or maybe even a wholesale opt-out for cloudwatch logs.

@lox
Copy link
Contributor

lox commented Mar 18, 2018

Thanks, odd that looks like docker debug logs. I'll check ours and see if we have the same and what costs ours incurs. I think a way to disable these logs would make sense.

@ankurd1
Copy link
Author

ankurd1 commented Mar 19, 2018

Thank you!

@lox
Copy link
Contributor

lox commented Jul 28, 2019

Doesn't look like we're sending docker debug logs, I might close this. Let me know if you are still seeing this.

@lox lox closed this as completed Jul 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants