Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-journal uses too much CPU #5102

Closed
phemmer opened this issue Jan 18, 2017 · 13 comments

Comments

@phemmer
Copy link

commented Jan 18, 2017

Submission type

  • Bug report
  • Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

231

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

CentOS/7

In case of bug report: Expected behaviour you didn't see

Less than 10% CPU usage

In case of bug report: Unexpected behaviour you saw

100% CPU usage

In case of bug report: Steps to reproduce the problem

base64 /dev/urandom | pv -L 1000000 | logger -t flood

Additional info

Running syslog-ng, with logger feeding to it on a TCP port, I am able to feed 1MB/s of data and it uses about 9% CPU. With the above command, journald bottle necks at 100% CPU and around 450KB/s. So it's about 20x slower.

When I do an strace on the process, the vast majority of the time spent is stating, opening, reading, and closing the various files in /proc/$pid

There was a PR the author claimed would help with this, #2280, but it has been closed.

This is causing a lot of pain for us as we have very log-heavy servers, such as internet-facing web servers. Here the systemd-journald process is generally consuming about 30% CPU.

@mnaboka

This comment has been minimized.

Copy link

commented Feb 16, 2017

we are able to reproduce this issue on CoreOS 1068.9.0 with systemd 229. When application begins to write extensively the CPU usage goes up to 100%. strace shows most of the time is spent in read, clock_gettime and open calls. Is there any explanation why would writing logs result in such big processor usage? And more important, is there anything we can do to reduce the CPU usage? We tried to control the CPU time quota with cgroups and it seems to work, however are there any caveats?

@binarykitchen

This comment has been minimized.

Copy link

commented Jun 1, 2017

i have this problem on ubuntu 17.04 too. systemd-resolved taking 100% cpu and dnsmasq around 50%. too much for a desktop machine.

what's the official solution here?

@phemmer

This comment has been minimized.

Copy link
Author

commented Jun 1, 2017

@binarykitchen This issue is about journald, not resolved.

@nise

This comment has been minimized.

Copy link

commented Jul 19, 2017

I have the same problem on ubuntu 17.04!

@pabl0

This comment has been minimized.

Copy link

commented Aug 26, 2017

Have you tried disabling compression (Compress=no in journald.conf)?

@binarykitchen

This comment has been minimized.

Copy link

commented Aug 27, 2017

just now, thanks - see how it goes then

@vcaputo

This comment has been minimized.

Copy link
Member

commented Aug 27, 2017

#6392 should make a substantial difference with regard to journald CPU usage under continuous logging.

There's no tagged release containing the commit yet, but if you're up for building your own, it's in master.

@poettering

This comment has been minimized.

Copy link
Member

commented Aug 29, 2017

Indeed #6392 should substantially speed things up under log flood. I will close this bug now because of that. I am sure there's more room for optimization, but for that we need more info about specific bottlenecks, and suggestions where to improve things, and I don't think there's much value in keeping a vague "needs to be faster" issue issue open until then. Hence, please try the new code, and if you identify specific bottlenecks or have specific ideas for optimizations, please file new, separate bugs. Thank you!

@poettering poettering closed this Aug 29, 2017

@josh-mesos josh-mesos referenced this issue Jul 20, 2018

Merged

Improved install docs for 1.11 #1192

0 of 3 tasks complete
@ctpstudy

This comment has been minimized.

Copy link

commented Oct 6, 2018

Same problem on ubuntu 18.04

@FreDeluz

This comment has been minimized.

Copy link

commented Jan 8, 2019

Same problem on ubuntu 18.04. Has someone found the solution?

@keszybz

This comment has been minimized.

Copy link
Member

commented Jan 8, 2019

I get 1MB/s with about 50% cpu usage (current git, compiled with -O0). As @poettering said when closing this issue: please open new bugs about specific things that should be improved if there's some bottleneck that we can fix.

@sabrehagen

This comment has been minimized.

Copy link

commented Jan 17, 2019

Same problem Ubuntu 18.10.

@keszybz

This comment has been minimized.

Copy link
Member

commented Jan 17, 2019

Please read @poettering's comment above. Saying "Same problem on some old version" in the upstream bugtracker is not going to have any effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.