Reduce default journal disk space size limit #17382
Comments
|
You can control journal size: |
I know it. But I am not talking about configuration but the defaults. |
|
The defaults are scaled by the available disk space, with an additional upper boundary enforced. I am not sure it's worth discussing whether 400M or 4G as a upper boundary are better. It's a bit of a bike shedding question no? |
poettering
changed the title
|
Might be better to describe the actual problem that the larger size/cap was causing such users? For me on a 128GB SSD, where only 20GB was assigned to root and difficult to resize partitions, I usually have to do house keeping with only 1-2GB at most usually available for the root partition. Often I manage to reclaim quite a bit of space from cleaning the journal down to <50MB. I could probably just setup a systemd timer to run the command, or if there's a way to have systemd respond to low disk space, trigger the same command to clean up the journal... although if some rogue process is spamming logs or anything else to use up disk space (legitimately or misbehaving), that could be an awkward event to rely on. I imagine this issue was raised with the intention that when available disk space is limited, it wouldn't be due to logs accumulating multiple GB. Rather than reduce the limit, it might be better to discard older logs to free up space? (not sure if that's good as a default however) Setting up a timer to automate cleaning the journal when disk space is low is probably the right way to go. |
|
I've been hitting this problem for years. I've stopped counting the number of times where Fedora desktops/laptops I maintain had "/" run out of disk space because journalctl has accumulated gigabytes of logs over a couple of months and is competing for space with the system-installed apps and the package manager's ability to apply updates or to system-upgrade (particularly problematic if you have GNOME Software + PackageKit downloading/caching updates for the currently installed and newest release). As it stands, I have to remember to set this on every system I maintain for friends and family, but I keep forgetting to set it after a fresh install, and I keep getting calls, "My computer is telling me the system disk is full, what do I do?!" There's a huge difference between, say, 100-200 megabytes max of system logs and letting it grow to 1, 2, 4 GB (resulting from "let's go up to 10% of the / partition size" or something, I presume). My systems have 10 to 15 GB at most for the / partition (used to be 7-10 GB, but that's hard to fit in these days, with PackageKit and localization files eating tons of space), especially if they're on a SSD, where every gigabyte still counts. I'm never going to wade through a gigabyte of logs, particularly on a workstation, this is never useful information to me because most incidents are about something that happened within the last few hours/days/weeks at most. And if the system were to generate over 200M of logs in a day, I'd assume it's a bug due to something spamming the system logs, and that the contents would therefore be garbage anyway. The only case where I could imagine someone wanting gigabytes of log is on some NASA space shuttle or a financial server where auditing is super important, but then they will probably have a storage device dedicated to it, or at least a huge / partition, and configured the setting accordingly because they are pros. This kind of setting is one of the reasons I do not feel comfortable letting a friend install Fedora on their desktop computer without awkwardly saying "wait, I need to tell you about a bucket list of optimizations you need to set on the commandline just to keep the system running" (which you then have to apologize for and it makes the Linux platform look bad). They shouldn't be required to have the technical knowledge and insight to think "oh yeah, right after installing I need to tell the logging system to limit itself to 100 megabytes or else I might not be able to apply system updates 6 months down the road". Even I consider myself a fairly technical person and I have to keep notes on this, and the most I am willing to do is to change |
|
Note that journald has a setting SystemKeepFree= (which defaults to 15% of the fs size), and it will leave that much empty. But of course, if you let journald fill the disk up so that only 15% are empty, and then use some other tool to fill the remaining 15% then journald won't lower its own use retroactively... In general, we ship defaults. On some systems they might be too large, on others too small. We pick something in the middle. and put upper and lower bounds on it, as well as a lower bound on what we leave free. I#d say this is about as robust as this can get. But of course, if those values don't work for you it doesn't mean they are something we totally definitely have to change, because that'll make other people unhappy. Picking the right values for this is ultimately bikeshedding. X likes 1G better than 2G, and Y prefers 3.5G, but Z says 100M is the most I can offer! From systemd upstream we can't possibly pick values that makes everyone happy. Hence we don't. I'd claim though that a Linux installation on a 10G disk is pretty much the exception though, and not what we should pick our defaults for. You can't even buy a disk of 10G size on amazon anymore... Now, i understand you artificially made they fs so small by partitioning it like that, but I'd argue that is kinda your own fault: if you understand computers well enough to pick your own partitioning scheme, you better know what you do, and should be able to tweak your disk use too, or deal with the consequences... hence, i am not particular sympathetic to changing anything on this. While you might be happier with smaller values I doubt everyone else feels like you. I certainly myself have no problem whatsoever with the 4G upper bound on my laptop. And that this is a contentious issue is pretty clear. For example on fedora-devel there was a discussion about defaulting to time-retention limits for the journal, i.e. deleting everything older than 3months by default. But already that created a heated discussion between those who don't want their logs "needlessly" deleted, and thos who think forgetting super old crap is a good thing. |
A lot of devices on which you can run, say, Fedora ship with 8GB or so for their default eMMC drive, so it's not unheard of to have small "hard disks" which can't get any bigger. Fedora Silverblue uses a separate We had similar discussions around zram swap sizings, and I think it's definitely worth having a service which can retroactively free up that space which we could enable by default on workstation installations. |
Sure, but 10% of 8G is 820M, which isn't a disaster I'd claim. |
journald actually used to do that, but @keszybz removed that because such behaviour sucks: it meant that if you do an accidental |
|
The default of 4 GB is a really bad value. It takes ages even with fast SSDs to open and scroll through the logs. What makes it worse, if you run journalctl without any arguments it prints the oldest logs first. If I then press the "End" key on the keyboard, it will read through all those 4 GB of logs to just jump to the end. This usually takes like 5 to 10 seconds where my terminal hangs/freezes. It should have much more sensible defaults that doesn't cause computers to freeze when you want to see some logs. |
For your info, eMMC is not like any other storage media, it only has around 5000 write cycles, wearing stats and average lifespan are often given on the whole capacity. Yet when filed with a Linux distro, only the remaining place left will be used for the wear leveling. Hence those "free blocks" will wear faster, especially during intensive "journaling". https://www.cnx-software.com/2019/08/16/wear-estimation-emmc-flash-memory/ https://mediawiki.compulab.com/w/index.php?title=Application_Notes:_eMMC_lifetime_optimization Regards. |
Is your feature request related to a problem? Please describe.
Journal log has a 4GB cap of storage space. But for almost all desktop users and most server users, 4GB is too much. Most people don't care about 6 month old journal logs. If they do, they should enlarge the limit manually. But the default cap shouldn't be this big.
Describe the solution you'd like
Change the journal log size cap to 200MB.
Describe alternatives you've considered
Change the journal log size cap to 400MB.
The text was updated successfully, but these errors were encountered: