ARC target size (arc_c) unnecessarily low #4869
Comments
|
@snajpa It looks like something broke with your application of one of the dnode limit patches. The arcstats are completely missing "p" through "data_size". I figured I'd better mention this before looking at the repo link you posted. |
|
@dweezil, it looks like something went wrong during copy/paste of the output :) I've updated the paste. |
|
One thing that stands out is the 'buffers' in /proc/meminfo always jumps up to insane levels that I've never seen on a healthy system. I have no idea, which buffers does that indicate, if it's not SLAB nor vmalloc'd memory, then what kind of kernel memory is that? |
|
@snajpa Can you tell the nbd to use O_DIRECT on zvol device? |
|
@tuxoko I think (and Google seems to agree) that the NBD server doesn't support O_DIRECT. Btw, this is happening even when using ZPL datasets, not just ZVOLs. |
|
@snajpa |
|
|
|
arc_c_max = 20GB: |
|
@snajpa Is your kernel the 2.6.32 version typically used on EL6 systems? ZFS probably thinks the system is low on memory and, therefore, is not increasing the ARC target size. Could you please show As to |
|
Yes, it's EL6 kernel. Here's a current example from a system with OpenVZ (node usecase: general purpose containers - web, mails, ...) |
behlendorf
added
the
Type: Performance
|
@snajpa Thanks for the fresh stats. The thing that still sticks out is the high value for |
|
@dweeezil Some of the systems where I see this aren't even NUMA, so I would guess that's not relevant, but I do have swappiness set to 0, and we don't use any swap. I can see kswapd hitting the CPU, but I have no idea how to trace why is it running. |
|
We've increased zfs_arc_min to 70GB on that one node, because it was unusable with just a few GBs cached, when the workload dataset is clearly larger. That seems to help, though there's now kswapd0/1 chewing up on something heavily (each one is using ~70-80% of a core). |
|
|
@kernelOfTruth I know +- why kswapd exists in the system, I have no idea though, how to say why the system thinks it's low on memory, when there's - according to the kernels own accounting - a lot of memory free. Also, is there any way how to make perf skip searching for symbols? I've got ~5k processes in the system, all of them are run in containers, but perf keeps searching for symbols locally - so I'd like to skip that, but haven't figured out how yet. Also it looks like it's impossible to launch perf even if I tried to wait for the search to fail through all of the different executables. Any ideas on how to debug this in a way that actually works would be extremely appreciated... Thanks! |
|
@snajpa A good start might be |
|
@dweeezil invalid or unsupported event: 'vmscan:*' https://gist.github.com/snajpa/c07187597bb9eb13779c6544861b43d5 |
|
@snajpa I was kind of afraid that the 2.6.32 kernel wouldn't have those trace points. In fact, after looking, I see its |
|
@dweeezil nothing which would have caught my eye, take a look: https://gist.github.com/snajpa/a8a02c207bec3b9b62c8b32b7118a99f |
|
Hey, any luck so far? I am seeing a similar issue in our environment in which our metadata size drops to an unacceptable level. |
|
@ireverent Are you also using a 2.6.32 EL kernel? |
|
No, we are using 4.4 |
|
I believe I am also seeing this. 128GB RAM, I have set zfs_arc_max to 113G. At some point, the system seems to be refusing to grow ARC at all. Snipped from arc_summary.pl: ARC Size: 70.81% 82.44 GiB I'm pretty sure this is wrong, as running arcstat.pl 600 shows lines like this: 11:23:54 68 21 30 21 30 0 0 3 48 82G 82G e.g. an ARC miss rate of about 70%, so one would think it would be filling ARC on these misses, no? There is no swapping going on at all: [root@nas1 ~]# free -m exporting and re-importing the pool seems to 'fix' this. This is v0.6.5.7-1 on CentOS 7, kernel 3.10.0-327.28.2. Is there any useful info I can provide? |
|
Interesting. I tried setting zfs_arc_max again, but nothing happened. I then tried setting to a value 1GB bigger, and voila: 11:50:45 66 20 31 20 31 0 0 4 52 82G 117G if I then reset it to the original max: 11:51:40 4 2 50 2 50 0 0 0 0 83G 116G that works as expected (e.g. it is 1GB smaller.) I suppose I can create a cron job that runs every hour to keep diddling it, although that could be tricky, as it doesn't seem to do anything if the size hasn't changed... |
|
@dswartz Could you please provide a couple samples of your arcstats a few seconds apart. |
|
I have it running with 60 second polling. I will tee the output into a file so I can see when it changes... |
|
I restarted with 10 second interval. I have seen instances where the target arc size was reduced (but not drastically.) In all 3 cases, it seemed to be triggered when the current arc size hit the target size. I will post an update when I have another event... |
|
How do I see arc_no_grow and arc_need_free? Neither is printed by arc_summary.pl... |
|
@dswartz |
|
Cool, thanks! |
|
Happened again. 79GB target ARC size now. Info you requested: 6 1 0x01 91 4368 1609989890 154836914265905 Apologies, but I don't know how to insert the command output and have everything line up correctly as the earlier poster did. |
You can add a line with "```" above it and one below: |
|
Umm, that didn't work. I ended up with one pile of output smooshed together :( |
|
|
|
Like the above I just posted... Unless I did something wrong? |
|
Well, in the comment you've deleted you started with a single "`" instead of three and ended it with a ~ instead of the three backticks. The second try was fine, but it's only the first line. Anyway, don't worry about it 😄. |
|
@dswartz Your ARC appears to have reached a stasis point with the ARC target likely growing and shrinking at the same rate. If you want to effectively give the ARC priority over other memory consumers, you can set |
|
Lnicola, thanks. I think I misread the 3 backticks as a single backtick in quotes :) |
|
Dweezil, thanks. This host is doing very little that would exert significant memory pressure. I just checked and it's now 53GB! I can't for the life of me, imagine what would have demanded 60GB. For now though, I'll just set zfs_arc_min to 100GB or some such... |
|
I've seen this happen on my own systems. echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag See if this helps. |
|
We've seen similar behavior on several systems. Our working set and available RAM aren't as large, but similarly we see ARC shrinking well below what would make sense. The constant ARC eviction when we perform git clones or status etc, really kills performance. For example, this system: Kernel is 4.8.2-1.el7.elrepo.x86_64 on CentOS7 (Kernel is from EL-Repo, not the default kernel of CentOS7) zfs_arc_min is 512MB and zfs_arc_max is 8GB. There's 16GB free, and yet it's trying to constantly shrink well below the max (which it could allocate and still have ~8GB left over). There's no deduplication going on, and no L2ARC to manage, but there is about 1.8TB of 2.6TB of the ZFS pool in use, mostly lots and lots of source code / images / etc for projects (lots of small files). I've just tentatively tried DeHackEd's suggestion to set hugepage enabled to never, and arc size appears to be (very slowly) growing at the moment, but I'm not sure what the trade offs are if this fixes it. |
|
I have just seen the same problem on a production system running zfs to host a web forum (with the zfs docker storage if it is of interest). I was seeing After deactivating Is there some scenario where the memory isn't being freed correctly which leads to the cache draining to the minimum size? |
|
I have a similar problem with a host that has 32GiB of ram and is a isci/nfs target. I've got ~28GiB dedicated to ZFS and the ARC target size keeps shrinking despite seemingly low pressure for ram and huge pages are disabled. Running 3.10.0-514.26.2.el7.x86_64 on CentOS 7 w/ zfs-0.7.0-1.el7_3.x86_64. Free output: zpool status arcstat.py 5 5 meminfo, vmstat, and arcstats: |
|
Same thing here on Linux 4.13.9 / ZoL 0.7.3 / 32GB RAM / single socket (no NUMA) / Disabled THPs. Currently 12GB used, no defined arc_min/max, ARC sits around 1.3GB (which gives unacceptable performance). |
|
Have any devs been able to reproduce this issue or do you still need more information? I'm running into this problem too. |
|
Thank you chrisrd, but I am using v0.7.6 on I did not see this problem until I upgraded from v0.6.5.9 on Like most (all?) of the users above, I too am using containers, but mine do not have memory limits. I am using LXC, but I see OpenVZ and Docker listed above. No idea if this bug is actually related to containers or if there is simply a good percentage of ZFS users using containers. A couple days ago I set minimum arc size to maximum arc size: waited for the arc to grow to max, then set it back to 0. Over the space of 10 minutes target dropped to just over 92% of max and stayed there for a day, but then began slowly climbing again and is now back to 100%. I will monitor target to see if it falls again and if so attempt to determine why, but this box does not normally experience memory pressure. |
@behlendorf any reason why this was closed? |
|
@mailinglists35 I was under the impression this has been resolved. If you're seeing behavior similar to this in 0.8.0-rc2 can you open a new issue with the relevant details. |
|
resolved in what commit? why open a new issue and throw away all valuable information from this one, since this issue was never solved and there are multiple users affected? |
|
Problem still exists in 0.7.12 on Linux version 3.10.0-957.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018 |
|
Can confirm the problem still exists on No obvious memory pressure(8GB free mem) but low ARC size and also super low L2ARC usage. Drops in both ARC and L2ARC have occured after the upgrade from 0.6.5 to 0.7.x, before both were used extensively, ARC and L2ARC were usually filled(even when I really can't see any reason why this happens and what's even worse imho is I can't really tell if it's even a problem. The system itself seems to perform similar to before. I can't conclusively say that that's good though, since the upgrade happened to get around the bug in 0.6.5 that would eat a lot of performance with |
After a system has been running for a while and the ARC rotated through quite bit of data, it happens from time to time, that the system keeps reporting lots (>60%) free memory, but arc_c keeps targeting numbers way below arc_c_max.
It has been happening across all the workloads I put on ZFS - our production nodes with OpenVZ containers with 256G RAM, where there can be over 70G free memory according to /proc/meminfo, and arc_c is bouncing around lower tens of GB, rarely hitting above 30 GB. Meanwhile, arc_c_max is set to 128 GB.
I also have a node doing network block device server (userspace zvol consumer), which exhibits the same behavior.
The common denominator is that we're using RHEL6 kernel (with and without OpenVZ containers patchset).
I've tested 0.6.5-stable, with and without @tuxoko's ABD patches and both with and without recent @dweeezil changes regarding ARC dnode metadata limit (you can see what we're currently running in production at https://github.com/vpsfreecz/zfs/commits/master).
I'll attach example meminfo and arcstats from the NBD server, which has by far the cleanest workload (only about 15 clients for the nbd server and nothing else).
The text was updated successfully, but these errors were encountered: