ARC Duelling with OS X File Cache

[jawbroken]

The following report is based on the 1.3.1-RC5 code, but I don't see anything in the final 1.3.1 release that would obviously lead to different behaviour.

I believe I'm seeing degenerate ARC behaviour as a result of the OS X file cache (as seen in the Memory tab of Activity Monitor) duelling with ARC. As I understand it, both caches are supposed to opportunistically use free memory to cache disk reads, but as neither is aware of the other this results in unstable behaviour. One cache sees there is memory pressure and frees some space, and the other cache sees that it can expand into this new space, etc, leading to them both seesawing randomly back and forth. In my experience, eventually ARC loses and gets relegated to a few hundred megabytes, which makes the pool uselessly slow (slower, in fact, than even a cold boot with an empty ARC).

The only way I've discovered to keep this under control is to periodically clear the OS X file cache (with the purge command line tool). I suspect that a real solution would need make ARC aware of the file cache so it could distinguish between real memory pressure and an overeager file cache.

I attached a chart of the size and target size from arcstat.pl over a period of about 2 days showing the behaviour I'm seeing. The memory in use by applications was essentially constant throughout this time, and the majority of the 16GB of memory was in use by either ARC or the file cache. I haven't included the file cache size on here, as I'm not aware of a command line tool to display it, but monitoring Activity Monitor showed that it will basically be a flipped version of the ARC size. You can see the section at the end where the ARC size became so small that I had to reboot in order to make the pool usable again.

[ilovezfs]

1.3.1 provides a new daemon zconfigd that lets you set a persistent arc maximum now:
7aa79ef

The configuration file is /etc/zfs/zsysctl.conf

You will find an example file is installed: /etc/zfs/zsysctl.conf.example

This is described on the wiki here: https://openzfsonosx.org/wiki/Performance#Memory_Utilization

[jawbroken]

I've read that page, but perhaps I don't understand how it's relevant. I'm not trying to change my arc_max, I'm happy for it to be the 13.3GB shown here. I also don't believe I'm trying to set a higher arc_min, as I vaguely understand that only sets a minimum target size and, as the chart shows, the growing file cache makes the ARC size drop below even the arc_min target.

[ilovezfs]

You need to set a lower arc max for your system. That is how it is relevant.

[jawbroken]

Again, perhaps there's something I'm missing. How will a lower maximum ARC size stop the OS X file cache from growing continuously and forcing the ARC target size to drop? I'm not at all concerned by ARC using 13GB of memory, and everything performs fine when it does.

[ilovezfs]

If you're happy with how the system is behaving, then no need for you to change anything, of course. Have you or have you not checked to see whether setting a lower arc_max affects the behavior you believe you're seeing over time?

[rottegift]

Do ilovezfs's test, and if that still results in an ARC vanishing away to unusability, try setting kstat.zfs.darwin.tunable.zfs_arc_min in the same way. Report back with graphs, please.

[jawbroken]

I'm happy with how it's performing when ARC happens to be of a reasonable size (maybe 5GB to ARC max). I'm unhappy with how it's performing when the OS X file cache is currently winning the “battle” shown in the chart, and especially when it wins so decisively that the ARC size drops well below even ARC min. Do you know of some current system in Open ZFS on OS X by which these two caches, ARC and the OS X file cache, would ever come to a sensible equilibrium, rather than seesawing back and forth? I don't see how such a system could exist without one cache being aware of the other, as they're both trying to expand to fill the free space.

I have not checked that, because I'm having no problem with the maximum ARC size and so I struggle to see the point. I'm not sure if we're on the same page. The size also drops below ARC min so I have a similar conceptual issue there.

If I'm going to run another multiday test, perhaps we could all agree on an ARC min and ARC max to use up front, so I'm not stuck generating these charts for weeks. Does anyone know how to read the OS X file cache size from the command line, also, because that might help make it more persuasive to you.

[jawbroken]

Okay, the file cache size is the number of file-backed pages in vm_stat, by observation and what I found here.

[jawbroken]

Okay, wrote a script. I'll test it with arc_max=8000000000 and arc_min=4000000000 on the 1.3.1 final. See you in a few days.

[rottegift]

The test ilovezfs told you to try seems counterintuitive, but when you set kstat.zfs.darwin.tunable.zfs_arc_max, the kernel code adjusts several variables.

Try setting arc_max to 8GiB and run with only that tunable set for a while.

Use the arcstat.pl tool. Note the number of reads vs number of misses, and size and tsize and whether misses go way up as size goes down. (If you have a small miss rate and a small ARC, why would you want a huge ARC, when other memory consumers on your machine clearly want more memory? ARC normally only decreases in size in response to memory pressure notifications from the kernel.)

[ilovezfs]

Script? You should be using /etc/zfs/zsysctl.conf not a script.

[jawbroken]

Yeah, misses go way up and it can take tens of minutes to list the contents of a directory. So are you saying not to change arc_min? When the file cache outpressures ARC and the size of ARC plummets am I just going to have the run the multiday test again, adjusting min?

The script is to gather the data on the ARC size and File Cache size, to be clear. I don't think it really matters how I change arc_max, because I'm not going to reboot during the test.

[ilovezfs]

Ah, I see. Thanks for the clarification.

[brendonhumphrey]

@jawbroken

I am the bloke who did most of the allocator work, so should probably comment here.

The behavior you described is exactly as designed. If the OS is under memory pressure, we receive various indications from the kernel, and respond by releasing spare memory to the kernel, presumably to be utilized for whatever activity that has caused the memory pressure. The memory that we release is anything that is "spare". In ZFS the definition of spare memory is necessarily the ARC. When we release memory from the ARC, the normal ARC release mechanisms and priorities apply.

Similarly within the kernel, if the machine is under memory pressure the kernel itself will release "spare" memory for others to use. It is generally the case that the fscache is the first and largest source of spare memory, and so it will shrink away before more extreme measures come into play, such as swapping to disk.

I would expect (and have observed) the following behaviors, for a machine with sufficiently small memory ->
rsync hfs <-> zfs : the two fs caches reach approximate equilibrium
rsync zfs -> zfs : arc grows, hfs fs cache grows
rsync hfs -> hfs : hfs cache grows, zfs arc shrinks

In my view, the busiest filesystem wins, and that would seem to be natural justice and good design.

As to why this was not observable with O3X < 1.3.1, there are probably two reasons - 1) we never utilised more than 25% of the machine (nominally at least), 2) the memory pressure response in those code bases was present, but so week as to be ineffective - it was easy to drive the machine into a very bad state (hence the 25% restriction).

Previous contributors to this thread have indicated that the arc maximum and minimum sizes are tunable. This is definitely the case. I saw rottegifts comments about ARC max. You should listen to him, he probably has the most experience tuning o3x for extreme workloads. If arc max does not work for you, consider raising the minimum arc size, that should set a threshold below which we should not release memory from the arc. If that lower limit is not honoured, I would consider it to be a bug. The consequences of raising the minimum arc size would be that we will no longer respond to pressure events from applications once the arc reaches a minimum. The machine will have to find memory through other means including swapping.

EDIT: I would note we are expecting plenty of questions about the memory behavior with the -RCs and eventual 1.3.1 release. We held up the release to put a persistent tuning mechanism in place so that users could to tune to their tastes if the default behavior does not suit. The tunable parameters are essentially the same as what is available on Illumos.

Further tuning information is available here -> https://openzfsonosx.org/wiki/Performance

[jawbroken]

I agree with what you're saying in the case of rsync and heavy reading & writing in general. Two file caches, even unaware of each other, are likely to behave reasonably in those cases and find some equilibrium. The load on my filesystems is more a steady stream of low volume reading and writing from HFS+ and ZFS, and I wonder if they will reach a reasonable equilibrium in this case, because that doesn't seem to be what I was seeing with RC5. Naïvely I would expect stability problems finding an equilibrium, such as you see when you have multiple uncoordinated control systems, and I seem to be seeing something like that. This is probably very sensitive to access patterns and volume, etc, so I don't know what a good testing approach would be.

If arc max does not work for you, consider raising the minimum arc size, that should set a threshold below which we should not release memory from the arc. If that lower limit is not honoured, I would consider it to be a bug. The consequences of raising the minimum arc size would be that we will no longer respond to pressure events from applications once the arc reaches a minimum.

The chart I posted shows the lower limit not being honoured on the right. At the time, the OS X file cache had grown to ~9GB and ARC was useless. Not sure if there are any fixes since RC5 that would impact this behaviour. If I could raise ARC min and it would be honoured then it would be a workable solution to my particular issue at this time.

Thanks, all.

[jawbroken]

Okay, took 8 days but here's a new trace showing the same issue with an 8GB ARC max. As you can see, the same issue still occurs on the final 1.3.1 release (previous chart was from RC5), and tweaking ARC max made no difference. ARC size is currently at 133MB, with an ARC min of 1744M. I've just purged the OS X file cache, leaving about 7GB of free memory, to see if the ARC will recover. Can we reopen this bug now?

[jawbroken]

And here's what happened when I cleared the OS X file cache manually (using purge).

[jawbroken]

And here's a significant undershoot of ARC min at 4GB. Might be behaving a little better with a raised minimum; looks it might be recovering when it gets significantly lower than the minimum, but it's impossible to tell as the two caches just wander around at random.

[lundman]

It is hard to tell what is going on but the ARC and kmem code is not finished, and we will continue to look at it. It is simply the best we could do for now, with lots of room for tuning.

There are even some new ARC patches upstream that we will work on adding. Feel free to research it and see if you can find a pattern to the problem.

[jawbroken]

Sure, I understand it's a work in progress. I think it might be helpful to have an open issue describing the problem and with the workaround of manually purging the file cache in it, but you might have a different opinion. I don't think I'm the only person seeing this, as this thread describes similar symptoms.

[emory]

(Possibly irrelevant sidebar: When discussing rsync it may be helpful to indicate which rsync you're using because the one Apple ships behaves differently than the one you can get via homebrew et al. Not only in terms of HFS+ metadata/spotlightstuff, but how it will behave when traversing directories.

When using rsync from homebrew (may have to tap a duplicate keg for it because it's often incorrectly considered a dupe by someone that doesn't know better), rsync will start working much faster and will keep finding files and directories as it goes, whereas the Apple rsync behavior will be to descend into every directory and appears to gather up every file before doing any operations — most of the time when I'm using rsync I use rsync -avzP $src $dst and it's possible I'm Doing It Wrong™ but anecdotally not using Apple's rsync binary means the operations start faster and are less greedy/thrashy.)

[jawbroken]

Just noting that my ARC has currently dropped to 88MB (well below the set minimum), and is struggling to recover to a reasonable size even after purging the OS X file cache. In the absence of any other suggestions, I think I will write a script that runs “purge” every 15m or so to hopefully keep the two caches from clashing like this. HFS performance is fine even with a purged cache, while ZFS performance is interminably slow when this happens. It is perhaps worth noting that it seems far worse when in this state than after a cold boot when ARC is empty. Maybe the small size is causing a lot of thrashing to and from disk.

[jawbroken]

This seems much better behaved for me in 1.6.1. It's been a week since I updated and I haven't been running my script that purges the macOS cache constantly and the ARC size is around 7GB. I'm closing this issue for now, as it is hopefully resolved. Thanks for your help.