New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance #8918
Comments
Is it worth benchmarking different blocksizes? e.g. |
It's a good thought. I did benchmark with larger block sizes, and the timings improve as you would expect but are still more than 2 orders of magnitude slower than they should be. For example, |
The code attached will automatically remount a CIRCUITPY device as synchronous, likely minimizing risk of data corruption #8449 and some of the perceived performance of delayed writes. It is human readable XML, uncommented. Install once, runs unattended, persists across Mac reboots and unlimited insertions/removals on any number of devices. This is a launch daemon property list configured to watch /Volumes for change events without polling. An embedded bash script examines all mounted CIRCUITPY devices on change. Devices mounted synchronous or read-only are not touched. Any other CIRCUITPY device is unmounted and mounted "synchronous,noasync" To use:
Monitor insertions using Apple's log utility:
|
The excellent workaround for #8449 by @sperok will work with Sonoma 14.3 and earlier. The reason that the workaround worked was that re-mounting the device caused the older MSDOS filesystem kext to be loaded. Unfortunately, Apple has changed mount behavior in Sonoma 14.4 beta 4 so that it continues to use the faulty user space MSDOS filesystem driver across a re-mount. |
bummer. wouldn't breaking "synchronous,noasync" behavior qualify as a bug? Can this be reported to Apple? |
@sperok Of course, but that's not exactly what's going on. Apropos #8449 mount flags are a red herring. I tested various combinations of flags on a remount, and found they had no effect. I later found that this was because re-mounting with any combination of flags caused Sonoma to pull in older kext code that worked properly vs. Sonoma's new user space driver. "Properly" in this context is that it did not delay metadata writes. We've been reporting the issue to Apple since last October. They responded to my report with "Potential fix identified - For a future OS update" and at about the same time changed behavior in Sonoma 14.4 beta like this:
So, Apple has resolved #8449 but has left us with a performance problem for which we have no workaround. Hopefully another round of problem reports to Apple will get the performance issue solved, too. Original Apple feedback for #8449 is FB13226668, feedback for this issue is FB13628505. |
Thanks for the detailed explanation. I've now gone back through #8449 and read the entire history, happy to learn "Tracing shows meta-data updates now follow a data write within a few hundred milliseconds." |
The default autoreload waiting time is 0.75 seconds after the most recent write. |
macOS Sonoma 14.4 beta 4 is out and still exhibits this behavior |
Hey @romkey, have you had a chance to try beta 14.4 5 yet? |
@UnexpectedMaker yes, just tested it in fact! Sadly no change, still slow but at least it also still works. Other Apple beta OS's are on their release candidates now but macOS 14.4 still isn't, so we may have a couple more betas before it's released. |
@romkey Thanks for checking. I'm not going to hold me breath about a fix coming in 14.4 ;) |
macOS 14.4RC out today, no change (if anything it feels even slower but I didn't actually time it) |
Ahh CR@P - Well, thanks for the update :) |
I was going to file a Feedback, and made up a USB stick with four partitions (16MB, 512MB, 1GB, 2GB) using gparted. Upon plugging this in to a Mac running 14.4 beta 5, one partition mounted quickly, one mounted slowly, and two never mounted after a couple of minutes. It looks like macOS was trying to set up the I will try for a simpler report about single small-sized partitions on USB sticks after some further experimentation. |
It's a bit of a pain, but you'll need to test with one partition at a time. |
The manual remount workaround still works for me (macOS 14.4RC). Doing this gets me fast writes to Using a Matrix Portal S3 here. Before testing I ran
ran it again after copying files and before manually remounting.
First mount line is from automatic mounting second is the manual remount - note the lack of
I haven't been able to find a |
I noticed this because I was trying to get a simple shell script working that demonstrates the problem using disk images, but I keep seeing fast writes with it. Not sure what's going on here as it appears to be using Its mount output looks like this When I manually copy the files it generates to an automatically mounted Here's the script:
|
The general public release of macOS 14.4 is out today. As expected, the slow writes are still an issue. The good news is that it still fixes the original issue and it's still possible to write to smaller filesystems without using any workarounds. So @dhalbert it's safe to update the blog post to tell people to update to 14.4. Hopefully 14.5 beta will be out soon and we can begin the testing/optimism/disappointment/surprise cycle anew. |
Yeah, I finally updated my M2Max Studio to Sonoma today because of the 14.4 release - been on Ventura since I bought it specifically because of this CP issue. |
macOS Sonoma 14.4.1 is out, bug fix release, no change in this behavior |
The (first?) macOS 14.5 beta is out. I upgraded, and realized I should have taken a baseline before upgrading. @romkey have you tried that beta yet? Thanks. |
This problem may also affect ExFAT. See this discussion at AppleInsider: https://forums.appleinsider.com/discussion/235967 |
@dhalbert sorry for the slow response, been deep in some other stuff. Comparing a 14.4.1 Mac with a 14.5 beta Mac, the 14.5 beta Mac is faster but still very slow (38 seconds to copy 24 files on 14.4.1, 18 seconds to copy the same files on 14.5 beta) |
CircuitPython version
Code/REPL
Behavior
Writes to the
CIRCUITPY
filesystem when mounted by USB MSC on MacOS Sonoma 14.4 beta 2 and above are 400 to 800 times slower than with previous releases of MacOS.Description
Previously, MacOS Sonoma 14.0 introduced a regression in its
MSDOS
filesystem support that caused a number of issues that are documented in issue #8449. The root cause of the issue was in Sonoma's newMSDOS
filesystem user-space support that replaced the earlier kernel extension support. The new user-space support delays writing filesystem metadata (in this case FAT table and root directory) for tens of seconds after writes to a file, causing a variety of undesirable outcomes.Apple subsequently fixed this behavior in 14.4 beta 2 by completely re-writing all metadata (the equivalent of a
sync
) after every few blocks written. This has the undesirable effects of slowing writes by 400 to 800 times and introducing unnecessary wear to the flash memory.It is possible to reproduce poor write performance using a USB stick. All of these tests were performed with a 16G stick. The test writes 512 blocks of 512 bytes of zeroes.
Performance jumps ~400x to 800x when the filesystem size exceeds 1G.
For convenience I made the test sticks on Linux. It should be possible to do the same with
Disk Utilities
on MacOS.Partitioned with
fdisk
creating a primary partition of the desired size, for example:Then create the filesystem:
Additional information
No response
The text was updated successfully, but these errors were encountered: