I/O error while formatting card

[bavariancake]

When formatting a Samsung Ultra 16GB SD card (the cheap kind floating in my drawer) I get I/O errors with Disk Utility every time:

dd has similar problems:

$ sudo gdd if=/dev/urandom of=/dev/rdisk2 bs=128k status=progress
gdd: error writing '/dev/rdisk2': Input/output error
1+0 records in
0+0 records out
0 bytes copied, 0.712663 s, 0.0 kB/s

This happens with the latest release as well.

Can you reproduce this issue as well, or do other cards work for you?

[cholonam]

What is your exact chip version? RTS525A or a different one? If RTS525A, I may be able to help you further if you send me the logs generated by the debug version of the kext. Also, it looks like it may be some problem with the card. Do other cards work? Does the card work on other OS?

[bavariancake]

So I've done a bit of testing, and this is where I am so far:

I have 4 SD cards:

• Sandisk Ultra 16GB
• Sandisk Extreme 16GB
• Sandisk Extreme Pro 64GB
• Sony 32GB MicroSD (low-end) via adapter

All of these cards were bought from an authorized retailer (not Amazon), so likely to be genuine cards. All of them encounter I/O errors when writing random data or zeros with dd in macOS. I tried the same in Linux, and all cards write without I/O errors, and the data written reads back correctly byte-for-byte, so it looks like the cards and the reader are ok (XPS 15 9570).

Output of lspci -v shows:

3c:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader (rev 01)
	Subsystem: Dell RTS525A PCI Express Card Reader
	Flags: bus master, fast devsel, latency 0, IRQ 143
	Memory at edb00000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [b0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [148] Device Serial Number 00-00-00-01-00-4c-e0-00
	Capabilities: [158] Latency Tolerance Reporting
	Capabilities: [160] L1 PM Substates
	Kernel driver in use: rtsx_pci
	Kernel modules: rtsx_pci

I'll collect debug logs tomorrow and get back to you.

[bavariancake]

Here are some logs I collected while writing to two cards in different ways. Let me know if something doesn't make sense or if there's something else I can test.

• Starting with initial boot, with boot arg -rtsx_mimic_linux, no card inserted:

rtsx-during-boot.txt

• Inserted an unformatted card (16GB Sandisk Ultra), clicked "Ignore", no error yet:

rtsx-insert-card-and-wait.txt

• Tried formatting card (16GB Sandisk Ultra) with Disk Utility -> Erase -> Mac OS Extended with GUID Partition Map:

rtsx-format-with-error.txt

In this case, Disk Utility gives this error:
POSIX reports: The operation couldn’t be completed. Input/output error. : (5)

There is also an I/O error in the log there.

• Tried writing zeros to another card (16GB Sandisk Extreme):

$ sudo gdd if=/dev/zero of=/dev/rdisk2 bs=128k status=progress
3179413504 bytes (3.2 GB, 3.0 GiB) copied, 497 s, 6.4 MB/s 
gdd: error writing '/dev/rdisk2': Input/output error
24259+0 records in
24258+0 records out
3179544576 bytes (3.2 GB, 3.0 GiB) copied, 497.81 s, 6.4 MB/s

The logs for this are here, also with a couple of I/O errors:

rtsx-dd-write-zeros-error.txt

I had to truncate the log file in the middle to remove 98MB of repetitive logs, but I didn't see an error in the middle there, only at the end of the log.

[cholonam]

Thanks for the logs. I think you may be having this same problem, which you should be able to solve easily by setting the card reader as read/write in the BIOS (check this post).

If that's not the case (assuming you are using Linux on the same laptop, Linux should fail too, unless it somehow bypasses the BIOS setting), are you sure you are not passing the -rtsx_ro boot flag?

[bavariancake]

Thanks for the logs. I think you may be having this same problem, which you should be able to solve easily by setting the card reader as read/write in the BIOS (check this post).

If that's not the case (assuming you are using Linux on the same laptop, Linux should fail too, unless it somehow bypasses the BIOS setting), are you sure you are not passing the -rtsx_ro boot flag?

So you're thinking I've set the BIOS or the driver to read-only mode. I assure you I have not.

I can write to an entire card from a file in Linux, then read back the result, and it will match byte-for-byte. The output of gdd (GNU dd installed via brew) above shows I could write 3.0GiB of a 16GB card before I hit an I/O error. If the card/BIOS/driver was set to read-only, I wouldn't be able to write 3GiB, and have it fail only then, right? It would fail at the beginning.

I'm looking into this issue because I maintain a repo for an XPS 9570 (https://github.com/bavariancake/XPS9570-macOS), and one of the users created a PR for the SD card reader. I like to test things thoroughly, which is why I ended up here. To be frank, I took quite some time to test that each card works in Linux, and will fail consistently in macOS using this driver. If you're not interested in debugging the driver and reproducing issues on your end, that's fine, it's really no problem, I understand that it can suck to have random people lodge issues and complain. But please let me know if you're not interested in maintaining the driver, and I will pick it up from here.

Now, if you are interested to getting to the bottom of the issue (as am I), could you test what happens when you write all zeros to an SD card with dd as with the following command?

sudo gdd if=/dev/zero of=/dev/rdisk2 bs=128k status=progress

Does this actually work for you? With more than one SD card?

Thanks.

[cholonam]

So you're thinking I've set the BIOS or the driver to read-only mode. I assure you I have not.

I can write to an entire card from a file in Linux, then read back the result, and it will match byte-for-byte. The output of gdd (GNU dd installed via brew) above shows I could write 3.0GiB of a 16GB card before I hit an I/O error. If the card/BIOS/driver was set to read-only, I wouldn't be able to write 3GiB, and have it fail only then, right? It would fail at the beginning.

I apologize. I missed the part that you are actually able to write some data.

I'm looking into this issue because I maintain a repo for an XPS 9570 (https://github.com/bavariancake/XPS9570-macOS), and one of the users created a PR for the SD card reader. I like to test things thoroughly, which is why I ended up here. To be frank, I took quite some time to test that each card works in Linux, and will fail consistently in macOS using this driver. If you're not interested in debugging the driver and reproducing issues on your end, that's fine, it's really no problem, I understand that it can suck to have random people lodge issues and complain. But please let me know if you're not interested in maintaining the driver, and I will pick it up from here.

Now, if you are interested to getting to the bottom of the issue (as am I), could you test what happens when you write all zeros to an SD card with dd as with the following command?

sudo gdd if=/dev/zero of=/dev/rdisk2 bs=128k status=progress

Does this actually work for you? With more than one SD card?

Thanks.

My free time at this moment is very limited, so I'm afraid I won't be able to look into this for some time. By re-checking the logs, I see that the first (meaningful) error that you get is this:

2020-06-06 12:10:53.868338+0200 localhost kernel[0]: (kernel) rtsx: SDDisk::read_task_impl_ : Returning an IO Error! (WRITE block = 6210048 nblks = 256 blksize = 512 physSectSize = 512 error = 12)

Error 12 is ENOMEM, which may mean there is a memory leak somewhere. I will try to reproduce your case and fix it when I get some time, which may be in a few weeks or in a few months.

Thank you very much for the thorough testing.

[cholonam]

@bavariancake, I have issued a new beta release that hopefully will fix your problem. I have not been able to reproduce it neither with release 2.2 nor with the new beta release.

Since I believe this issue is due to DMA memory fragmentation, it should be more likely to happen on systems with low memory or after a long time of OS usage (whenever DMA region becomes fragmented). Just to confirm, can you please answer these two questions?

1. How much RAM does your system have?
2. Did the error happen right after a fresh boot? Did it happen after a long time without reboot?(I mean full reboot, not just sleep/wake.)

Please check the new beta release whenever you get some free time and let me know how it goes. It is posted here.

[bavariancake]

@bavariancake, I have issued a new beta release that hopefully will fix your problem. I have not been able to reproduce it neither with release 2.2 nor with the new beta release.

I still see errors formatting with Disk Utility, and using dd seems to error out after anywhere from 2-14MB written, whereas before I could write a few GB before encountering an error. I saw that there were a few responses on InsanelyMac also, but just to add my experience.

1. How much RAM does your system have?

32GB

2. Did the error happen right after a fresh boot? Did it happen after a long time without reboot?(I mean full reboot, not just sleep/wake.)

This happens 5 minutes after a cold boot, i.e. right after I added the new kext, shutdown, then booted.

Please check the new beta release whenever you get some free time and let me know how it goes. It is posted here.

I found the file in the releases section of the repo, since it seems to be removed from InsanelyMac. I assume they're the same.

[bavariancake]

Just wanted to mention that I know firsthand how difficult it can be to have people post issues but not resolve them. I've had my fair share of that maintaining my XPS 9570 repo, even though ultimately I'm glad I can help other people out. To that end, I'd like to offer a bug bounty of $200 to fix the I/O errors I reported in this issue. If someone provides an update to this repo that can pass my test with dd with the four SD cards I have, they'll get the bounty.

Maybe this is interesting to you @cholonam, just as a way to acknowledge the effort. If someone else wants to take this on, that's cool too.

[cholonam]

That's really weird.. are you getting error 12 (ENOMEM) again or other error? Error 60 (ETIMEDOUT) is not very meaningful since it's just a timeout meaning the card reader is not responding, probably because of bad error recovery from the OpenBSD driver.

Just to make sure, I assume you are passing the -rtsx_mimic_linux boot parameter, and preventing sleep during the copy process somehow (i.e. caffeinate command). Let me know if otherwise.

I was not able to reproduce your results neither before with a 1GB card, nor now with a 16GB card. Here is my test procedure:

0. Card used: SanDisk Ultra 16GB
1. Dump card contents to hard drive (sudo gdd if=/dev/rdisk4 of=$HOME/N16G.IMG bs=128k status=progress)
   → SUCCESS
2. Format card as exFAT
   → SUCCESS
3. Write random data to whole card (sudo gdd if=/dev/urandom of=/dev/rdisk4 bs=128k status=progress)
   → SUCCESS
   • At the very end (block ) I get an IO error, which is expected because gdd is trying to write above the card's capacity (this same error occurs when trying to write to a USB stick past its capacity).

gdd: error writing '/dev/rdisk4': Input/output error
121549+0 records in          <-- The card's capacity is 121548 records
121548+0 records out

4. Write image back to card
   → SUCCESS

% sudo gdd if=$HOME/N16G.IMG of=/dev/rdisk4 bs=128k status=progress
15924330496 bytes (16 GB, 15 GiB) copied, 1821 s, 8.7 MB/s 
121548+0 records in
121548+0 records out
15931539456 bytes (16 GB, 15 GiB) copied, 1821.85 s, 8.7 MB/s

Now, what's next? I appreciate the bounty, but since I am not able to reproduce the error, it will be almost impossible to solve. Hopefully somebody reads this and is willing to help.

You can send me the debug log again and let's see if I can find something. You should not be getting error 12 now, probably you are getting an IO error and the driver is not doing error recovery properly so it just stops responding (lots of error 60, timeouts). This IO errors may be due to the cards, or your hardware, but definitely it's not common (since I am not getting them). Do you know of any other users having these error or are you the only one? If you are the only one, I'm afraid I'll have to close the issue as "not able to reproduce" (unless I see something I can fix in the logs).

[bavariancake]

That's really weird.. are you getting error 12 (ENOMEM) again or other error? Error 60 (ETIMEDOUT) is not very meaningful since it's just a timeout meaning the card reader is not responding, probably because of bad error recovery from the OpenBSD driver.

I'll test again and see which error codes I'm getting.

Just to make sure, I assume you are passing the -rtsx_mimic_linux boot parameter, and preventing sleep during the copy process somehow (i.e. caffeinate command). Let me know if otherwise.

Yes, definitely passing -rtsx_mimic_linux boot argument. I have sleep disabled in Energy Saver and laptop is always plugged in.

• At the very end (block ) I get an IO error, which is expected because gdd is trying to write above the card's capacity (this same error occurs when trying to write to a USB stick past its capacity).

Yes, makes sense, since the input to dd is infinite. I agree we should be seeing an error there 😊

Now, what's next? I appreciate the bounty, but since I am not able to reproduce the error, it will be almost impossible to solve. Hopefully somebody reads this and is willing to help.

I'll do the same tests again and see what error codes I get, and I can verify that the driver gets the boot flag with a print statement. Which version should I be using? 2.2 or 2.3-beta? For some reason, I was getting I/O errors after writing just a few MB with 2.3-beta, but maybe the error codes are different. I'll get back to you.

[cholonam]

Which version should I be using? 2.2 or 2.3-beta? For some reason, I was getting I/O errors after writing just a few MB with 2.3-beta, but maybe the error codes are different. I'll get back to you.

Use 2.3-beta. With 2.2 you may get memory errors. 2.3-beta should NOT give error 12, so I'd like to see which error it's giving you.

[stemnic]

I can confirm that I'm having a similar issue with 2.3-beta-debug on my XPS 9570. I seem to be able to read/dump just fine but the problem arrives when you try to format or use dd to write random data to the card. I have in the past with 2.2 been able to read and write under "normal" use like copying/pasting an image without any obvious faults

I'm using a Kingston 8GB micro sd card with an adapter.

Ran the tests you mentioned in your last post @cholonam.

Bootup and dumping a FAT32 formatet card

stemnic@MBP ~ % sudo gdd if=/dev/rdisk2 of=$HOME/Desktop/K8GB.IMG bs=128k status=progress
7969701888 bytes (8.0 GB, 7.4 GiB) copied, 967 s, 8.2 MB/s
60832+0 records in
60832+0 records out
7973371904 bytes (8.0 GB, 7.4 GiB) copied, 967.446 s, 8.2 MB/s

Worked just fine

bootanddumpRevised.log Due to the size of the original file (200MB) I removed most of the logs from the dump since I did not see anything obvious other than raw read logs

Writing urandom to sdcard

stemnic@MBP ~ % sudo gdd if=/dev/urandom of=/dev/rdisk2 bs=128k status=progress
Password:
gdd: error writing '/dev/rdisk2': Input/output error
16+0 records in
15+0 records out
1966080 bytes (2.0 MB, 1.9 MiB) copied, 0.963012 s, 2.0 MB/s

urandom write.txt
Failed

Trying to format to ExFat

exFat format error.txt

Trying to format to ExFat - Using terminal

stemnic@MBP ~ % sudo diskutil eraseDisk exfat test disk2
Password:
Started erase on disk2
Unmounting disk
Creating the partition map
Error: 5: Input/output error

formatTerminallogs.txt

Running urandom write again after formating to exfat in windows

stemnic@MBP ~ % sudo gdd if=/dev/urandom of=/dev/rdisk2 bs=128k status=progress
Password:
2752512 bytes (2.8 MB, 2.6 MiB) copied, 1 s, 2.7 MB/s
gdd: error writing '/dev/rdisk2': Input/output error
25+0 records in
24+0 records out
3145728 bytes (3.1 MB, 3.0 MiB) copied, 1.42574 s, 2.2 MB/s

urandom write 2 after format.txt

Trying to write imaged dumped in the first test back

stemnic@MBP ~ % sudo gdd if=$HOME/Desktop/K8GB.IMG of=/dev/rdisk2 bs=128k status=progress
gdd: error writing '/dev/rdisk2': Input/output error
17+0 records in
16+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 1.08193 s, 1.9 MB/s

write image back.txt

For the sake of completeness, this is a list of the PCI devices with the sdcard reader on my XPS

Hope this helps!

Edit: I also have -rtsx_mimic_linux as an argument during boot

[stemnic]

After more experimenting this seems to be an intermittent issue. I tried with another sdcard a Lexar 1GB SD and initially had the exact same problems with the other card but suddenly it started working after a few remounts. Logs attached are writing urandom data and working not the faulty state.

logs lexar trimmed.txt
(Again removed the rtsx_xfer_adma data since it just made the file to big)

Tried the card I used earlier in my last post and got other errors than previosly.

2nd formaterror.txt

3rd formatError.txt

Afterwards I tried formatting the card in windows with no problems
Used H2testw todo a write and verify test on windows and the same card came out just fine
I used another windows machine todo this and never rebooted the xps itself

Going back to the lexar car again on the hackintosh worked just fine and not been able to reproduce any errors with it again.

[stemnic]

Also tested a transcend 8GB microSD with an adapter and it is not detected at all (works fine in linux and windows on the xps)
Don't know if it's related but thought it might be useful since this seems to depend on the cards.
transcend mount fail.txt

[cholonam]

@stemnic @bavariancake Thanks for the efforts. Please wait until I upload a new beta with longer timeouts to continue testing. It will only take me some hours.. thanks!

[cholonam]

@stemnic @bavariancake I have uploaded a new beta version which allows for longer timeouts, with the hope that what you are getting are true timeout errors (that would be the best case scenario).

Here are some instructions to test it when you get some time:

1. Make sure you are using these boot parameters: -rtsx_mimic_linux rtsx_timeout_shift=3 (this will increase timeouts times 2₃=8, which is a lot, and will make the card take a longer time to show on the desktop on attach, be patient). Note that this parameter does NOT start with a dash.
2. Make sure you are using the debug version of the kext.
3. Do the format or writing test and send me the logs (make sure you pass --debug --info to the log command. This will generate a lot of logs, I know. You can erase them after with sudo log erase -all.
4. Send me the logs. You can trim them as you have been doing.

I have to say that it's likely that what you are getting are not really timeouts but just the card acting weird or just not responding. I did a test with this beta and I get very low timeouts. For example, for the STOP_TRANSMISSION command, which is the one giving the timeouts in @stemnic 's logs, the longest timeout I am getting is 7.3 ms (writing urandom to a 1GB TOSHIBA SD card). The kext is waiting up to 100 ms, but the original BSD code was waiting up to 1 sec. and I think Linux may be waiting up to 300 ms (not 100% sure though). With the rtsx_timeout_shift parameter you'll be waiting 800 ms, which should be more than enough.. However this driver is far from being complete. There are lots of corner cases it just does not handle, that's why some cards will work on Linux/Windows and not with this kext. If longer timeouts don't solve the issue, it's very likely these cards will never be compatible with this driver.

[stemnic]

Thanks for the update, seems like the card which was not detactable before now shows up. However I still have problems formating the kingston and trancent microSD cards. The Lexar as mentioned before apperares to work just fine.

transcentFormatError.log

kingstonFromatError.log

[cholonam]

Thanks for the update, seems like the card which was not detactable before now shows up. However I still have problems formating the kingston and trancent microSD cards. The Lexar as mentioned before apperares to work just fine.

• transcentFormatError.log I cannot see any error in this log. It looks like the card is just working perfectly.
• kingstonFromatError.log The card gives an IO error on the SWITCH command during initialization (line 3562), which is weird, but it looks that it may happen with some cards (both the Linux and OpenBSD code just ignore this error). After that, the card gives several IO errors again on several READ_BLOCK_SINGLE commands (from line 3652), and after it gives an IO error on a READ_BLOCK_MULTIPLE, it starts giving timeouts (just not responding).

So it seems increasing the timeouts does not really help. What I can do is add some retry logic to the code, to see if that helps with IO errors. And for the timeouts after READ_BLOCK_MULTIPLE, I'll change the code a bit, because I see the Linux code sends a STOP_TRANSMISSION command even if READ_BLOCK_MULTIPLE multiple fails, but the OpenBSD driver (on which this kext is based) does not. It may be that the card needs the "STOP_TRANSMISSION" to go back to a "waiting for commands" state.

I'll need one or two weekends to implement those changes, though, and I do not want to create false expectations: The fact that I can't reproduce these errors just makes it harder to fix, and I'll definitely be depending on your help for testing. You have to understand that despite all the testing, it may happen that these cards will never work with this driver. There are just too many corner cases that the Linux driver takes into account and this driver doesn't, so the most likely is that this driver only works with the "most well-behaved" cards. Let's hope for the best after I add the retry logic and STOP_TRANSMISSION logic.

[cholonam]

@stemnic @bavariancake I have just uploaded a new release with the changes mentioned above. I hope these changes can solve the problems you are having. Please test it when you find some time and let me know the results.

[bavariancake]

@stemnic @bavariancake I have just uploaded a new release with the changes mentioned above. I hope these changes can solve the problems you are having. Please test it when you find some time and let me know the results.

Thanks for the update! Started testing yesterday, will get back to you when finished.

[bavariancake]

OK, so after a lot of testing, here's what I found.

Tests performed

I started off with four SD cards I had flying around in my drawer (see above), and repeatedly did the following tests:

• Write random data to the entire card, with block sizes of 128k, but also 4k, 8k, 16k, and 32k.
  sudo gdd if=/dev/urandom of=/dev/rdisk2 bs=128k status=progress

• Write zeros to the entire card, also with block sizes of 4k, 8k, 16k, 32k, and 128k.
  sudo gdd if=/dev/zero of=/dev/rdisk2 bs=128k status=progress

• Read from the entire card and discard the result.
  sudo gdd if=/dev/rdisk2 of=/dev/null bs=128k status=progress

• Use the card as one normally would, by formatting HFS with Disk Utility, then copying a large number of small files to the card, then using diff -r to compare the source and written data.

I've done each test across many reboots/cold boots, to rule out any kext loading issue on a particular boot or something similar.

Test results

Each card tested would either fail consistently, or would pass every test. The main difference between cards seems to be the age of the card, i.e. older and slower cards would fail consistently, and newer or more high-end cards would never fail.

For example, an old Sandisk Ultra 16GB would always fail, but a Sandisk Extreme 16GB of the same vintage would not. Newer or faster cards, like the Sandisk Extreme Pro 64GB would read/write at a higher speed than before as well, around 13MB/s, which was an unexpected benefit. Looking at a reputable OEM retailer like B&H, I found that I couldn't even buy the older/slower cards anymore - even the cheapest Sandisk Ultra card feature 32GB with a UHS-I rating.

While it would be nice if slow/old cards worked perfectly, it's not a priority since most new cards on the market would be fast enough. I suspect this points to a bug handling a busy state with slow cards, and I could mail you those old/slow cards if you're interested @cholonam.

There are two more issues I found:

• SD card slot becomes unusable if card is withdrawn during write.
  This requires a reboot - nothing is recognized after a card is pulled during a write op.

• Occasionally, an SD card would not be readable when inserted. kextstat will show that the com.sinet3k.Sinetek-rtsx kext is running, but no card will show up. The system must be rebooted.

Only the last issue is somewhat of a problem - rebooting just to use the SD card slot is a bit of a hassle. The first problem can be ignored for now, i.e. tell the user not to do that 😊

Conclusion

While it would be nice to support old/slow cards, this new driver is much more reliable than the prior version, and is noticeably faster than before, an unexpected benefit. While there are some quirks, the only concern is that it seems to not work at all on some occasions, requiring a reboot. Is this something you can reproduce @cholonam?

If the "no SD card on some boot" problem can be fixed, I'd be happy to pay the bug bounty plus a bonus for your efforts. If you can't reproduce this problem, you still qualify for the bounty since the situation is much improved, but I'm curious if that last bit can be fixed. Let me know what you find. Thanks @cholonam!

[cholonam]

Thank you for the thorough testing @bavariancake.

It's good to know that at least the issues seem card-specific and affect seemingly older/slower cards. I only have two cards: one old 1GB card, and a newer 32GB one), but for me the kext is working quite well. I only have one issue: sometimes (maybe one out of ten) the card is not detected on insert, but it's detected if I pull-off/insert again. I can live with that for now :)

As for the other issues you mention:

• SD card slot becomes unusable if card is withdrawn during write.
  This requires a reboot - nothing is recognized after a card is pulled during a write op.

This could be fixed, but I consider it very low priority, since the user is not expected to pull the card during a write operation, anyway. A different, more serious issue would be the laptop sleeping during a write operation, but I don't think that may happen (I do not use sleep, so never tested it).

• Occasionally, an SD card would not be readable when inserted. kextstat will show that the com.sinet3k.Sinetek-rtsx kext is running, but no card will show up. The system must be rebooted.

This may be related to the issue I am also getting, only that I do see the card after pull-off/reinsert. It may be related to timeouts or error recovery.

Anyway, given that at this moment my free time is very limited, and that at least the most critical issues are solved, I doubt I will be doing any further development on this kext for the time being, apart from merging pull requests, if I get any (for example, to improve stability with chips other than RTS525A).

I do have a plan to port the Linux driver in the future (which is what I wanted to do from the beginning), but I definitely will not have time to start that task in the next one or two years, and by then I may have a different laptop/card reader... :(