Fix really slow /sdcard access on Android 11

[programmerjake]

Accessing files on /sdcard can be more than 200x slower (tested on my Pixel 4a) on Android 11 when accessing many small files. I think a potential fix might be to enable MANAGE_EXTERNAL_STORAGE:
https://developer.android.com/training/data-storage/manage-all-files

[harieamjari]

How much MB was the files? Did you mean "acessing files" as opening a file with a text editor like nano or vim?, or acessing files as copying the files from /sdcard to $HOME?

Could you do:

time cp file $HOME
stat file

How fast would you think it must so?

[ghost]

I think a potential fix might be to enable MANAGE_EXTERNAL_STORAGE

And this potential fix will require much more work than you can think. Termux doesn't support target API higher than 28.

[programmerjake]

running time rm -rv /sdcard/Minetest/ gave me:

real    5m22.694s
user    0m0.400s
sys     0m2.183s

5min is absurdly long for <100MB and <6000 files!

time rm -rv ~/Minetest/ (an identical copy) gave:

real    0m1.584s
user    0m0.030s
sys     0m0.282s

du /sdcard/Minetest/ -h gave:

96M     /sdcard/Minetest/

find /sdcard/Minetest/ | wc -l gave 5787

[programmerjake]

I guess a basic summary is Android 11's new sdcard filesystem is trash and we really should figure out how to work around it...

[ghost]

out how to work around it

Use root to manually place files to /data/media/* or place files only on internal storage such as $HOME. No other ways to workaround. You can't turn off FUSE overlay on /sdcard without intruding into system.

[harieamjari]

I created a 100MB file from /dev/urandom and created several copies at ~/storage/downloads and ~/storage/external-1
My results are not the same:

$ dd bs=1000 count=100000 if=/dev/urandom > raw.bin
100000+0 records in
100000+0 records out
100000000 bytes (100 MB, 95 MiB) copied, 21.3016 s, 4.7 MB/s
$ cp raw.bin ~/storage/downloads
$ cp raw.bin ~/storage/external-1
$ time rm raw.bin -rv
removed 'raw.bin'

real    0m0.125s
user    0m0.010s
sys     0m0.100s
$ time rm storage/downloads/raw.bin -rv
removed 'storage/downloads/raw.bin'

real    0m0.144s
user    0m0.020s
sys     0m0.120s
$ time rm storage/external-1/raw.bin -rv
removed 'storage/external-1/raw.bin'

real    0m0.307s
user    0m0.010s
sys     0m0.280s

Tested on Android 6.0.1.

Accessing the sdcard at /storage/3162-3030 gives me a permission denied error.

The /sdcard in my android is just a symlink to /storage/self/primary

$ ls -la
...
lrwxrwxrwx   1 root   root       21 Jul 12 17:35 sdcard -> /storage/self/primary
...
$ time rm /sdcard/raw.bin -rv
removed '/sdcard/raw.bin'

real    0m0.121s
user    0m0.010s
sys     0m0.100s

[ghost]

Android 11, though as my device doesn't have external sd card slot, such test is omitted:

---

Android 6 doesn't upgrade from "general shared storage" to scoped one, so unlike Android 11, performance of internal storage and shared (~/storage/downloads) will be nearly same.

[locuturus]

I think a potential fix might be to enable MANAGE_EXTERNAL_STORAGE

And this potential fix will require much more work than you can think. Termux doesn't support target API higher than 28.

I understand, and I hope that other blocking work is resolved sooner than later. For many reasons. MANAGE_EXTERNAL_STORAGE would, among other things, give filepath access to USB drives. What it might not do, however, is meaningfully speed anything up.

[twaik]

This behaviour can not be fixed since that is Android limitation.