Bangle.js 2: Reproducing compaction issues #4532
Replies: 1 comment
-
Posted at 2022-01-26 by HughB I think it might help to slightly extend the contents of the app file after it has been erased then write it back. Otherwise the chances are you might just write it back to the same slot in the storage. This would be a more real world use case. Posted at 2022-01-26 by @fanoush
You don't do that. That's why the compact is there. You can't write to same place twice without block erase. So you always write to 'new' erased space. And once you are out of it you start erasing to reclaim dead space and fill it again with valid data next to each other. then you end with one free erased area and continue until it is filled again. Or did I miss your point? Could be. What I mean is that https://www.espruino.com/Reference#l_Storage_erase just marks dead space which is unusable until you do Posted at 2022-01-26 by @gfwilliams I think I got it. I need to come up with a reliable way to reproduce, but if you have a flash page that starts with 0xFF but then doesn't end with it, then a file that fits within that area of 0xFF, compact can get in a situation where it thinks the page is empty but it's not. Posted at 2022-01-26 by @gfwilliams It turns out you can reproduce reliably with:
So it's a tricky one because often right after a compact the storage will look ok, but it's only when you start writing new files to it that soon you'll get an error and everything falls apart. The absolute latest firmware build should now have this fixed! Posted at 2022-01-26 by HilmarSt Is the fix already in 2v11.45 ?
Posted at 2022-01-26 by myxor Yes it is. You can see the builds sorted by git commit hashes here: http://www.espruino.com/binaries/travis/?C=M;O=D Posted at 2022-01-26 by HughB @fanoush - looks like @gfwilliams has found the issue. My point was that writing the same file size back after a deletion would not result in fragmentation. The fragmenter stress test app needed to start with a file of say 500 bytes, (maybe packed with spaces on the end), then write it back with a smaller size (say 480 bytes). This then maroons 20 bytes that have to be tracked. Repeat multiple times with different app files that are shrunk and then expanded and you would fragment the storage. @gfwilliams, great news, will update my firmware and try it out. Posted at 2022-01-26 by @fanoush
if you write file with 500 bytes, then 'overwrite' it with 480 bytes you end with 500 bytes of (tracked) dead space + 480 bytes of newly allocated space because nothing is reclaimed until you fill everything and/or compact Posted at 2022-01-26 by HughB The fragmenter code was reading into a variable, erasing the storage, then writing the same file back from the variable, thats not the same as an overwrite which I assume (could be wrong) would skip the erase. In such situations I would expect Espruino to find the first available slot for the file which could be the same space it had before. Having said that I am guessing as I have not read the code. My point was that a stress test needs to try to get close to what a user will do in practice. My use of the Bangle goes like this.
Posted at 2022-01-26 by @fanoush
maybe re-read post #3 I would otherwise just copy paste it :-) there are no slots to fill again as you cannot erase flash with byte granularity. Posted at 2022-01-26 by HughB
ok. quite a simple file system then. I guess a block of bytes is marked as occupied by some sort of header that records the length and that cant be changed until you compact. Not something you want to muck about with if its working. Posted at 2022-01-27 by @gfwilliams Yes - it was designed for smaller, more tightly coupled flash devices, and to allow large contiguous areas of flash memory to be allocated (which means SPIFFs/FAT/etc isn't really an option). It's not optimal on the 4MB/8MB flash chips in Bangle.js, but it is very good on reducing flash wear. Posted at 2022-01-28 by HughB Tried out 2.11.253. Done 3 successful compacts, no data loss. Looks good. Really nice to see how quickly the loader opens up with 20 apps installed. Posted at 2022-01-28 by PeerDavid I can confirm this. Upgraded to 2.11.53 - compact worked totally fine and my BangleJs2 feels extremely fast now - I was already happy before but now it feels 10x faster :D Really great thx a lot! Posted at 2022-01-28 by nicoboss Hi, Posted at 2022-01-28 by HughB This was a nasty bug to find. It was a case of smoke without fire. I'm so glad this is fixed. Posted at 2022-02-21 by nicoboss hi, latest firmware 2v12.40 and bootloader Posted at 2022-02-21 by @gfwilliams Argh, that's a shame. As always - if anyone has a way to reproduce this it'd be great. Did you ever "Install Default Apps" since moving to 2v12? Because it is actually possible that corruption caused by previous firmwares could have broken things for you. Posted at 2022-02-21 by nicoboss No, i never used it. EDIT : successful, I ll now try again to restore the fav apps Edit : fav apps successfully installed too Posted at 2022-04-21 by nicoboss Damn.... looks like it s happening again EDIT : flatening the battery launched itself (I probably touch the screen while removing the watch from my wrist ?) I stopped it with the appropriate longpress as suggested. Posted at 2022-04-21 by nicoboss so far, a few essential files corrupted edit : and at least 21 apps missing Attachments: Posted at 2022-04-21 by HughB My advice would be factory reset. Updrade firmware to 2v13 and test again. Attachments: Posted at 2022-04-22 by @gfwilliams @nicoboss which firmware were you on when this happened? Posted at 2022-04-22 by nicoboss @gfwilliams the last one : 2v13.6 Posted at 2022-04-22 by nicoboss
Not the first time it does fail to compact the storage, but it worked very well (and took only one minute) the rest of the time. And i always try to keep the firmware updated
Could this file be similar to the corrupted files i found ? Posted at 2022-04-25 by @gfwilliams @hughb's file looks totally fine to me? I forget but I'm pretty sure I was someone with a custom app that created tcx files somewhere recently. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2022-01-26 by @gfwilliams
Hi,
It seems a few of you have had issues with Storage compaction breaking Storage and losing your data (specifically on Bangle.js 2). I've been trying to reproduce this for ages so I can figure out what happens and fix it, but I just can't reliably cause issues.
It's been suggested that a
Bangle.factoryReset()
followed by installing certain apps would cause issues, but I've tried it here - now installing over 50 apps from favourites - and I can't reproduce it.Another issue might have been having hugely fragmented storage, and I have come up with this code which can be run in the IDE which should really fragment everything:
But running that for ages (with 50 apps installed) and then running:
(running from the IDE will at least allow us to see if there are any error messages).
This doesn't seem to cause any issues immediately after compaction for me either.
However I did manage to experience something - it's possible that while compact works fine it doesn't clear out the pages of flash storage immediately after the last file, which would cause a "file written with different data" error. I'm going to continue to look but any other input/hints others have would be hugely appreciated.
You can check what's actually in storage at the last (and subsequent pages) using:
Beta Was this translation helpful? Give feedback.
All reactions