Reduce SD card wear?

[mzakharo]

Hello, are there any concerns with SD card wear running BirdNet-Pi continuously?

Looking at the logs, the temporary .wav file is created for analysis inside /home/pi/BirdSongs/ directory, which is the SD card. Could this instead go to /tmp ramdisk? Only once the bird is recognized, could the.wav file be then moved to SD? Most of the night birds a sleeping, we could then avoid many writes to the SD. Wouldn't ramdisk use also improve performance/latency a bit? Thoughts?

[mcguirepr89]

Great idea!!! -- Have you tested this at all yet?

[mzakharo]

Yes I have. First things is to enable tmpfs on /tmp https://askubuntu.com/questions/1232004/mounting-tmp-as-tmpfs-on-ubuntu-20-04

Then, to use the latest server.py from https://github.com/kahst/BirdNET-Analyzer/blob/main/server.py for predictions.

Shameless plug: I wrote my own BirdNet-app for raspberry pi with Influx and telegram support that interacts with server.py: https://github.com/mzakharo/birdnetapp . One of the design goals was to avoid running out of disk-space, and not use the SD card for temporary file storage.

[mcguirepr89]

This is great! I'll play with this quite soon (I'm putting together a pre-installed image for which this would be a great touch) and hopefully add it to the installer.

Thank you, Mikhail!

[mcguirepr89]

I'm tinkering with this now, and since there are separate config variables for where the raw audio goes, where the Extracted audio goes, and where processed audio goes, I set the directories like this in birdnet.conf:

...
RECS_DIR=/dev/shm/BirdSongs

PROCESSED=/dev/shm/BirdSongs/Processed

and kept my EXTRACTED= variable the same.

I had to create the BirdSongs directory manually since its a non-default path, and make one other small adjustment, but after doing that, restarting services, now it is all being written to virtual memory except the extractions.

This should cut doing SIGNIFICANTLY on SD card wear! This approach doesn't require any other changes, as /dev/shm is already virtual memory (where /tmp is SD card by default).

I will test this out more and then will likely add this to the default installation and make it available as an update.

Thank you so much for the great idea, @mzakharo !

[mcguirepr89]

Testing shows that it DOES write to RAM, but that more configuration is needed to ensure RAM is not filled. Will update here again after more testing.

[mcguirepr89]

I've been playing with this off and on for a few days and really find it promising.

I put together a test branch for this that will, by default, use RAM for the recording and then will store extractions on the SD card.

It required a change to the way the system cleans things up -- the new test branch has a new variable in birdnet.conf (and "Advanced Settings") called "STORAGE_LIMIT" that can be adjusted to specify exactly how much of the RAM you are willing to dedicate to the RAW data. By default, it is set to 50M, but can be adjusted. This way, if you have a low-RAM model, you can keep the RAW data down to around 40 bytes (not bad!!).

I'm likely going to "call" for testing on this since it will need some sustained and vigilant observations to determine that it is as stable as the standard system. I'm very hopeful that it will not only be as stable, but more stable by virtue of cutting down so drastically on SD card read/writes

To test for now, you should be able to update "Tools" > "System Controls" > "Update", then in the terminal, type "update_birdnet.sh -b new_cleanup"

Then, in your birdnet.conf
set these variables as described previously and create the new BirdSong directory:

RECS_DIR=/dev/shm/BirdSongs

PROCESSED=/dev/shm/BirdSongs/Processed

mkdir /dev/shm/BirdSongs

and finally

restart_services.sh

for things to take effect.

Please let me know if you are able to test this method out a bit, and if so, I would love any feedback you can provide.

My very best regards,
Patrick

[issakandji]

Hey,
Trying update because had latency and full disk issues.
Here are my finding:
Update button keep looping to restart service.
Questions : why running restart script when update button restart service too ?
After a reboot service seems to work smoother.
Storage Limit could be adjusted while throwing some errors even though the reported file exist
birdsounds:~/BirdNET-Pi$ ls scripts/labels.txt
scripts/labels.txt
I think i need more insights. Thanks for this update, by the way

[issakandji]

label was also missing under model dir.
All went fine after cp
~/BirdNET-Pi/model$ cp labels.txt.old labels.txt

[mcguirepr89]

Hi, @issakandji -- this is still actively being tested, so whatever feedback you can provide will be helpful.

Take a look here for more information on the changes and current testing needs for this tweak -- please let me know whatever feedback you have right there on the Testing discussion, if you don't mind.

My very best regards,
Patrick

[Phaeton]

I will note that I have it running on a 2GB model without any issue. I assigned only 100MB to storage as I saw no reason to hold any more. Is there a benefit to increasing that size? It only holds unprocessed and processed recordings correct? If the Pi can keep up with the recordings, is there any reason to cache the processed recordings?

[Phaeton]

I accidentally deleted that branch, but still have it locally, so it's not lost.

Any chance I could get you to repost the modified cleanup.sh. I thought I had backed it up, but overwrote it with your most recent updates today :|.

[Phaeton]

I made a quick fix, reducing the check from 100 files down to 30 to keep the ram size in check for now. I am looking forward to when the write to ram makes it to the main channel :).

[isZumpo]

@mcguirepr89 I recently ran into this discussion when noticing that "in processing" recordings were saved directly to the SD card rather than kept in ram. I was very happy to see that there already was something in the works to solve this issue. However, the new_cleanup branch is still gone and it seems like no such feature has been included into newer releases of birdnet pi. Would it be possible for you to re-upload that branch? :)

[MueJosh]

I recently ran into this discussion when noticing that "in processing" recordings were saved directly to the SD card rather than kept in ram. I was very happy to see that there already was something in the works to solve this issue. However, the new_cleanup branch is still gone and it seems like no such feature has been included into newer releases of birdnet pi. Would it be possible for you to re-upload that branch? :)

A little late to the party, but the nachtzuster fork has an experimental ram drive.
Haven't tested it but it should work.

[garbled1]

My solution was to simply mount a 100G NFS to BirdSongs. That also lets me keep an absurd number of recordings.