Testing

[mcguirepr89]

Hi, folks!
As requested, I am working on building this out from the BirdNET-Lite repo as well. If anyone can share successes in getting this built on aarch64 machines, I'd love to know more. I will post updates to this pursuit here and will update the BirdNET-Lite Project.
Wish me luck!
Patrick

[nilspupils]

That's great news! Good luck and hit me up if you need someone to test.

[mcguirepr89]

I surely will -- Did your microphone ever arrive for you to test out this repo?

[nilspupils]

Not yet! It is coming from far far away...

[ghost]

Which Microphone can you recommend? Which one did you use? I think about to buy a Rode NGT VideoMic.

[DD4WH]

@Christoph-Lauer
have a look here:
https://github.com/mcguirepr89/BirdNET-system/discussions/26

[DD4WH]

@mcguirepr89: very good to hear this! Advantage of the Lite version would be (from my perspective):

• we could use the BirdNET-Lite classifier (5000+species), which works really well (tested by me on several Terabyte of audio recordings)
• less power hungry!?? Could this run on RPi3 or even 2?
• faster?
• easier to implement other parallel real time tasks (spectrogram etc.)
  Good luck! And I would also be ready to test a first alpha version ;-)

[mcguirepr89]

@DD4WH I have a lot of tinkering to do with this before I can say anything one way or the other regarding BirdNET-Lite. Regarding the heat, however, I can say with confidence that these systems will definitely stay hot on any device since the analysis takes all of the CPU. (I overclock mine also, which really heats things up.) Passive cooling through a heat-sinking case coupled with a small fan provides great results for me. According to this, "On aarch64 OS, performance is about 4 times higher than on armv7l OS." So I'm hoping to still use aarch64, but may shrink the OS down to using the RaspiOS_Lite_ARM64. Currently, the BirdNET-system installs just fine on the raspios_lite_arm64 image, but cannot use the Birders_Guide_Installer.sh (installation one-liner), and must instead be installed using the ~/BirdNET-system/scripts/install_birdnet.sh. That method supports an interactive installation or can accept a pre-filled birdnet.conf to automate the installation. I suspect I will create documentation for a "headless" installation that can be performed on the RaspiOS_Lite_ARM64 image to help folks begin testing out the resource needs.

But you're trajectory is definitely the right one: make every footprint smaller and do anything available to reduce the system's needs. Getting Lite working on the aarch64 is a hurdle, but I'm working on it and will keep you posted so those pygmy owls can have their say :)

[mcguirepr89]

@DD4WH Hello, again -- I need your help if you can supply it. I have gotten the BirdNET-Lite model to install with the same ease as the regular BirdNET repo and it passes all tests and expectations with flying colors! The issue I'm running into is getting recordings in the expected audio format that works with BirdNET-Lite. The repo specifies

"Note: BirdNET expects 3-second chunks of raw audio data, sampled at 48 kHz."

which I thought I was creating with the newer recording script, but librosa complains for all of the recorded audio I give it. It doesn't complain, however, when processing the example audio included in the repo's "examples" directory.

My question for you is: for the Terabytes of audio data BirdNET-Lite was able to analyze for you, what can you tell me about those file formats?

I need to figure out the best resource available to the Pi to get recordings in the audio format BirdNET-Lite accepts, but so far, my attempts at making 3-second chunks of raw audio data sampled at 48kHz have failed.

Thank you very much for your help with everything --
Best regards

UPDATE: I got it working :) Thanks anyways

[DD4WH]

@mcguirepr89, Patrick, I copy a link to one of those audio files here:
https://github.com/DD4WH/monitoR/blob/master/AK/Au23_2021_09_08_05_40_01_Sperlingskauz_Rivalengesang.wav

I record one minute recordings with wav-header, 16bit PCM, 48ksps sample rate, so nothing exotic . . .
The python script in BirdNET-Lite then automatically cuts those files ionto 3-sec chunks, as far as I understand the code:
https://github.com/kahst/BirdNET-Lite/blob/main/analyze.py

UPDATE: my post crossed with your update :-) Good luck and continue the good work!
Very exciting advances here! Will test your suggestion (7d31737) as soon as I find time! Getting late here, time to go to bed

[ghost]

@mcguirepr89: very good to hear this! Advantage of the Lite version would be (from my perspective):

• we could use the BirdNET-Lite classifier (5000+species), which works really well (tested by me on several Terabyte of audio recordings)
• less power hungry!?? Could this run on RPi3 or even 2?

YES

• faster?

YES

• easier to implement other parallel real time tasks (spectrogram etc.)

YES, for spectograms you couls also use my program --> https://github.com/Christoph-Lauer/Sonogram

Good luck! And I would also be ready to test a first alpha version ;-)

NOTE: The Lite version uses the TENSORFLOW Lite machine learning library, which is much more better than the Machine Learning Libraries used in the NOT-Lite system. I highly suppose to use the Lite (TF-Lite) ML Libraries from Google instead the previous version.

[mcguirepr89]

Hello, everyone -- I'm happy to report that beta testing can begin for the BirdNET-Lite-system (needs a better name).

The new beta-testing one-liner: curl -s https://raw.githubusercontent.com/mcguirepr89/BirdNET-Lite/rpi4/Birders_Guide_Installer.sh | bash

Some exciting results:

• BirdNET-system (built on the original BirdNET) would take about 6-8 minutes between live audio to analyzed and extracted data. Built on the BirdNET-Lite model, however, the time from live audio to analyzed and extracted data is between 3-15 seconds. My results are really promising for only having put this together today, so I'm hoping you guys are going to be able to help me out and start collecting some data to see what needs tweaking.

Other notes:

• The "By_Date" section of the file server is not, in fact, sorting things by date. I'll work on that tomorrow.

My best regards as always,
Patrick

[CaiusX]

Hi Patrick

Great work in getting it running!! Some "birder-beta" comments:

1. Installation script installed flawlessly & started the services after reboot.
2. Identification of local South African birds working well - after 2 hours of monitoring 8 unique species identified at 0.7 confidence, 6 correct and 2 false positive.
3. Certain species which are definitely vocalizing are not being identified at 0.7 confidence. Will run at lower confidences and see what the "sweet spot" is for True Positive/False Positive, suspect around 0.3-0.5 eyeballing at the actual detections data.
4. Microphone could also be a factor - Boya M3 USB Lavalier direct plug in doesn't seem to catch high and low frequencies (Southern Boubou and Collared Dove are very vocal, but not registering above 0.1 confidence) and volume is low. Any recommendations from other users for a cost-effective microphone solution or a DIY build?
   4a. Could be that the model needs better training, which is another rabbit hole I'm completely ignorant of ....... see my ideas below.
5. birdstats.local page reports "-56 detections" but "-0 species identified so far".

Some ideas for future development:
a. As previously suggested by @DD4WH , "realtime" ID along the lines of the https://birdnet.cornell.edu/live/
b. Set up a system for gathering potential "crowdsourced" training data along the lines of https://www.allenbirdcam.com/train-ai

Thanks for the hard and successful work - I will continue testing and report any problems/ideas. I'm happy to share my installation on the internet, I have a domain but don't have a clue on how to publish & keep my system secure. Could you point me in the direction of an "idiot's guide" for this?

Best
DennisC

[mcguirepr89]

I'm happy to share my installation on the internet, I have a domain but don't have a clue on how to publish & keep my system secure. Could you point me in the direction of an "idiot's guide" for this?

No "idiot's guide," but there is an "awesome South African" guide that ought to work :)
I'm going to create a new Discussion called "Sharing Your BirdNET-system" that you can use to alter the setup to be reachable at your domain!

[mcguirepr89]

A Sharing Guide is now available. Please let me know if it is clear enough to follow (but on that discussion page, of course 😄)

[DD4WH]

@mcguirepr89
Fantastic, the system works absolutely fine!

INSTALLATION

• Installation process works perfectly!
• a bit irritating that there are several warnings during the installation process: "running pip as a root user", but that does not seem to affect functioning

FUNCTION

• tested for 2 hours with different bird songs/calls from my smartphone in different volumes: very good ID results!
• IDs are all from my region (eg. no North American species identified here in Central Europe), although I did not specify a custom species list: is there a possibility to deliver a custom species list?
• birdstats.local prints the correct no. of detections, but always stays at ZERO species detected
• "HOTSPOT-mode" is working fine! --> no LAN cable connected to the RPi, I set up a hotspot with my smartphone, this automatically connects to the RPi (before rebooting the RPi, I had to connect the hotspot once and enter the password on the RPi) and then I can remotely control the RPi via NoMachine from my smartphone-->open the browser on the RPi and enter: http://raspberrypi.local) --> even the audio stream gets transmitted

COMMENTS/OPTIMIZATION

• it would be good to save volume space and not save redundant files under "Extracted/by common name" and "Extracted/by scientific name": but maybe this is already done??
• it would be better to use multiple of 3 second chunks as the default recording length: the script cuts the 10sec-file into 3-second chunks and the last file (1 second long) gets augmented by lots of zeros, so better use 12seconds, or maybe 24 or 48 seconds?

NEW FEATURE REQUESTS

• is it possible to set parametres in realtime while the system is running?, eg. confidence score threshold
• a button/possibility to reset the species list would be nice
• the possibility to show a species list for the last day(24hrs), week, month
• is there a possibility of using an USB stick for saving the extracted files? This would make it very easy to transfer the extracted files by just swapping the USB stick

MODEL TRAINING

• @CaiusX : I think we would have to rely on models trained by Stefan Kahl or other specialists in this topic. The training of the model is so complex with millions of training files used and carefully chosen, noise added to those files in order to improve ID in real-life situations etc. I myself would not dare to train such a model by myself . . . maybe you can ask Stefan Kahl how much regional coverage he has achieved in your region (South Africa)

@mcguirepr89 👍 thanks a lot for this BirdNET-Lite system: this is very close to a system that is ready to throw into the field and get really exciting and useful results from it!

[mcguirepr89]

@DD4WH -- I always appreciate your feedback, so I want to be sure you get some from me.

INSTALLATION

• a bit irritating that there are several warnings during the installation process: "running pip as a root user", but that does not seem to affect functioning

I can silence those warnings, but will probably rewrite the build to take place within a virtual environment. The original BirdNET-system built in a virtual environment, since that is usually best practice. But since this whole OS is dedicated to running the python interpreter for BirdNET purposes, I have the BirdNET-Lite install all site-packages into the main library locations. If anyone wants to add any python development to the project in the future, they would need to be sure to use a virtual environment, so I ought to go ahead and isolate the BirdNET python needs to make way for others.

FUNCTION

• is there a possibility to deliver a custom species list?

Yep! It's coming up soon. I hope that will be available in the next few days. The plan is that you place a custom species list in the BirdNET-Lite directory. If it is there and is readable, then it will get used -- if there is not one present, or it is named something other than "custom_species_list.txt" (or whatever), no list is used.

• birdstats.local prints the correct no. of detections, but always stays at ZERO species detected

Thank you, I'll correct that.

COMMENTS/OPTIMIZATION

• it would be good to save volume space and not save redundant files under "Extracted/by common name" and "Extracted/by scientific name": but maybe this is already done??

You are correct, and it is already so -- the only files that actually exist are the "By_Date" extractions. What was formerly "By_Species" and is now both "By_Common_Name" and "By_Scientific_Name" are just symbolic links that point to the original "By_Date" files.

• it would be better to use multiple of 3 second chunks as the default recording length: the script cuts the 10sec-file into 3-second chunks and the last file (1 second long) gets augmented by lots of zeros, so better use 12seconds, or maybe 24 or 48 seconds?

Brilliant catch -- I will make the change to 9 and will compare that to 12.

NEW FEATURE REQUESTS

• is it possible to set parametres in realtime while the system is running?, eg. confidence score threshold

Yes! Especially with the BirdNET-Lite model. I haven't developed a very user-friendly way to do this yet, like through the web browser as has been suggested. But you can change the CONFIDENCE value in the birdnet.conf file at any time and the changes will be adopted upon the next service cycle, which for the BirdNET-Lite model is about 10 seconds or so. You can also change the OVERLAP value there, too.

• a button/possibility to reset the species list would be nice

Noted! For now, you can simply delete the files ~/BirdNET-system/IdentifiedSoFar.txt and ~/BirdNET-system/IdentifiedSoFar.txt.bak to wipe it clean. (If you have a smartphone application that will allow you to send SSH commands, you can do that with a one liner like: ssh pi@rasberrypi.local "rm ~/BirdNET-Live/IdentifiedSoFar.txt{,.bak}").

• the possibility to show a species list for the last day(24hrs), week, month

I'll tinker with this as I'm able to collect data.

• is there a possibility of using an USB stick for saving the extracted files? This would make it very easy to transfer the extracted files by just swapping the USB stick

Absolutely! The process will certainly need to be abstracted for the average user, but the idea is sound. The requirements would simply be that the USB drive be formatted as either FAT32 or a linux file system (ext4). With that, I can arrange an automated systemd /etc/fstab entry that will auto-mount the USB to "/home/pi/BirdSongs" so that all data is stored right on the disk. Ideally, I would also set some checks in place so that the system still collects data in another directory anytime a USB is not present. I would want to really make this process robust so the average user could swap out the sticks without too much fuss. Great suggestion! I'll have to get my hands on a few USB sticks to test with.

Thanks again,
Patrick

[DD4WH]

Patrick, thanks a lot for your answer!

• after thinking about it a bit more, maybe the 10sec default recording length is just fine: if you adjust the overlap [I always use 0.5sec overlap, which I have good experience with] the 10sec file will be split into four parts and the last one contains 2.5sec of real audio and only 0.5sec of zeros ;-) And it is probably not worth it adjusting this for all possible settings of overlap . . .

• very good to be able to change parametres in birdnet.conf in realtime!

• However, I did not manage to clear the species list. Probably I was not clear enough: I meant to reset/delete ALL the species extractions etc., so that I can start with real birds getting detected (not all the extractions of bird calls from my smartphone which I used to test the system :-)

• USB stick support would be excellent!

[mcguirepr89]

To remove all of your extractions so far and start fresh, run

rm -drf ~/BirdSongs/Extracted/By_Date/*
rm -drf ~/BirdSongs/Extracted/By_Common_Name/*
rm -drf ~/BirdSongs/Extracted/By_Scientific_Name/*
There are little things I'm catching as data is coming in that need to be tweaked, and the BirdNET-system Report is one of those things. I have fixed it so that it shows some info now, but nothing special. You can see for yourself if you want to pull in the new birdnet_stats.sh file. Run

cd ~/BirdNET-Lite && git pull

http://birdstats.local runs the birdnet_stats.sh fresh each time the page is loaded, so just running a git pull from the main directory ought to get those changes for you. Like I said, still nothing special at this point while I work on smoothing out all of the differences between the original BirdNET and BirdNET-Lite.

[mcguirepr89]

FYI, 9 seconds is the sweet spot for the default

[mcguirepr89]

I put together a small tool that is better that the list of commands above. It is an included script called ~/BirdNET-Lite/scripts/clear_all_data.sh which does just that.
Get the new tool with cd ~/BirdNET-Lite && git pull

[DD4WH]

@CaiusX
I am currently developing an idea for a DIY mic for the BirdNET-Lite system.
My idea is the following:

• use the ultra low noise electret omnidirectional mic capsule EM272 [SNR 80dB !] https://micbooster.com/gb/primo-microphone-capsules/199-primo-em272.html
• use a mic line driver as proposed by Elliott at ESP [https://sound-au.com/project93.htm have a look at figure 6]
• use a low noise phantom power supply to deliver power to the mic line driver and the mic itself [https://sound-au.com/project96.htm]
• use a standard USB audio interface [low cost]
• When I have a prototype ready, I will post about it here
  However, this will not be a project suitable for someone holding a soldering iron for the first time . . .

[DD4WH]

Have a look here at this DIY mic/preamp/power supply setup:

https://github.com/DD4WH/SASS/wiki/Stereo-recording-low-noise-low-cost-system

[DD4WH]

in order not to hijack this thread, I opened a new thread/discussion on the topic "USB microphones for BirdNET-system"
https://github.com/mcguirepr89/BirdNET-system/discussions/26

[mcguirepr89]

You guys are awesome!!! I really appreciate the beta testing and the great observations and suggestions. I love all of the ideas and really appreciate the dialog that's going on while sleeping here in North America!

I will start putting together some project cards to keep track of next steps and will share here with what to expect.

If you guys don't mind, could you send me the ~/BirdNET-system/log.tar.gz from your newest installation? I'm eager to take a look at a few small expected differences to determine how to move forward. (Please email it, since it does still contain 1 password. Although that password is only used on the Raspberry Pi's loop back network to access its own icecast2 stream, there are a lot of smarty pants out there who would love to try something nefarious).

I expect there will be a new version for you all to test tomorrow :)

My best regards and tons of gratitude,
Patrick

[mcguirepr89]

PS
I forgot to mention that this new BirdNET-Lite build is divorced from the Anaconda build process, which means its use is no longer subject to Anaconda's EULA!!!! Hooray!

[mcguirepr89]

Hello, again! I have made several changes based on what you all have seen during beta testing. The following changes have been made and are ready for more testing:

• BirdNET-system Reports "Species So Far" fixed
• Recording duration changed to 9
• By_Date is back and is properly organizing files
• The system now accepts custom_species_list.txt files it finds in the ~/BirdNET-Lite directory. If it finds none, it won't use it -- if there is one there, it will pick it up and apply it.
• Confidence scores are now integrated into the extraction file name (With the % character)

Installation one-liner: curl -s https://raw.githubusercontent.com/mcguirepr89/BirdNET-Lite/rpi4/Birders_Guide_Installer.sh | bash

[CaiusX]

Hi @mcguirepr89

Installed the latest version on a fresh SD card.

The installer appeared to hang at the first reboot - some message about not finding the birdnet conf directory when I ctrl-c 'd. Unfortunately didn't stop there and send you a log.

After a second reboot the system installed fine and works as advertised.

I'll mail you the installation logs separately.

Best
Dennis

[CaiusX]

Hi @mcguirepr89

System running well all day. One issue is that time in file names is 12 hr format - 24 hr would be preferable?

Best
DennisC

[mcguirepr89]

Thanks for the update! I'm glad it's working well for you. Here in North America, I've had to crank the minimum confidence level way up (0.9) for BirdNET-Lite not to give me very diverse false-positives. I live in such a noisy area, it's great to see that BirdNET-Lite is so creative in matching these ambient demolition and human noises to bird sounds, as I imagine that leads to more detections deeper in nature (or even just in less urban areas).

And I agree that 24-hour time formatting should be available -- though I'm on the fence about setting it that way by default. I hope others will weigh in on this. Regardless of what I ultimately set as the default time formatting, I am going to add a birdnet.conf variable, TIMESTAMP_FORMAT that will either be set to 12 or 24 so that the user can decide how they want these files to be organized. I'm not close minded about this, but know that a lot of the bird hobbyist in North America will probably want this in 12h format, while academics and researchers will probably prefer the 24h formatting. Is 24h time formatting preferred in South Africa?

[mcguirepr89]

The TIMESTAMP_FORMAT variable is now testable. If you don't want to go through a whole clean install, you can help me by testing the included ~/BirdNET-Lite/scripts/reconfigure_birdnet.sh. That script is a piece of the installation process, so is sort of like cutting in line on the main installer.
You can do the following:

1. cd ~/BirdNET-Lite && git pull
2. reconfigure_birdnet.sh

In that script, you can also go ahead and set the BirdNET-Lite system up for internet sharing. During the reconfiguration, you'll be asked about the timestamp you'd wish to use, but then you'll be asked eventually about sharing over the internet. Here, instead of putting http://raspberrypi.local or http://localhost, input your domain (with the https), so for example, https://pmcgui.xyz. That will automatically configure the Caddyfile as described here. Also, you'll notice that I have incorporated confidence scores into the extraction file names.

Good luck!

[CaiusX]

Updated as per above, will report back on time format after mid-day.

Diverse false positives also an issue. Most "interesting" one is that it frequently identifies my voice as an African Penguin! Interestingly I live hundreds of miles from the ocean. I created a whitelist of Southern African bird species which occur in the BirdNET-Lite data (/BirdNET-Lite/model/lables.txt) and removed the offending penguin - seems to be working & the penguins have stopped being logged!!

Thanks for getting the confidence score in the filename, its going to force me to learn "advanced" parsing; the "-" character being used as a delimiter, as well as being part of bird names, e.g. 586-Cape_Robin-Chat-99%-2021-10-01-birdnet-04:21:38.wav.

I'm looking at this stage to create a database including the following:

Date -- Time -- Bird_ID -- Lat -- Lon -- Confidence -- Confidence Threshold -- Overlap -- Sensitivity

and then doing various analytics -
e.g.
Number of unique species
Time series distribution (hour/day/month/year/decade) - (as you can see, I'm in this for the long haul!)
Frequency distribution
Confidence/false positive analysis (to try and find a sweet spot for confidence threshold and any other model parameters such as overlap, sensitivity......)

Sorry for the meandering post, but may spark some other ideas and suggestions?

Best
African Penguin

[CaiusX]

24 hour timestamp working perfectly.

Tx

[mcguirepr89]

We're on the same track -- I have had databases on the mind lately -- check out the new discussion and we'll make it happen!

[DD4WH]

Wow, lots of new updates, It seems my system is outdated already . . . :-)

Had my system in the field for two days (and it catched Eurasian Pygmy owl with a confidence of 99% ! And I can confirm it truely is)
89-Eurasian_Pygmy-Owl-99%-2021-10-01-birdnet-06 20 03.zip

Further comments:

• works nicely unattended
• I have it in the field with an LTE router, it is connected to the router by LAN cable. With my smartphone I connect to the router by WiFi, then I can use NoMachine to listen to the system etc.
• However: from my home (when I am not connected to the LTE router) I am unable to connect to the RPi by NoMachine (it is not found and probably I have no public IP adress with that LTE router!? not sure, I am a newbie to these network things . . .)
• Does anybody have an idea how I can connect to the RPi with birdnetsystem on it, which is connected to an LTE router (with permanent internet access, but obviously without a public IP adress)?

FALSE POSITIVES.

• I think we should be careful here not to have false expectations! the BirdNET-Lite system can be controlled by several parametres: sensitivity, minimum confidence score, overlap, and week and some others
• it would be awesome for the user to have control over these parametres
• for example: if you would like to lower the number of false positives, it would be much better to reduce sensitivity (and thus increase specificity) of the classifier model by lowering the sensitivity parameter. If there are still too many false positives after that, you can then increase the minimum confidence score
• for my application it is totally acceptable to have false positives. However, it would be desirable to be able to have a graph or table with an overview (best would be a histogram!) of the achieved confidence scores for the detections of a specific species--> from this histogram you can quickly and very easily see whether you have a false positive or a real detection

RECORDING LENGTH

• my feeling is that it would be better to have a longer recording length (maybe 30sec or one minute) instead of 9 or 10 sec and to have the extraction procedure cut out longer recordings
• the reason is that it is much easier and safer to confirm a detection by listening to a recording of 20 or 30 seconds, because you can also listen to the context of the recording and maybe quieter additional sounds that BirdNET has not detected, but which can be heard by ear

CUSTOM SPECIES LIST

• seems to work nicely! However, I cannot prove this, because I get no feedback which species list is being used. But up to now, no weird foreign species appeared

OK, I will now pull the latest version and make a fresh install !

Keep up the good work, Patrick!

[mcguirepr89]

Had my system in the field for two days (and it catched Eurasian Pygmy owl with a confidence of 99% ! And I can confirm it truely is)
89-Eurasian_Pygmy-Owl-99%-2021-10-01-birdnet-06 20 03.zip

THAT IS SO AWESOME!!!

However: from my home (when I am not connected to the LTE router) I am unable to connect to the RPi by NoMachine (it is not found and probably I have no public IP adress with that LTE router!? not sure, I am a newbie to these network things . . .)

I'm going to upload a screen shot of where you might be able to find the public IP address for that LTE router. NoMachine won't find clients on a different network unless you've registered the server with a license that you would then somehow link to that client. For personal use, if you click the red NoMachine icon in the top right of the desktop and choose the option to "Show server status" you'll see a list of the IP addresses NoMachine is serving on. With the BirdNET-system connected to the LTE router, make note of those IP addresses and you may find that one is publicly available.

FALSE POSITIVES . . . sensitivity, minimum confidence score, overlap, and week and some others

As it is, the user can set the minimum confidence score, overlap, (week is set automatically based on recording date).
I have not put anything together for the sensitivity, so I will definitely add that. I've never been sure how to use it, so I forgot about it during development 😕 BUT it will be available (probably tomorrow or later tonight).

RECORDING LENGTH

This part interests me the most.
I have had a few local iterations of the system that will concatenate the audio before and after the extraction to give that context. It is also helpful for longer bird calls.

Knowing that form a biologist's perspective the extractions ought to have more context. Since finding the right values is important, I think this will be the plan:
I will create (surprise surprise) another variable in birdnet.conf called RECORDING_LENGTH that can be set to adjust the recording length.
I will also revisit how the extraction.service splits the audio selections for another (surprise surprise) birdnet.conf variable called EXTRACTION_LENGTH. Then I will count on your savoir faire for finding the best, most usable values.

Once we know what is best, I will probably go back and remove those variables and hardcode what works, or will move those variables down as defaults so that folks can adjust them as needed, but will be in good shape with the values we discover through testing.

CUSTOM SPECIES LIST
seems to work nicely! However, I cannot prove this, because I get no feedback which species list is being used. But up to now, no weird foreign species appeared

You'll notice in this next installation, that I saw the same thing and wanted the user to be sure that the custom species list was being accepted and used. Now, you'll see that the http://birdlog.local shows you exactly the line called for the BirdNET analysis.

OK, I will now pull the latest version and make a fresh install !

This will be a very verbose installation for you, but that is temporary. I still plan to build the birdnet-lite python virtual environment.

Best of luck! I look forward to your next update :)

[mcguirepr89]

RECORDING LENGTH
my feeling is that it would be better to have a longer recording length (maybe 30sec or one minute) instead of 9 or 10 sec and to have the extraction procedure cut out longer recordings
the reason is that it is much easier and safer to confirm a detection by listening to a recording of 20 or 30 seconds, because you can also listen to the context of the recording and maybe quieter additional sounds that BirdNET has not detected, but which can be heard by ear

I have made the adjustment so that recording length can be set dynamically. A service restart for birdnet_recording.service needs to take place after changing the variable for it to take effect.

I am tinkering with extraction length -- what would you recommend as the duration for an extraction?

[mcguirepr89]

So the RECORDING_LENGTH testing has thrown some curve balls. I don't recommend tinkering with it just yet. I will let you know.

[mcguirepr89]

Hello, all! UPDATES:

There are now two new variables for testing: RECORDING_LENGTH and EXTRACTION_LENGTH

RECORDING_LENGTH defaults to 12 seconds
EXTRACTION_LENGTH now defaults to 6 seconds -- the original 3-second audio selection, plus 1.5 seconds before and after.

RECORDING_LENGTH can support values between 12 and 60 (in seconds)
EXTRACTION_LENGTH can be set to any value less than or equal to RECORDING_LENGTH

@DD4WH I tested with setting RECORDING_LENGTH to 60 and EXTRACTION_LENGTH to 20 and sure enough, right at 10 seconds you hear the identification. I hope the extra audio context is helpful 👍

[CaiusX]

Hi @mcguirepr89

Instructions for installing this update needed by the "birder level community" -

Applying
cd ~/BirdNET-Lite && git pull
reconfigure_birdnet.sh

results in http://extractionlog.local/-

sounds.sh[18263]: ++ last_command='(( $(echo "${END} > ${RECORDING_LENGTH}" | bc -l) ))'
Oct 02 06:05:09 raspberrypi extract_new_birdsounds.sh[18263]: ++ current_command='set +x'
Oct 02 06:05:09 raspberrypi extract_new_birdsounds.sh[18263]: + set +x
Oct 02 06:05:09 raspberrypi extract_new_birdsounds.sh[18263]: -to value smaller than -ss; aborting.
Oct 02 06:05:09 raspberrypi systemd[1]: extraction.service: Main process exited, code=exited, status=1/FAILURE
Oct 02 06:05:09 raspberrypi systemd[1]: extraction.service: Failed with result 'exit-code'.
Oct 02 06:05:12 raspberrypi systemd[1]: extraction.service: Service RestartSec=3s expired, scheduling restart.
Oct 02 06:05:12 raspberrypi systemd[1]: extraction.service: Scheduled restart job, restart counter is at 128.
Oct 02 06:05:12 raspberrypi systemd[1]: Stopped BirdNET BirdSound Extraction.
Oct 02 06:05:12 raspberrypi systemd[1]: Started BirdNET BirdSound Extraction.
Oct 02 06:05:12 raspberrypi extract_new_birdsounds.sh[18750]: Starting numbering at 2142
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: Extracting audio . . .
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='set -x'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='[ -z ${EXTRACTION_LENGTH} ]'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + '[' -z ']'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='[ -z ${EXTRACTION_LENGTH} ]'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='SPACER=$(echo "(${EXTRACTION_LENGTH} - 3 )/2" |bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ echo '( - 3 )/2'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ bc -l
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + SPACER=-1.50000000000000000000
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='SPACER=$(echo "(${EXTRACTION_LENGTH} - 3 )/2" |bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='START=$(echo "${START} - ${SPACER}"|bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ echo '3.0 - -1.50000000000000000000'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ bc -l
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + START=4.50000000000000000000
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='START=$(echo "${START} - ${SPACER}"|bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='END=$(echo "${END} + ${SPACER}"|bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ echo '6.0 + -1.50000000000000000000'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ bc -l
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + END=4.50000000000000000000
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='END=$(echo "${END} + ${SPACER}"|bc -l)'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='(( $(echo "${START} < 0" | bc -l) ))'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ echo '4.50000000000000000000 < 0'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ bc -l
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + ((  0  ))
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='(( $(echo "${START} < 0" | bc -l) ))'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='(( $(echo "${END} > ${RECORDING_LENGTH}" | bc -l) ))'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ echo '4.50000000000000000000 > '
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ bc -l
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: (standard_in) 2: syntax error
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + ((    ))
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ last_command='(( $(echo "${END} > ${RECORDING_LENGTH}" | bc -l) ))'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: ++ current_command='set +x'
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: + set +x
Oct 02 06:05:14 raspberrypi extract_new_birdsounds.sh[18750]: -to value smaller than -ss; aborting.
Oct 02 06:05:14 raspberrypi systemd[1]: extraction.service: Main process exited, code=exited, status=1/FAILURE
Oct 02 06:05:14 raspberrypi systemd[1]: extraction.service: Failed with result 'exit-code'.

Best

[mcguirepr89]

I think that my most recent pushes from yesterday were premature -- I also left a ton of debugging on in the extraction.service, which was an accident, but is pretty helpful here.

I'm sorry for the inconvenience and I will do my best to get things straightened out very soon.

My best regards,
Patrick

[DD4WH]

@CaiusX
thanks for testing and posting your experiences: Your experience with "African penguin" is amusing ;-), here in Central Europe, human speech (especially kids screaming) is mostly getting ID´ed as "Whooper Swan" :-)
@mcguirepr89
thanks, Patrick for your efforts and the new version!
I will set up a totally new SD card and test with that this weekend [will be busy with other things also, so feedback is likely not to come before Monday]. I was not aware of the possibility of installing a version onto an existing SD card with an older version!? Maybe you could clarify what the prefered way of installing your updates would be.
Thanks a lot! Best regards, Frank DD4WH

[mcguirepr89]

I was not aware of the possibility of installing a version onto an existing SD card with an older version!? Maybe you could clarify what the prefered way of installing your updates would be.

@CaiusX @DD4WH @nilspupils I should be clear that "updating" is not yet supported (git fusses when substantial changes have been made, and there are files throughout the system that need updating, too, so this will need to be its own tool more powerful than what git can deliver.)

There is an included script called reconfigure_birdnet.sh that simply uninstalls all configurations and service files and then runs the manual installer to reconfigure everything that was removed. It is not necessarily going to show the user what the newest iteration of the system will do (since the manual installation does not carry with it any of the installation one-liner defaults.)

I am aware that going through the full re-installation process (especially if making a new SD card for it is involved) is painstaking and time consuming. And for updates like last night, where I break stuff!, you would hardly want to go through that all to have a system that is a dud.

I'm going to work on the kinks you all have found and hopefully won't push any breaking changes again!

[DD4WH]

@mcguirepr89
no problem! I am ready to test all alpha versions you provide!

[nilspupils]

Great news! Ready to switch to main!
Would
git checkout main
do the trick?

[CaiusX]

Thanks @Svardsten53 "Todays Detections" now working!!

[DD4WH]

@CaiusX
That is really awesome!!! The Species Stats page is exactly what one would wish for!!! Perfect!

In order to really appreciate that, I would like to take the bird.db from a former installation and put that into the new installation. However, I did not manage to do that, because it seems the bird.db gets renewed and filled with 0 bytes as soon as I do that.
Surely, I have missed instructions how to get the database into a new installation. Could anyone provide me with a link ?

[DD4WH]

@CaiusX
Your graphs are very very good and they leave nothing more to wish for, I think.

However, there is always something that could be improved :-). There are two points where interpretation could be improved, I think:

• time representation: maybe make this switchable for the diurnal phenology: relative to sunrise / sunset vs. absolute time
• no. of detections vs. time windows --> at the moment, the no. of detections is the no. of 3sec-intervalls where a bird species is detected. It would make more sense to eg. take 10 min-intervalls and determine whether a bird species has been detected in that 10min-intervall and use this as an indicator of the presence of a bird in a phenological diagram. This is the standard way that bat researchers analyse their data. Also this has been proposed for birds, have a look at figure 6 here:

https://avesrares.wordpress.com/2022/02/27/nocmig-a-modern-approach/

I know this is a very complex and difficult to implement graph, especially if it is dynamic, but that would be the high-end graph ;-). Would that be feasible / approachable ?

[mcguirepr89]

Welcome to the version 0.13 gang! I really like the new Species Stats.

I also had an issue with having "Today's Detections" not working, but found what was wrong.

• Go to /home/pi/BirdNET-Pi/homepage/views.php
• Find the line "if($_POST['view'] == "Today's Detections"){include('todays_detections.php');}" and change it to "if($_POST['view'] == "Today's Detections"){include('scripts/todays_detections.php');}"

Thanks, @Svardsten53!!
I fixed that in the installer:
5a79fe0
0c28d2e
2e9bbb7

[CaiusX]

Hi @DD4WH
Frank - thanks for the great ideas-

Including weather data has been on the back burner for a while, its time to revisit the wittr service and get the local temperature/rainfall/dusk/dawn etc data into the DB.

This will facilitate dark/light coloring of the polar plot background using the "mode/mean?" sunrise/sunset for the selected time range.

The Figure 6 is fantastic. Ideally I'd like to click a species in the species bar chart (which is scrollable rather than zoomable) and get the data on the right two charts (replacing the bottom one with Fig 6), together with spectrogram and sound. I'm sure all is do-able as I head up the python visualisation learning curve.

For the hourly chart how about a 6 minute standard time slice (so max 10 readings per hour) and then scaling the radial vector as a per-cent? As can be seen in my Cape Robin Chat, although they're around all day, the morning chorus swamps the picture. Other alternatives would to have a hard cut off at, say 10 detections in an hour, or a logarithmic scale?

[mcguirepr89]

Hi, Frank! @DD4WH

In order to really appreciate that, I would like to take the bird.db from a former installation and put that into the new installation. However, I did not manage to do that, because it seems the bird.db gets renewed and filled with 0 bytes as soon as I do that.
Surely, I have missed instructions how to get the database into a new installation. Could anyone provide me with a link ?

I can confirm that you have found a bug :( -- a crappy bug.

I can confirm that the process of restoring a database onto a new installation works (take a look at my installation right now, as I've imported @CaiusX's data to my installation for testing), but I was unable to use the "File Manager" to accomplish this task.

I think, this means that the PHP upload feature in the "File Manager" is broken. To confirm, I suspect you will be able to restore the old database to a new installation using WinSCP to just copy the old birds.db to the new system. I'm about 90% sure this will work for you (as the protocol I used to transfer @CaiusX's data to my machine was scp).

I look forward to hearing from you about whether WinSCP allows you to restore the database with the ease and simplicity it should require.

[DD4WH]

Patrick, thanks a lot!
I am not sure if I do the right thing:

• my aim is to replace the fresh (empty) database of a new installation with the database of an installation that had been running for a couple of weeks
• what I do is: copy the bird.db from the old installation to my PC desktop
• open WinSCP and connect to the new installation --> copy the bird.db to the scripts folder, thus replacing the fresh (empty) bird.db
• now I can see many more detections and many more species displayed on the overview page
• I got to Species Stats in order to view data from the "old" database
• however: I cannot change the date range, which is still restricted to the date range of the new installation

[mcguirepr89]

I noticed the same thing during my testing and can confirm that this is the streamlit cache. You can go to "Tools" > "Services" and restart the "Streamlit" service to have it reload the cache

[DD4WH]

Perfect! Works now!
So, the recommendation for transfering a database from one install to the other is:

• copy bird.db to your PC
• use WinSCP to copy it to the scripts folder in the new installation
• run Restart Streamlit Service

Thanks a lot, Patrick!

Fantastic statistical graphs thanks to @CaiusX !

[mcguirepr89]

Yes, the Species Stats page is AMAZING now -- it is so cool seeing migratory data from @CaiusX's db and to pick up on trends within the day. I am really looking forward to someone setting up a node network that could then be used to have some really localized migratory (presence) data to know how the birds move around within an area over time. Fun fun fun! I hope to see some screen shots of your Species Stats soon :)

[mattluttrell]

Hi everyone! This past weekend I used an old webcam and old Pi I had laying around, ran the install script, opened a window, tossed the webcam out the window and closed it. Everything worked great!

Testing has been gone great and I have decided to upgrade my hardware. @DD4WH I have ordered microphones similar to what you suggested (had to use slightly different models due to lead times), @mcguirepr89 suggested a better power supply for my 4B+ to get around the low voltage errors. I've also ordered the foam wind shield. So I might do stereo microphones, with foam shields spaced at the appropriate distance.

Either way, I'm the site in Norman, Oklahoma (or Noble depending on your map). Outside my windows at 3am I have a bunch of coyote puppies and a large Great Horned Owl right now. Hope this starts appearing in my data. This is a pretty exciting project with highly motivated people. I'm excited to be a part of it!

[mcguirepr89]

You also helped us reach a new milestone, which is that you're the project's 50th star gazer! 🎉
And the 9th BirdWeather station powered by BirdNET-Pi!! 🎉

[mcguirepr89]

Hello everyone!! I hope this finds you all doing well ~~

@DD4WH Frank and I have been testing BirdNET-Pi on the Raspberry Pi 0W2 for the last few days and the results have been a pleasant surprise -- you can take a look at my example installation here to get an idea of how the system responds on the RPi0W2 hardware.

I wanted to invite anyone else interested in testing BirdNET-Pi on the Raspberry Pi 0W2 hardware to go ahead and give it a go. I've put together a brief set of instructions that cover the few changes that need to be made for the system to work on the RPi0W2 properly.

In terms of testing, I'm really just interested in user feedback so that I can land on exactly which steps to recommend and can properly set expectations for folks interested in running BirdNET-Pi on this hardware.

Lastly, but certainly not least, I wanted to thank my new sponsors for their kind donations to the project. Thank you all so so much!! And thank you to all of my testers!!!

My best,
Patrick

[Molokai]

Just quick feedback - recently installed the Main line on my rpi3b+ with the newest 64bit lite OS, and it went extremely smooth - and is very responsive and looking good so far.

[mcguirepr89]

@Molokai That's great!! I know you said you installed "with the newest 64bit lite OS," but I just wanted to confirm:
Did you use the latest image that requires that you setup a user? (I.e., are you using the newest image that has discontinued the default pi user?)

[mcguirepr89]

PS
I wanted to echo my offer to provide you with a subdomain to thank you for your testing ~~ If you are at all interested, I would be grateful to be able to add Finland.birdnetpi.com to the collection. You won't hurt my feelings if you'd prefer keeping your installation local only, I just wanted to be sure you saw the offer :)

[DD4WH]

Hi Patrick,

the new installation works fine. However, it gets stuck when I try to switch on Privacy mode:

System is responding, but no more recording/extraction etc.

Also, no way to revert my "Privacy mode" switching. Privacy mode is indicated as OFF, but recording/extraction is stuck.

[mcguirepr89]

Thank you! I swear I tested that, but I know that I'm wrong about testing it, because I just tried myself and encountered the exact same issue.

The fix is on line 48 in privacy_server.py:
This:

with open(userDir + '/BirdNET-Pi/thisrun.txt', 'r') as f:

needs to be changed to

with open(userDir + '/BirdNET-Pi/scripts/thisrun.txt', 'r') as f:

Then things will start working again, and then you can change the Privacy_Mode setting in the "Advanced Settings" again and it should work now

Sorry for the blip!

405ffbc

[DD4WH]

Perfect, works now!!!! Thank you!

[Svardsten53]

I have installed the latest version on my test system and found that Swedish letters are not displayed correctly after the last update. Sound files and spectrograms are also missing.

[Svardsten53]

I have done a new installation with OS Lite and now it works! It's a little strange that there is a difference between Lite and Full for how locale is treated.

I can live with the Lite version because I took a closer look at "File Manager" which is in Tools and it can be used for most things I want and for the rest I have to try with the command prompt.
At first, the list of files and directories looked a bit strange, but I changed the following in "/home/userid/BirdNET-Pi/scripts/filemanager/filemanager.php":
$ root_path = $ _SERVER ['DOCUMENT_ROOT'];
to
$ root_path = '/ home / userid /';
Then I got the root I am used to with all folders and files. Maybe this can be changed in the installation? It would also be nice to have "Show Hidden Files" set to OFF in the installation.

The date format can be changed to get the same as in BirdNET-Pi:
$ datetime_format = 'Y-m-d H: i';

[mcguirepr89]

That's great news that the installation is working for you

At first, the list of files and directories looked a bit strange, but I changed the following in "/home/userid/BirdNET-Pi/scripts/filemanager/filemanager.php":
$ root_path = $ _SERVER ['DOCUMENT_ROOT'];
to
$ root_path = '/ home / userid /';
Then I got the root I am used to with all folders and files. Maybe this can be changed in the installation? It would also be nice to have "Show Hidden Files" set to OFF in the installation.
The date format can be changed to get the same as in BirdNET-Pi:
$ datetime_format = 'Y-m-d H: i';

This actually a really great idea in general -- it would be best for users to also have as much system administration at their fingertips as possible, and access to a proper $HOME directory is a pretty essential piece of that puzzle. Thank you for this great tweak!!!

PS, you'll notice a few new tweaks from today (depending on when you did the installation, you may already be due for an update!)
A few of these things are things you've requested:

• Load 40 detections at a time in "Today's Detections"
• ALL pages now have the navigation bar, so no more clicking "BirdNET-Pi" to go back when you're viewing the spectrogram, logs, or species stats pages
• The spectrogram now updates using javascript instead of having PHP reload the page -- this is a much prettier spectrogram view!
• The live audio stream is muted when you want to listen to a recording while also listening to the stream, then the stream volume is resumed after listening to the requested segment
• The live audio stream can now stay on while changing pages, so no more pauses when loading special pages (there may be a few stragglers, but I'll get to them as we encounter them)
• You can navigate "back" in the "Recordings" view

[Svardsten53]

I tried an update, but got this:

fatal: cannot change to '/var/lib/caddy/BirdNET-Pi/scripts/BirdNET-Pi': No such file or directory
fatal: cannot change to '/var/lib/caddy/BirdNET-Pi/scripts/BirdNET-Pi': No such file or directory
fatal: cannot change to '/var/lib/caddy/BirdNET-Pi/scripts/BirdNET-Pi': No such file or directory
PRIVACY_MODE can be set to 'on' or 'off' to configure analysis to be more
sensitive to human detections. PRIVACY_MODE 'on' will purge any data that
PRIVACY_MODE=off

[mcguirepr89]

Hi, Lars,

Try this:

1. Go to "Tools" > "Web Terminal"
2. Type git pull
3. Go to "Tools" > "System Controls" > "Update" > "OK"

[Svardsten53]

Worked and I can see all the new changes. Great!

[Svardsten53]

I have noticed that daily_plot.py, which creates the Top 10 graph, is updated now and then. In the previous version, you could easily comment out "reading = xx" to get all the day's findings and I managed to get it pretty good except that if there were fewer than eight species, the labels at the bottom were not fully visible.

In the latest version, it is possible to specify the number of species you want to see, but have not been able to get a graph when the variable has been commented out. The problem is that the more species you want listed, the more compressed the list of species in the graph becomes as the height of it seems to be fixed.
I, and perhaps others, would like to list all the species of today. I would like to say that those who come further down the list are more interesting than those at the top, who appear in large numbers every day.

The ideal solution would be that you could specify the amount of species you want to include in the graph or a value for all species and that then the height of the graph is adjusted according to the species that are actually listed, ie not according to the maximum number you have specified.

I hope it can be solved.

The following screen dump lists 28 species.

[mcguirepr89]

I can try to take a look at daily_plot.py to see what I can do about that.

I think, though, the ideal solution from my vantage would simply be to keep the Top 10 species in the "Overview" to keep "Overview" succinct, and then add a heatmap plot to Streamlit "Species Stats" so that you can have the dynamic approach you're interested in (adjusting the number of species) and more (adjusting the date range as well). Streamlit will be the way to get the data visualized properly with more control and it avoids creating files (though facilitates exporting the data).

@CaiusX and I have discussed the development around Streamlit a good bit and it will also be able to pull in the "best recording" for the desired species and play the sounds etc.

Regarding the updates to overview.php, the most recent updates from today allow "Overview" to update the Top 10 Chart, Most Recent Detection, and Currently Analyzing as updates become available (thanks @ehpersonal38) without reloading the page. Updating the table values as they change ought to be coming soon, too

I'll let you know if I'm able to solve the issue with the Top 10 Chart growing and labels overlapping ~~

My best!

[Svardsten53]

Thanks for understanding and investigating how to solve this. The thing is to have a quick look in the first view. I know the bird observatory I'm working with really like the overview on the first page. It doen't need to be a fancy solution with installation options. I'm completely satisfied with changing value in variables or comment/uncomment some lines in the script. It would also be nice to have the possibility to show the total for each species.

Thanks again!

[CaiusX]

Hi

I think this is the version you're looking for - all bells & whistles?
I'm Not at a pi or computer for a while, so if it's wrong I'll look at it on Wednesday.

https://github.com/mcguirepr89/BirdNET-Pi/files/8116983/daily_plot_fix_abf.py.txt

Best

AP

[Svardsten53]

It's been quite a while since I got this special solution, but I and several others use it and we like to see a total picture of all species immediately when we enter the page. At the beginning of the script, there have been some changes to the database call itself (which I probably updated myself from the original script), but the actual creation of the graph is the same code.

Recently, a couple of users have gotten a decimal in the count for each species without making any changes to the script. We have tried to find what is wrong without success. Does anyone have an idea how to fix this?

[Svardsten53]

I have now managed to reproduce this error myself. It most certainly occurs when doing new installations. I did uninstall and then install on my test installation (no OS update) and then I also got the decimal. The script itself is unchanged, it must be something else in the system that is causing this. Here is the file we use (had to add txt extension to have it uploaded.

daily_plot.py.txt

[ehpersonal38]

I can't reproduce this on my machine with that script, so you're right it's probably something with the DB specific to each installation. Would casting value to int fix this?

ax.text(_x, _y, int(value), ha='center', va='center', size=8, fontweight='bold', color='darkgreen', bbox=dict(facecolor='lightgrey',pad = 4.0))

[Datore7]

It didn't work somehow with int(value), but I found a solution,
in this code, a "," (comma) is replaced by an "n"

`# Function to show value on bars
def show_values_on_bars(ax,label):
i = 0
for p in ax.patches:
_x = p.get_x() + p.get_width()* 0.9
_y = p.get_y() + p.get_height() / 2
value = '{:.0%}'.format(label[i])
# Uncomment for Species Count Total
(old) value = '{:,}'.format(p.get_width())
(new) value = '{:n}'.format(p.get_width())

Change '{:,}' to '{:n}'`

I haven't found the combination of using integer and thousands separator yet.

Soultion here: https://docs.python.org/3/library/string.html#format-string-syntax

I found how to get thousand seperator, change the 'n' to ',g' like this: value = '{:,g}'.format(p.get_width())

[mcguirepr89]

Thank you all for your help with this project! I am going to be working on putting together a pre-installed image with the help from my generous sponsors who helped me get my hands on a large enough SD card to start working on a pre-installed image to make getting started with a BirdNET-Pi a little bit easier!

I will let you know when it is finished and ready to install~~

My best regards to you all,
Patrick

[mcguirepr89]

Howdy, gang -- it's been a while and oh so much has changed!!!!

Testing is certainly different these day -- the pace for the main project is breathtaking and the improvements have been fundamental. Though we still need a testing ground when our sites are on new territory, and my sights have been on a fundamental change to how the system does its dance.

A new user, @mzakharo, recently made a great suggestion, which is to utilize the system's virtual memory for the processes preceding analysis. Brilliant! This means that we can send the 24/7 recordings to virtual memory, run the analysis, and when there is actually data worth keeping, then and only then is the data written to the SD card.

I'm no mathematician, but were I to make my most educated guess, I would expect this change will cut down on SD card read/writes by a bout a zillion and half percent, carry the 2, and then you've got your savings right there on the receipt to show off next time you go shopping. <---- yep, I'm no mathematician, so I had to use a little grocery store math to finish my calculations, I hope it's not noticeable.

Testing wise, the system will really be organized so differently that I want to be thorough before even suggesting PRs to main. Right now, the changes (I think they're all included at this point) that need to be tested live in a new branch called new_cleanup (I wasn't very thoughtful with the branch naming, so its not semantically helpful). The changes are highlighted below:

• The new default directory locations for the RECS_DIR and PROCESSED are /dev/shm/BirdSongs and /dev/shm/BirdSongs/Processed, respectively.
  /dev/shm is a universally readable/writable tmpfs mount point acting as shared memory for applications. Anything placed therein is written straight to the virtual memory (RAM) and is available for use by any other application/process on the system. By writing our raw data to /dev/shm, we are able to collect and analyze the data without ever reading/writing from/to the SD card.
• Disk maintenance (the cleanup.sh script/cron job) has been very finely tuned. Now, instead of always keeping the 100 most recent RAW recordings containing detection data (the PROCESSED directory), the script adheres to a newly adjustable STORAGE_LIMIT found in birdnet.conf and "Advanced Settings." This variable can be set to exactly the amount of raw data you wish to keep on the system at any given time. The default is 50M of data. This means /dev/shm will not grow much larger than whatever is defined as the STORAGE_LIMIT, protecting your RAM from filling up. This allows users with minimal RAM to allocate just a tiny bit of the memory for the task (40Bytes is the smallest allowed STORAGE_LIMIT).
• Small adjustments to extract_new_birdsounds.sh that have been overlooked for a long time now allow it to always look in the correct RECS_DIR for Analyzed files.
• /dev/shm is properly monitored in "Tools" > "System Info" now to aid in testing this change.

If anyone is able to help out with these test, please let me know -- you should be able to test an installation with the following command (untested as of the time of this writing, but should work):

curl -s https://raw.githubusercontent.com/mcguirepr89/BirdNET-Pi/main/newinstaller.sh | sed 's/main/new_cleanup/' | bash

Please accept my very best regards and constant gratitude!
Patrick McGuire

[mattluttrell]

I kind of feel like I'd regret not saying at least once that this is one of the reasons to switch to UTC for all components. Generally these problems just get bigger and bigger until someone takes the plunge to switch over.

[DD4WH]

Ethan, sorry to cause you so much trouble!

it seems it still uses a different time zone . . .

Taking a break of one hour now . . . will be ready to test after then

[ehpersonal38]

Okay all I can advise you to do is set your timezone to the correct one with "raspi-config," and then adjust your time to be correct. If that doesn't work then I'm all out of ideas. Also do an update since I reverted some changes

@mattluttrell Good idea!

[DD4WH]

Ethan, thank you so much for all your hard work and debugging !
The option to go into the web terminal and manually modify time zone by raspi-config did not come to my mind. Shame on me . . .
Now it works perfectly and the time delta is 18 seconds!

Keep up the excellent work!

[ehpersonal38]

You're very welcome! I suppose I knew the issue was the timezone all along, I just wanted to add a redunduncy that would work regardless of timezone. I guess I just need to know when to just concede and use the easiest solution!

Also your spectro still looks a few seconds off (which is expected, there's a lot of variables here), but if you're interested in making it very accurate then I would recommend:

• Open spectrogram.php, search for the line "mic delay,"
• The very next line you'll see a part where 6.8 seconds is subtracted from some other numbers.
• You can try changing 6.8 to maybe 10, then trying again. And keep messing with the number until you're satisfied.

^ All of this is assuming that the labels are off by a consistent amount each time, but I don't know that that's true or not for your setup (from the screenshot it looks so?).

Happy birding! 🐤

[DD4WH]

and this is the Processed directory:

The problem seems to be that the BirdNET-Pi uses a US time zone, because I can set the time manually in the Settings or tick the box, but the labels of the files seem wrong!
They show 15:26, but 21:26 is correct.

[DD4WH]

not sure, maybe this has been caused by me while I changed from offline version of the image to WiFi and back.

I will install a completely new and fresh version and see whether the problem remains!

[DD4WH]

Yes, the problem remains on a fresh install, so it is probably related to the time zone problem . . .

[DD4WH]

I opened an issue for the time problem:

#354 (comment)

[DougWelch]

Folks, I just want to completely endorse Matt's perspective. Timezones are a rabbit-hole best avoided, at least on the collection and database record end. Cheers, Doug

…

On 6/9/22 11:12, Matt Luttrell wrote: I kind of feel like I'd regret not saying at least once that this is one of the reasons to switch to UTC for all components. Generally these problems just get bigger and bigger until someone takes the plunge to switch over. — Reply to this email directly, view it on GitHub <#11 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYGKVXPP2IYA4IPGPZTA73VOICXTANCNFSM5FSCXBRQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mcguirepr89]

I love you all and really appreciate your input and insights on the timezone trouble!!! I will, however, still probably use a non-UTC system timezone as the solution here. This is important because BirdNET-Pi is a server running many services and is not a script or a single service. I don't believe UTC will be in keeping with the user-friendly, you-don't-need-to-know-computers approach the system is meant to maintain. Most users will want their system reporting things to them with their correct timezone.

I do believe this truly is easy enough to accomplish -- to set the timezone for the entire system, one simply needs to symbolically link the already included desired timezone indicator found in /usr/share/zoneinfo to /etc/localtime. A drop-down selector next to the "Tools" > "Settings" > Set System Time that allows folks to simply choose their timezone the way they expect to set if for a Linux server will achieve what we need. The server-side script simply needs to ln -sf the selection to /etc/localtime and then the entire system (PHP, the system journal, cron jobs, everything) will be using the correct time.

With that said, I haven't heard any arguments yet for UTC, other than those against standard timezones -- the exception being that there is a "rabbit hole" that needs to be avoided -- since the rabbit hole ships with Debian and all the TimeZones anyone would ever need, I'm inclined to let the user reach into the rabbit hole and pull out the timezone they need to use?

I'm certainly open to more specifics and information around this from those willing to make a case, though you can see which direction I'm leaning for development at this time.

My best!

[jurriaan]

@mcguirepr89 How does using a local time zone for filenames work with BirdNET-Pi when DST is involved? Will it overwrite files because the filenames can be the same after the time moves back an hour for example?

[DD4WH]

I do not think so. Accidental overwriting of extracted audio files would only occur, if the very same bird species is detected with exactly the same confidence score exactly one hour (exact to the second!) after DST switching. This is extremely improbable and it would only occur at night time (2AM to 3AM) when very few birds are active.
So, I would think this is not a problem.

[jurriaan]

True, it is a tiny chance in that case, but it's a possibility. When having a processing backlog the chance of overwriting unprocessed files is a bit bigger. My main point is that during DST-transitions the timestamps in the filename are not unique, so there's a possibility of conflict, and a possibility of interpreting the data incorrectly since you don't know when the file was created exactly looking at the filename.

I know this is mostly a theoretical thing, but this is the reason that I would prefer using UTC timestamps in those filenames (or the DST offset, at least something that makes it possible to interpret the filename correctly).

[DD4WH]

Do you actually use BirdNET-Pi and have had time to look at your files and their species names, confidences and time stamps ? If yes, you will see that this problem is virtually non-existent.

For my purposes, I need time stamps in my local time zone and will accept that extremely unlikely situation. If it should ever occur, it will be deleting only ONE single detection event out of half a million which are present on my RPi at the moment.

But your mileage may vary . . . and you can easily change your software running on your specific system to use UTC instead.