Performance Comparisons and Optimisations

[Mattk70]

One of the benefits I claim for Chirpity is that it is fast. I thought it would be useful to share my own benchmark comparisons of the various models. I used these machines in the test:

• 13" Macbook Pro M2 (8GB ram)
• 8th Gen core i5, with 6 CPU cores, 32GB RAM and equipped with an NVIDIA GTX 1660 Super

Chipity default settings were used for each test:
For the CPU backend, the threads value is set to match the CPU cores - so, 8 for the M2, 6 for the i5.
For the GPU backend, the default is to use 1 thread
The batch size for both backends is 32.

The table below shows the time taken to process a wav file that is 3 hours and 6 minutes long. The number in brackets is how much faster than real time each result represents:

Model	Core i5 CPU	Core i5 GPU	M2 CPU	M2 GPU
Chirpity native	113s (99x)	58s (192x)	67s (167x)	172s (65x)
Chirpity BirdNET	148s (76x)	270s (41x)	81s (137x)	991s (11x)
BirdNET Analyzer	549s (20x)	N/A	N/A	N/A

So, the takeaways are that you should avoid the GPU backend on an M2 Macbook (and by extension I would say the M1 as well), and that a decent graphics card makes a huge difference to the Chirpity model, but not the BirdNET one. In this test, BirdNET ran nearly 4x faster on the PC in Chirpity compared to using the BirdNET-Analyzer GUI.

[Vollpflock]

I'm amazed by BirdNET's speed in Chirpity compared to the BirdNET-Analyzer GUI. And even more impressive how fast you've made Chirpity.

I ran a few tests with eight 1-hour long .wav files on my M1 Macbook Pro. It processed them in 160 seconds, averaging 20 seconds per 1-hour file – that's quite speedy! Not sure if that's comparable, though.

My settings were: CPU with 8 threads, batch size 64, using BirdNET.

The device I used is a MacBook Pro 16-inch, 2021 model, featuring a 10-core CPU with 8 performance cores and 2 efficiency cores.

[Mattk70]

For that kind of processing - lots of long files, you should be able to get another 5-10% speed up if you max out both settings: 10 cores, 128 batch size. See how it goes?

[DD4WH]

Very interesting, thank You for the comparisons!

My first experience of a comparison of Chirpity vs BirdNET is the opposite (I ran an analysis with more than 10,000 files of one minute length each and calculated the average processing time):

• BirdNET needs 0.7 secs on average for a 1 minute recording (8 cores)
• Chirpity BirdNET model needs 1.4 secs on average for a 1 minute recording (8 cores, everything else default settings)

So, BirdNET analysis has twice the speed of Chirpity BirdNET model for processing many small files.

Probably the opening/closing of the files seems to be the bottleneck in this special analysis configuration of mine.

I use one minute recordings as a standard in my field recorders, so that its very easy and fast to scroll through them and view the whole minute recording as ONE single spectrogram. But maybe I need to rethink that workflow!?

I will have time to test some more in the next weeks and will report here.

[Mattk70]

Thanks for the feedback @DD4WH ,

I have a scenario where I do something similar - with mp3 files of between 3-60 seconds duration. The default settings are tuned for long files, because Nocmig. Processing a batch of short files is of course slower, for the reasons you surmise: opening and possibly extracting the files.

However, tuning the batch size to the files' length is important in this case. With the default batch size of 32, Chirpity will create batches that comprise 96 seconds of audio (32 x 3 second windows). If the file is shorter than this, the batch is padded with silence (not always, but I'll leave that for another time). Silence or not, the model will create a spectrogram and make a prediction for those padded windows, so you'll be wasting a lot of compute.

With Chirpity, there is really no need to split your files this way. You can 'zoom out' the spectrogram - i.e. adjust the length of audio displayed, using the minus key. Holding shift (+ or -) will adjust the number of FFT samples used in the spec display, so you can sharpen the calls up if they are blurry. Page up / down will move you through the file one screen at a time.

If you still find processing slow after trying the adjustments I've suggested, you could post a screenshot of the diagnostics window found under Help > Diagnostics.

[Vollpflock]

I did some very rudimentary testing with the BirdNET-model in Chirpity, just for myself.

Settings:

Threads: 10
Batch size: 128
Device: M1 MacBook Pro 2021 16-inch
File type: .wav, each is 1 hour long and 158.8 MB in size.

Results:

100 x 1-hour files: Processed in 43 minutes 4 seconds, averaging 25.4 seconds per file.
300 x 1-hour files: Processed in 2 hours 21 minutes 7 seconds, averaging 28.2 seconds per file.

I observed the device overheating over time, with kernel_task CPU usage increasing to around 150%.

Post-analysis, the detections are accurate and the app performs smoothly without any lag, including experimental features. That is just awesome!

[Mattk70]

Hi @Vollpflock, interesting results!

Whilst I'd expect such a prolonged task to cause the CPU to get hot, I'd be worried if it was overheating. kernel_task is there to prevent that.

Inference - the analysis - is a very intensive operation. To achieve the speeds Chirpity does, it intentionally tries to maximise its use of the available compute resources. You are able to control this and run cooler (and a little more slowly) by reducing the number of threads in the settings.

[Vollpflock]

Got it, that's what I meant :)

[Mattk70]

A performance update, for v1.0.1. I used the same settings and file in the tests.

Model	Core i5 CPU	Core i5 GPU	M2 CPU	M2 GPU
Chirpity native	113s (99x)	28s (388x)	67s (167x)	172s (65x)
Chirpity BirdNET	148s (76x)	270s (41x)	75s (148x)	991s (11x)
BirdNET Analyzer	549s (20x)	N/A	N/A	N/A

So, not a typo, the native Chirpity model processed the 3 hour 6 minute file in 28 seconds. There is a more modest improvement to the Mac CPU performance too. I plan to remove the GPU backend option for BirdNET in the next release. It is rather pointless.

[Mattk70]

A performance update, for v1.7.0, in which oneDNN optimisations have been enabled. This leads to big processing improvements for PC users. For these tests, I used the same settings and file once more.

Model	Core i5 CPU	Core i5 GPU	M2 CPU	M2 GPU
Chirpity native	80s (140x)	28s (388x)	67s (167x)	172s (65x)
Chirpity BirdNET	106s (106x)	270s (41x)	75s (148x)	991s (11x)
BirdNET Analyzer	549s (20x)	N/A	N/A	N/A

This represents a 41% speed up for the Chirpity model and a 39.5% improvement for the BirdNET model running on an old PC with an i5 processor compared to previous Chirpity versions. More modern PC processors which support AVX_VNNI and FMA should see a 2x speed improvement!

EDIT: If you would like to run tests using the file I have used for benchmarking, I have made a zipped version of the file available here (N.B. it's 730MB).