Intel Quick Sync support (Smart Fades)

[odwulf]

Hello,

I'm running my stack as containers (Music Assistant container runs alongside Home Assistant container, no HaOS). Smart Fades now shows an error because of the lack of AVX2 support.

I use Plex on the same machine, and its music analysis for a large music library runs flawlessly thanks to Intel QuickSync hardware support. Is there any way for MA to use that? Would there be a point? Or am I misunderstanding something completely?

[chrisuthe]

Multiple things at play here.

First, what CPU do you have?

Second, you said "Containers" so I assume docker, which wouldn't mask flags, but if virtualization is any part of your stack please make sure your CPU is being passed through correctly so flags aren't being masked, if you have AVX2 you want to use it.

Finally, "Quicksync" (really other GPU accelerated processes- QS is a killer intel feature but isn't helping your audio analysis on plex or here):

Lets break it down by our providers:
Smart Fades (provider.py:87-111) is a pipeline of small models, not one nural-net we could use GPU acceleration for:

Beat This (Spect2Frames, "small0" checkpoint) — a compact spectrogram→frame CNN/transformer for beat & downbeat tracking, then dynamically quantized to int8 at provider.py:96-98.
DBN downbeat tracker (dbn_postprocessor.py) — a Dynamic Bayesian Network / HMM with a sparse CSR transition matrix and Viterbi decoding. This is classical sequential CPU math, not a neural net at all. The use of the quant version limits us to AVX2 support, even though it CAN fall back to a very very slow/memory/cpu intensive verison (we don't want that).
skey (VQT + ChromaNet) for musical key, and a SpectralCentroid torchaudio transform for energy.

Sonic Analysis (init.py:1) is Microsoft CLAP zero-shot + librosa scalars:

CLAP's HTSAT audio encoder runs over fixed 7-second windows at 44.1 kHz (init.py:57-67) — only 1, 3, or 8 windows per track depending on the quality setting.
The CLAP text encoder never runs at inference — prompt embeddings are precomputed and vendored as an .npz (clap_prompts.py:12). So per track, the GPU-eligible work is just a handful of ~7s audio-encoder passes.
Both run with device = "cpu", and the shared runtime (audio_analysis.py:169-206) deliberately caps torch/BLAS threads to ~25% of cores — it's tuned to be a polite background CPU citizen, not to saturate anything.

So, in total, we have some optimization to do (I think I saw plex was on V9 of their audio analysis) but we're taking a different approach in a lot of ways. Plex runs (last I knew) a completely custom tensorflow model on tflite (entirely CPU based and not supporting ARM) for their music analysis, we are working with a best of breed open source setup here where we're bringing in the best parts and pieces we can (and trying not to re-invent the wheel).

From Plex's Sonic Analysis Page:
Windows, macOS, and Linux (64-bit only) are supported (this includes most NAS devices) ARM-based CPUs are not supported (with the exception of Apple’s M1 systems, which use Rosetta 2 emulation)

I hope Analysis can work on your machine, it's really nifty, if you let me know the CPU and Virtualization setup I'll let you know what to expect.

[odwulf]

Thanks for the detailed explanations. I feel both smarter for what I learned and dumber for what went over my head.

The machine I run most on my stuff on is a NAS, with a Celeron J3455, which is not optimal, obviously, given this and that and its age. And though I can hack it a bit, I'm stuck to the vendor's Docker version and setup. Replacing it has its pros and cons, and I'm adding this to the pile, but for the time being, it's still doing most of what I need it for.

If I got it right, AVX2 support should show up in the flags displayed by lscpu if present. That's not the case, nor it is for any of the other machines I run parts of my stack on, so I guess I'll have to live without smart fades, which is something I've done most of my life. I'm trained.

I'll take that into account if we ever see the end of the rampocalypse and I decide to buy a new board.

Once again, thank you.

[chrisuthe]

Ah man, yea, J3455 is just that bit "too old" sadly.

I feel you, I ran 99% of my home lab on a Synology with a similar chip for a really really long time, it was great having a single box that wasn't amazing but got it done so to speak, now it's just an offsite backup target as it too doesn't have AVX2 so frigate, MA now, etc don't love it.

One think to consider, you can always add a box and use your NAS as storage, only migrate what you need to :)

[odwulf]

One think to consider, you can always add a box and use your NAS as storage, only migrate what you need to :)

Down the line, that's the idea. Though I really like how simple it is to have my apps and my media on the same disks. I'd have to study the best ways to handle huge video files through network storage.

Am I right to understand that AVX2 is and will stay Intel exclusive? No hopes for any ARM Smart Fades support?