How do you test multimedia today?

[varsill]

Hello everyone!

We are building a tool for testing streaming pipelines.
A very early POC is already available - currently it's a simple TypeScript SDK you can use to stream via RTMP, play the HLS stream, and assert that video is synchronized with audio.
We have ideas about where to take it next, but before that it would be really helpful to know how you handle multimedia pipeline testing today. If you have any particular needs don't hesitate to inform us about them and we can consider adding them to our roadmap!

Below there is a couple of questions, however feel free to discuss anything related:

1. How do you test your multimedia pipelines today — manual tests before each release, automated regression tests on CI, or monitoring on production?
2. If A/V drifted by 200ms tomorrow, how would you find out?
3. What was the last bug you found after release that a test could have caught?
4. Have you ever tried automating multimedia pipeline tests and abandoned it? If so, why?

Any opinions and ideas are more than welcome!

[varsill]

To kick things off: the setup that got us into this was a pipeline combining a few streaming providers with some custom FFmpeg and Membrane workers.

Multimedia pipeline testing relied on manual tests before each release. What is especially interesting, some "multimedia assertions" like A/V sync weren't even written down anywhere - it was just assumed the tester would report it if audio got desynchronized or the stream took too long to start. As a result, bugs caused by things like an unexpected change in a provider's behaviour were only caught in production.

We started wondering what to do about that. The obvious first thought was to write some FFmpeg scripts to arrange tests on a dev environment and assert a few things (which is especially easy now with LLMs 😉) but scripts like that tend to be flaky and hard to maintain - so we thought we could provide some "solid" building blocks for such tests.