Lag can cause Audio to get behind Video

[Sebanisu]

If there is lag that causes the sound to skip it will desync. The video will end up ahead of the sound. This mostly only happens to me on linux in my vm. But I imagine someone with a weaker pc might have this issue on windows.
It basically needs to keep the Dynamic Sound Interface fed with audio samples. I was trying to increase the buffer size sent to it. But if I sent something other than 32 samples (approx 128 bytes) it caused sound glitches. I'm unsure if there is a way to detect if my audio is behind.
Possible Solutions:
Just increase the amount pre buffered?
FFMPEG audio processing in it's own thread?
Something faster than Mashal.Copy to copy from the byte* to byte[];? //when I switched to using Marshal.Copy to copy to a byte[] then feed it to Dynamic Sound Interface it was noticeably slower. But when I filled a memorystream with the entire uncompressed audio it would eat a lot of ram and that ram wouldn't always release correctly. Maybe there is a middleground.

[Sebanisu]

Well I thought of one optimization while eating dinner. I bypassed a Marshal.Copy. By using a byte array to store the resampled sound data instead of a byte pointer. I didn't realize I could use fixed to point to the array.

fixed (byte* tmp = &_convertedData[0])
{
outSamples = ffmpeg.swr_convert(ResampleContext, &tmp, ResampleFrame->nb_samples, null, 0);
}

So it skips the whole convert byte* to byte[] bit. Might be able to do more of that in some other spots.

[Sebanisu]

On saturday night I did some small optimization. I minimized some coping to buffer. I put all the needed header data into a byte[]. So it's ready to go and just need to send it to ffcc. with the info of where the data is in the data file and the data file's name. On the step where it reads from the file it only reads enough to fill the buffer for ffmpeg and closes the file.

--edit--
This Introduced a crash that only happened when running through the audio channels. But it's fixed. Just had to Dispose before leaving fixed.

[Sebanisu]

So we got threading last week. And increased the buffer to 100 chunks of samples. That seemed to help. I thought of another change to make when I'm home.

I can track how many samples are sent to the audio buffer, subtract the precached buffer, divide that by sample rate. And that gives us the seconds of audio sent to buffer. I think I can use that to detect if video got ahead of the audio. And basically pause the video till audio catches up.

There might be a need to check number of chunks in the buffer. But I don't know if all chunks will always be 32 samples.

If you resize the window during playback it's an easy way to force this to happen.