track + trackComposite issues (writing audio).

[neilkinnish]

I'm noticing an issue with track and trackComposite.

I'll try and explain – I'm still investigating – (I have files which I can supply).

Example: Audio recording which is 32 seconds where the user spoke through out. Resulting file has the correct duration, but the audio is within the first half of the recording.

I can confirm that if I record the roomComposite (as well) or/and add video to the trackComposite then the audio playback is fine, although additionally I am seeing drift on occasion in this area.

My assumption right now is just how the frames are being written, but I haven't looked deeper at this stage. I wanted to raise this issue first to see if there is any knowledge of this before I look deeper.

[neilkinnish]

To clarify, the audio is not sped up, more like silent frames are being dropped. This doesn't seem to be consistent, other participant audio recorded at the same time seems to be okay. The recording devices are both standard macOS with default camera and mic.

[neilkinnish]

UPDATE: I've been looking a little deeper and it seems every audio only recording is affected.

[frostbyte73]

A few questions, since I'm having trouble reproducing this:
Are the users muting themselves, or just not speaking sometimes?
Which version of egress are you running?
What player are you using?

[neilkinnish]

Hi @frostbyte73 thanks for looking into this.

1. Muted: No, I have been testing on my own and with people just counting with pauses (no muting) I have some file outputs to demonstrate the issue if it helps.
2. Egress: We are currently on 1.4.5 but have also tested custom build from master.
3. Player: vlc, browser, etc. We are also checking/converting the files in ffmpeg.

Some additional info on some more recordings made today....

File reporting duration of 34 seconds, but actually playback is 24 seconds and different players will report either one (or in some cases both). VLC does playback to the full 34 (which is what we would expect - but most other players 24 - converting to other formats results in 24).

We have checked the output ogg file from the egress in opusdec and opusinfo.

opusdec is reporting quite a few Warning: Hole in data.

opusinfo is reporting:

WARNING: sequence number gap in stream 1. Got page 15 when expecting page 14. Indicates missing data.
WARNING: discontinuity in stream (1)

Hope this helps.

[neilkinnish]

@frostbyte73 I’ve been running a few tests with the sample builder and writer (actually I’m doing a quick test with pion sample builder, basic writer against livekit) very quick and rough. First I was logging silence data and then I adjusted to fill and this is solving the issue.

I’m not familiar with the differences in the livekit sample builder & writer, at a quick glance I can see it’s making different decisions, handling muted etc, but I think silence also needs to be filled.

[frostbyte73]

yup, currently working on mixing silence in

[frostbyte73]

fixed in #146

[neilkinnish]

Thanks for fixing so quickly @frostbyte73.

I've been testing and I am encountering broken audio when it goes from silence to sound (cut outs), I can replicate this in every test.

I tested an adjustment to the VP8 Key frame you are using...

VP8KeyFrame16x16 = []byte{0x10, 0x02, 0x00, 0x9d, 0x01, 0x2a, 0x10, 0x00, 0x10, 0x00, 0x00, 0x47, 0x08, 0x85, 0x85, 0x88, 0x85, 0x84, 0x88, 0x02, 0x02, 0x00, 0x0c, 0x0d, 0x60, 0x00, 0xfe, 0xff, 0xab, 0x50, 0x80}

To this ....

VP8KeyFrameSilence = []byte{0xf8, 0xff, 0xfe, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}

And this solves the problem for me.