If It use multiple audio groups in fmp4 format, startPTS will be incorrectly recognized.

[wise-kim-vr]

What version of Hls.js are you using?

1.4.12

What browser (including version) are you using?

116.0.5845.140 (official build) (x86_64)

What OS (including version) are you using?

mac os 13.4.1

Test stream

No response

Configuration

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90,
  "liveSyncDuration": 4,
  "maxMaxBufferLength": 4,
  "maxBufferLength": 4,
}

Additional player setup steps

We're testing a new ABR algorithm.
But, this behavior is not related to the new ABR algorithm.

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Prepare a fmp4 live stream. This composition was the audio and the video with multiple adaptive levels. Below is the mater playerlist I use.

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a1",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a1/a1.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a2",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a2/a2.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="a3",NAME="a0",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",LANGUAGE="ko",URI="a0_a3/a3.m3u8"

#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=3192000,BANDWIDTH=6384000,CODECS="avc3.640020,mp4a.40.2",RESOLUTION=1280x720,FRAME-RATE=60.000,AUDIO="a1"
v0_v3/v3.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=8192000,BANDWIDTH=16384000,CODECS="avc3.64002a,mp4a.40.2",RESOLUTION=1920x1080,FRAME-RATE=60.000,AUDIO="a1"
v0_v1/v1.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=1128000,BANDWIDTH=2256000,CODECS="avc3.64001f,mp4a.40.2",RESOLUTION=854x480,FRAME-RATE=30.000,AUDIO="a2"
v0_v5/v5.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=628000,BANDWIDTH=1256000,CODECS="avc3.64001e,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=30.000,AUDIO="a2"
v0_v7/v7.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=264000,BANDWIDTH=528000,CODECS="avc3.640015,mp4a.40.2",RESOLUTION=428x240,FRAME-RATE=30.000,AUDIO="a3"
v0_v8/v8.m3u8

2. Prepare by adding a new setting to the Network Condition provided by chrome's developer tools. Set the speed to 750kbps or 1Mbps and the delay to 100ms.

3. It induces a buffer stall phenomenon. For example, you're playing to stream for some time on a good network, and suddenly you limit the network to 750kbps option.

4. If there are no symptoms, unrestrict again and repeat step 3.

Expected behaviour

Restricting the network will cause the VIDEO LEVEL to switch.
The AUDIO GROUP specified by VIDEO LEVEL is also switched.
If all went well, it should have played normally.

What actually happened?

Audio level switches with video level. And causing Audio's startPTS to go back in time.
The PTS has gone back in time, but it still downloads the latest segment of the Index. until the PTS of the video and the PTS of the audio are similar. and it plays. But the synchronization of AUDIO and VIDEO is already broken.

This test is an extension of a past issue 5756.

Console output

Chrome media internals output

No response

[wise-kim-vr]

@robwalch please, check.

[robwalch]

Hi @wise-kim-vr,

I am unable to reproduce the issue. When switching variants with different audio groups containing bit-identical audio tracks of the same language, the switch is seamless. The live stream I am testing with has program-date-time so the audio playlists were aligned using the date values across playlists:

hls.js/src/controller/audio-stream-controller.ts

Lines 519 to 529 in d73801b

	if (
	!track.details &&
	newDetails.hasProgramDateTime &&
	mainDetails.hasProgramDateTime
	) {
	// Make sure our audio rendition is aligned with the "main" rendition, using
	// pdt as our reference times.
	alignMediaPlaylistByPDT(newDetails, mainDetails);
	sliding = newDetails.fragments[0].start;
	} else {
	sliding = this.alignPlaylists(newDetails, track.details);

If not present, the player will align on discontinuity sequence, and failing that media sequence number. All that is to determine which fragment to load from the new playlist. Regardless of which fragment is loaded, media timestamps will be used to align media after an audio fragment is loaded and parsed.

Please provide a sample stream that can be used to reproduce the issue and clarify these items in your bug report:

Audio level switches with video level.

This is expected.

And causing Audio's startPTS to go back in time. The PTS has gone back in time, but it still downloads the latest segment of the Index.

Back in time how? The latest segment of what index?

until the PTS of the video and the PTS of the audio are similar. and it plays. But the synchronization of AUDIO and VIDEO is already broken.

So it sounds like the playlist are not aligned somehow. But it is not clear from your report how or why. Please provide a sample so that I can see if it is an issue with your HLS asset or an HLS.js player issue.

[wise-kim-vr]

Hi. @robwalch

Our Origin and Edge Server is located in South Korea. As a result, maybe communication may not be smooth.

https://kdnl-skb-cbt-orisa-edge.play.kakao.com/sl9u19ntjldgf37rf3odvntuc/adaptive.m3u8?orisa-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0aWQiOiIxMjM0IiwiZXhwIjoxNjk3MDg1ODE0LCJpYXQiOjE2OTQ0OTM4MTR9.KsnWkLNW7hVQkKiWrZu0MI8r6P-RiXSvY_V94Z48z4c&orisa-edge-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJtYXhicHMiOjgwMDAsImV4cCI6MTY5NzA4NTgxNH0.DSqHWob5ZgWLHleUhT8CUmenkpjFhOp1J7PFzCaqLqA&bps=8000

In order to access this service system and playback live stream, you need to insert a cookie in the header or query params with a url.
See the code below for how to do it. The cookies I serve have an expiration date of one month.

let url = "https://kdnl-skb-cbt-orisa-edge.play.kakao.com/sl9u19ntjldgf37rf3odvntuc/adaptive.m3u8?orisa-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0aWQiOiIxMjM0IiwiZXhwIjoxNjk3MDg1ODE0LCJpYXQiOjE2OTQ0OTM4MTR9.KsnWkLNW7hVQkKiWrZu0MI8r6P-RiXSvY_V94Z48z4c&orisa-edge-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJtYXhicHMiOjgwMDAsImV4cCI6MTY5NzA4NTgxNH0.DSqHWob5ZgWLHleUhT8CUmenkpjFhOp1J7PFzCaqLqA&bps=8000"

let params = url.split("?")[1] || '';
function appendParam(url: string, params ? : string): string {
  if (!params || url.indexOf(params) > -1) {
    return url;
  }

  let parsed_url = url.split('?');
  let url_path = parsed_url.shift();
  let url_param = parsed_url.join('?');
  return url_path + '?' + params +
    (url_param ? '&' + url_param : '');
};
const config = {
  xhrSetup: function (xhr: any, url: string) {
    url = appendParam(url, params);
    xhr.open('GET', url, true);
  },
};

const videoEl = document.getElementById('video') as HTMLVideoElement;
const player = new Hls(config);
player.loadSource(url);
player.attachMedia(videoEl);

If you need access beyond one month, please make a request to me.

[robwalch]

Tried but can't connect to the remote host:

curl: (28) Failed to connect to kdnl-skb-cbt-orisa-edge.play.kakao.com port 443 after 75004 ms: Couldn't connect to server

Times out from the browser as well.

[wise-kim-vr]

Hi. @robwalch.
It looks like you've been blocked because It's accessing from overseas.
Can you access and playback the URL below?

https://live-orisa-edge-ak.play.kakao.com/rl23c6y6z7szrapl67g8y21u1/adaptive.m3u8?__token__=exp=1697090935~acl=%2Frl23c6y6z7szrapl67g8y21u1%2F%2A~hmac=927f82206fc2218dc1e4661693f6ebad78118c9cd913e295201c10cefa3311c6&orisa-edge-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJtYXhicHMiOjgwMDAsImV4cCI6MTY5NzA5MDkzNX0.v7t28zUm-ksTqQp0kLPkOJH1uoVqiSJcC4tO-KgYz_M&bps=8000

[robwalch]

Let me know how the fix in #5818 performs https://bugfix-align-on-switched-pla.hls-js-4zn.pages.dev/demo/

[wise-kim-vr]

Hi @robwalch
I have tested several times in environments where the problem was previously reproduced.
As a result, it was never reproduced. I'm very happy that it was fixed immediately.