Not inserting cue markers.

[anuj998877]

Hi Adrian,

I think because of latest changes there is a bug where If I try to add Cue marker immediately with
below sidecar data it is not working.
printf '0,/DAhAAAAAAAAAP/wEAUAAAAJf78A/gASZvAACQAAAACokv3z\n' > sidecar.txt

Where as if replace 0 with Time that is less than next_start it is working.

I think bug is somewhere here if I am not wrong.
https://github.com/futzu/x9k3/blob/8ed2b65b3d667e0ec47e22ef8fd98917ef619160/x9k3.py#L375

[cooliobr]

i saw the same problem here.

[futzu]

hmmmmm, let me look into it. I just can't seem to get everything working at the same time.

[futzu]

if self.started <= splice_pts <self.next_start: we need that one.

https://github.com/futzu/x9k3/blob/8ed2b65b3d667e0ec47e22ef8fd98917ef619160/x9k3.py#L345

I'm thinking if we change self.next_start to self.nowit will work.

I hate markdown, I spend more time editing markdown that writing damn code.

[futzu]

https://github.com/futzu/x9k3/blob/8ed2b65b3d667e0ec47e22ef8fd98917ef619160/x9k3.py#L343

change that to :
if float(line.split(",", 1)[0]) ==0.0:

One other thing to keep in mind. I'm no longer clearing the sidecar file after it's read,
so if you do
printf '0,/DAhAAAAAAAAAP/wEAUAAAAJf78A/gASZvAACQAAAACokv3z\n' > sidecar.txt
more than once, when x9k3 compares the sidecar file to the last time it read it, it will be the exact same and it will ignore it. Just use the double redirect >>t to append to the file and you can do several times if you want. Or you could just blank it out yourself.

• append
  printf '0,/DAhAAAAAAAAAP/wEAUAAAAJf78A/gASZvAACQAAAACokv3z\n' >> sidecar.txt

• blank it
  printf '' > sidecar.txt

I'll do an update shortly.

[cooliobr]

good, thanks

[futzu]

I have to hold off on the update, I got something wrong on the splicing and I need to figure it out, but that fix should work for you.

[anuj998877]

hmmmmm, let me look into it. I just can't seem to get everything working at the same time.

Yeah completely understandable!

I hate markdown, I spend more time editing markdown that writing damn code.

Haha me to0, I think I will have to spend more time on github doing more opensourcing :)

[anuj998877]

I have to hold off on the update, I got something wrong on the splicing and I need to figure it out, but that fix should work for you.

Is the chunk size exceeding target duration problem related to the same issue?
cause I tested yesterday and many times the chunk size exceeded Target duration. Even with enabling Shulga Mode.

[futzu]

No, the issue was I'm an idiot. :)

I was sending a CUE-OUT without a CUE-IN and didn't understand why it was returning from the break.
So I finally realized this morning what I was doing and added an auto CUE-IN.

It will exceed target duration if your GOP size is too big.
Segments have to be cut on iframes if you want the video to play correctly.
Use a GOP size of 60 or better yet 30.

v0.2.29 fixes the sidecar issue.

[futzu]

I just pushed v0.2.31

[anuj998877]

Thanks Mate, My GOP size is 48 and my target duration is 4 seconds on ffmpeg as well as Umzz/X9k3, Sometimes I see it exceeding the segments size than Target duration is it anyway related to timer and reader classes? If not with this one.

…

On Thu, Dec 14, 2023, 8:43 PM Adrian ***@***.***> wrote: No, *the issue was I'm an idiot*. :) I was sending a CUE-OUT without a CUE-IN and didn't understand why it was returning from the break. So I finally realized this morning what I was doing and added an auto CUE-IN. *It will exceed target duration if your GOP size is too big.* Segments have to be cut on iframes if you want the video to play correctly. Use a GOP size of 60 or better yet 30. *v0.2.29 fixes the sidecar issue.* — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BB3OMKRNMXEVJY3OTROXBULYJMJRLAVCNFSM6AAAAABASXWLB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJWGAZTENBUHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[edward-rafalovsky]

Hi, there are still a number of problems with the correct CUE tag inserting. First of all, if we are talking about muxing a pre-transcoded stream with tags, transcoder always puts an iframe after the CUE tag. So in this case, chunk splice should be maid exactly at tag's pts. The second point is that there is also an "immediate" tags, which is set at the moment the command arrives (self.now).

[futzu]

Thanks Mate, My GOP size is 48 and my target duration is 4 seconds on ffmpeg as well as Umzz/X9k3, Sometimes I see it exceeding the segments size than Target duration is it anyway related to timer and reader classes? If not with this one.

I got to ask , how did you come up with 48? I That's kind of odd. I use a GOP of 60 for an iframe every two seconds, 48 is not going to line up as easily. You see what I'm saying?

[futzu]

Hi, there are still a number of problems with the correct CUE tag inserting. First of all, if we are talking about muxing a pre-transcoded stream with tags, transcoder always puts an iframe after the CUE tag. So in this case, chunk splice should be maid exactly at tag's pts. The second point is that there is also an "immediate" tags, which is set at the moment the command arrives (self.now).

We are talking about is echoing a PTS,Cue pair into the sidecar file while x9k3 is running,
for a splice immediate. Using 0 as the pts is shorthand for splice immediate, that was the problem,
a zero was not triggering a splice immediate. That's fixed .

Then we were talking about chunk size exceeding target duration and I explained that chunk size is dependent on GOP size, we can't hit target duration unless we have iframes where we need them.,

If there are a "number of problems" , then you need to send me the streams with the issues.

All the segments from x9k3 start on iframes, check them yourself.

[futzu]

@anuj998877
30 frames a second and GOP size of 48, I'm guessing your segment duration is tending closer to 5 or 6 seconds?
Is that what you're seeing? You would probably do better at 60 than at 48.

[futzu]

So with a GOP size of 48, here's how the iframes are placed.

iframes msnbcGOP48.ts

5354.066667 # 4 seconds would be 5358.066667
5355.666667
5357.266667
5358.866667 # this is 4.8 seconds, and as close as we can get to 4 seconds
5360.466667
5362.066667
5363.666667 # 4.8 seconds from last segment
5365.266667
5366.866667
5368.466667
5370.066667
5371.666667
5373.266667

[futzu]

here's GOP size 60 iframes, they line up every 4seconds

a@fu:~$ iframes msnbcGOP60.ts
5354.066667
5356.066667
5358.066667 # 4 seconds
5360.066667
5362.066667 # 4 seconds
5364.066667
5366.066667 # 4 seconds
5368.066667
5370.066667 # 4 seconds
5372.066667
5374.066667
5376.066667
5378.066667
5380.066667
5382.066667

[anuj998877]

Thanks Mate, My GOP size is 48 and my target duration is 4 seconds on ffmpeg as well as Umzz/X9k3, Sometimes I see it exceeding the segments size than Target duration is it anyway related to timer and reader classes? If not with this one.

I got to ask , how did you come up with 48? I That's kind of odd. I use a GOP of 60 for an iframe every two seconds, 48 is not going to line up as easily. You see what I'm saying?

Let me explain my setup.

There is ffmpeg process which reads input on UDP port and then generates ABR hls stream .
it has 25 fps and GOP was set to 48 by mistake, now it is set to 50 and now I am getting 4 seconds chunks as output.
Let's call it Original Stream

Above output is then passed to Umzz / x9k3 to insert SCTE 35 CUEs in the HLS playlist it is working really well for about half or a day so and then I start seeing Variable Chunk size in the UMZZ/X9k3 output where as the Original Stream is working fine without any problem.

here is my umzz command:
nohup umzz -i /hls/master.m3u8 -s sidecar.txt -t 4 -w 10 -l -o /hls/dai > log/umzz_output_15DEC_5pm.log 2>&1 &

and here is output log of umzz/x9k3: if u see here duration is quite constant but sometimes it is 60 or 64

Note : even with single X9k3 process I saw similar issue. In original Stream I don;t see variable chunk now.

/1/seg14278.ts: start: 85901.421333 end: 85905.421333 duration: 4.000000 throttling 3.98 /1/seg14279.ts: start: 85905.421333 end: 85909.421333 duration: 4.000000 throttling 3.97 /1/seg14280.ts: start: 85909.421333 end: 85913.421333 duration: 4.000000 throttling 3.96 /1/seg14281.ts: start: 85913.421333 end: 85977.421333 duration: 64.000000 throttling 63.96 /0/seg14166.ts: start: 85925.421333 end: 85929.421333 duration: 4.000000 throttling 3.96 /2/seg14217.ts: start: 85929.421333 end: 85933.421333 duration: 4.000000 throttling 3.94

[futzu]

@ edward-rafalovsky
@anuj998877

Are you using 0x9k3 .2.31 with umzz?
Here's what I get with a GOP size of 48 and frame rate of 30 and 4 second segments.
gop48.m3u8.txt

Here's what I get with a GOP size of 60 , frame rate of 30 and 4 seconds.
gop60.m3u8.txt

Your GOP has got to be a multiple of your frame rate if you want to be able to cut accurately on iframes.
There is no other way to do it It's not me, it's just math.

GOP size / frame rate = iframes per second

30 frames per second, GOP size of 30 , gives you 1 I frame every second.
30 frames per second , GOP size of 48 gives you 1 I frame every 1.6 seconds.
30 frames per second, GOP size of 60 gives you 1 I frame every 2 seconds.
29.97 frames per second, GOP size of 30 gives you 1 I frame every 1.001001001001001 seconds

You know the oplayer guys? They are the ones who did ll-hls, they say the same thing.

https://www.theoplayer.com/blog/optimizing-ll-hls-the-impacts-of-gop-size-on-viewing-experience

[futzu]

here is umzz doing 4 renditions, over a network, parsing SCTE-35 from the segments, all in sync.
umzz.log.txt
4 identical index.m3u8 files.
3index.m3u8.txt
2index.m3u8.txt
1index.m3u8.txt
0index.m3u8.txt

That is perfect.

[anuj998877]

@ edward-rafalovsky @anuj998877

Are you using 0x9k3 .2.31 with umzz? Here's what I get with a GOP size of 48 and frame rate of 30 and 4 second segments. gop48.m3u8.txt

yeah after updating x9k3 to latest version the variable chunk issue seems to be resolved, but with live option enabled the umzz is not clearing sidecar cues and hence cues are reoccuring, which I have fixed on my local environment. I tried to raise Merge Request but got this error remote: Permission to futzu/umzz.git denied to anujchourange. Please Ignore this as it is not that important.

Apart from above issue I found another one where x9k3 and umzz both were consuming lot of CPU after a keeping these process running for more than 18 hours. Here is link to the issue

futzu/umzz#11 (comment)

[futzu]

you're using the splice immediate with umzz, and that is kind of tough. umzz can't always push to all renditions fast enough. I would not use splice immediate with umzz.

What do you consider a lot of CPU? This sort of thing is CPU expensive, ffmpeg uses like 500% of my CPU, I haven't seen x9k3 above 20%.
Why is yours 69%, I have no idea, and you haven't given me enough information.
You need to prove me that the problem is on my end.
I have a job too man, you're taking a lot of my time and you aren't giving me anything to work with, I need to see the stream, and more information. What do you expect to do with that?

[futzu]

you could always just restart the process and continue it. That's why we added the --continue_m3u8
I'll leave umzz running for a while and see what I get.