x9k3 not create .m3u

[sergeyshulga]

Hello. I try to use x9k3 segmenter but have no success.

First problem - its not create .m3u manifest file.
OS enviroment - Win7. I try to use Pypy3 or Python3, with admin rights - no success. Any suggestion?
Also its possible to select target directory for m3u and ts segments?

Second - it crash after some time with the same error.

Thank you.

[futzu]

do a git pull
and then try:

python3 x9k3.py -i https://slo.me/longb.ts

output should look like

seg0.ts         start: 1.433000 duration: 5.339
seg1.ts         start: 6.772000 duration: 2.369
seg2.ts         start: 9.141000 duration: 2.969
seg3.ts         start: 12.110000        duration: 5.339
seg4.ts         start: 17.449000        duration: 5.339
seg5.ts         start: 22.788000        duration: 5.338
seg6.ts         start: 28.126000        duration: 5.339
seg7.ts         start: 33.465000        duration: 5.339
seg8.ts         start: 38.804000        duration: 5.338
seg9.ts         start: 44.142000        duration: 5.339
seg10.ts        start: 49.481000        duration: 5.339
seg11.ts        start: 54.820000        duration: 5.338
seg12.ts        start: 60.158000        duration: 5.339
seg13.ts        start: 65.497000        duration: 5.339
seg14.ts        start: 70.836000        duration: 5.338
seg15.ts        start: 76.174000        duration: 5.339
seg16.ts        start: 81.513000        duration: 5.339
seg17.ts        start: 86.852000        duration: 5.338
seg18.ts        start: 92.190000        duration: 5.339
seg19.ts        start: 97.529000        duration: 5.339
seg20.ts        start: 102.868000       duration: 5.338
seg21.ts        start: 108.206000       duration: 2.770
seg22.ts        start: 110.976000       duration: 2.569

Let me know if that works.

[futzu]

I found what I believe to be the issue.

Please upgrade threefive to 2.3.33 , I fixed a weird bug in the threefive.Stream class.

python3 -mpip install --upgrade threefive

pypy3 -mpip install --upgrade threefive

Let me know if this fixes it.

[futzu]

Talk to me Goose.....

[sergeyshulga]

Sorry for late reply. ))
Updated threefive to 2.3.33.
Tested with your TS (https://slo.me/longb.ts) - all working fine, segments and m3u created.
On my TS with Mpeg2 video its still not create segments and playlist. Sorry, i missed that x9k3 support only H.264/H.265.
After switch video to H.264 its working ok, creates segments and m3u, but have some messages in log like "pid: 0x3e9 last cc:13 cc:4". What that mean? Paket loss?

[sergeyshulga]

Also when i have UDP TS on x9k3 input, created segments not spliced correctly - it have mssing/dropped frames on the start/end of segments.
If i record my UDP TS to file (.ts) and put it to x9k3 input, it splices correctly - no drops or blury frames between segments.
Is possible to setup input buffer size for x9k3?

[futzu]

Thanks for getting back to me man.

1. Is the MPEG2 video codec valid for HLS? Apple's hls page says only 264 and 265, but the actual RFC does not specify the video codecs so I really don't know. x9k3 is currently only setup to handle h264 and h265 video codecs. For now, the MPEG2 video codec is not going to work correctly.

2. Sort of. The messages are continuity counter checks. you're seeing mpegts packet loss, but a few packets should not be enough to negatively impact the video quality as you describe.

• Are you using multiple video streams?

• Why are you using UDP? Is it a requirement? I use gumd for multicast and it works really well.

• What software are you using stream the video via UDP?

• I'm thinking I should have x9k3 use a larger receive buffer for the UDP socket connection like it does for multicast streams. I'll try that later today and let you know.

Don't worry man, we'll figure it out.

Adrian

[sergeyshulga]

1. Amazon support MPEG2 in HLS, but i think not needed to develop handling Mpeg2 in x9k3. Its not common used in 2022 ))
2. I not use multiple video streams, only 1 H.264 video + 1 AAC audio stream inside TS. Example of my stream https://www.dropbox.com/s/5qswgdnt22ykp6d/test1.ts?dl=0
   Tested x9k3 with multicast - it has much less (or totally no) CC errors. When switch to unicast - x9k3 always have CC errors.
   Probably need increase input buffer/cache in x9k3. (Maybe make it as command-line parameter?)
   I try sending stream via VLC, TSduck and other software - no difference.
   If i analyze my stream via TSReader, TSDuck, VLC - its no CC errors or dropped frames

Thank you for your time.
Serg.

[futzu]

We have two problems

1. Mpeg2 video codec support.

   • The problem is iframes, I am reading the h.262 spec to find out how to parse for them.

2. UDP

• Since multicast works for, I am going set the UDP sockopts for the receive buffer much the same as the multicast and enable broadcast support for the socket

#2, probably tomorrow.

#1 will take a few days.

[futzu]

2. UDP and X9K3

I was able to reproduce the screwed-up video that you described, as well as the bad continuity counters.
Updates to the reader function resolved both the playback and continuity counters.

I just push threefive v.2.3.35 with the fix.

python3 -mpip install --upgrade threefive

python3 -mpip install --upgrade threefive

let me know what happens.

Adrian

[futzu]

Much to my surprise, x9k3 does handle mpeg2 video

• I ran several tests today with mpeg2 video and it worked well.
• I think all of your problems were due to threefive handling UDP poorly. I believe I fixed that today with v.2.3.35

Let me know what happens.

[sergeyshulga]

You are the best! Unicast and multicast H264 are splitting now without issues.
But Mpeg2 not working on my side - tested with udp or .ts file
Example of Mp2 stream - https://www.dropbox.com/s/v5vrnlbqpqsc7xn/test2_mp2.ts?dl=0

Also - is possible to add two features?

1. Select target directory for m3u playlist and segments. Now they are stored in python dir.
2. Delete unused segments (segments that not listed in playlist) for --live mode. Currently segments not deleted and directory grows unlimited.

[futzu]

Do a git pull on x9k3, just to make sure were in sync

test2_mp2.ts works perfectly for me.

a@fumatica:~/scte35-hls-segmenter-x9k3$ pypy3 x9k3.py -i udp://192.168.1.154:5005
seg0.ts         start: 162.917311       duration: 2.000
seg1.ts         start: 164.917311       duration: 2.000
seg2.ts         start: 166.917311       duration: 2.000
seg3.ts         start: 168.917311       duration: 2.000
seg4.ts         start: 170.917311       duration: 2.000
seg5.ts         start: 172.917311       duration: 2.000
seg6.ts         start: 174.917311       duration: 2.000
seg7.ts         start: 176.917311       duration: 2.000
seg8.ts         start: 178.917311       duration: 2.000
seg9.ts         start: 180.917311       duration: 2.000
seg10.ts        start: 182.917311       duration: 2.000
seg11.ts        start: 184.917311       duration: 2.040
seg12.ts        start: 186.957311       duration: 2.000
seg13.ts        start: 188.957311       duration: 2.000
seg14.ts        start: 190.957311       duration: 2.000
seg15.ts        start: 192.957311       duration: 2.000
seg16.ts        start: 194.957311       duration: 2.000
seg17.ts        start: 196.957311       duration: 2.000
seg18.ts        start: 198.957311       duration: 2.000

[futzu]

Also - is possible to add two features?

1. Select target directory for m3u playlist and segments. Now they are stored in python dir.

It's on my 'to do' list. I don't really know Windows, and I have been concerned about handling Windows file paths, but since you're here, you can help me test it. I'll get on it.

2. Delete unused segments (segments that not listed in playlist) for --live mode. Currently segments not deleted and directory grows unlimited.

I hadn't considered this, but it makes a lot of sense, I'll add it.

[sergeyshulga]

Do a git pull on x9k3, just to make sure were in sync
test2_mp2.ts works perfectly for me.

Strange. I used last version of x9k3, but it not splice mpeg2 TS.

[futzu]

• I fixed the problem you just had.
• I also added a thread to process the packets of a queue, as to not slow down the reader.
  • I am not 100% sure it will work on Windows,

git pull

• version should be 0.0.93

x9k3$ pypy3

>>>> import x9k3
>>>> x9k3.version()
'0.0.93'
>>>> 

• run it the same way and see if the threading blows up.

[sergeyshulga]

Yes, its working now. Splices Mpeg2 as needed. Thank you.

[futzu]

That is super cool that the threads work on Windows.

Sorry about the "hey" everywhere, I was checking when a method was getting called,

[sergeyshulga]

Output dir option also workin fine ))

[futzu]

Output dir option also workin fine ))

Cool. I tried something new to handle Windows file paths and I was hoping it would work.
I'm still thinking about how I want to handle deleting files for live stuff.

[sergeyshulga]

As variant - delete segments (if exist) with number < MEDIA_SLOTS, when create each new segment.
Maybe code like that
os.remove(f"seg{self.seg_num-MEDIA_SLOTS}.ts")

[futzu]

I think deleting would be best done externally. especially when the source is UDP, it's just too easy to drop packets. Deleting will have to interrupt x9k3 from reading from the socket, parsing the data does the same thing and it was dropping packets, and that's why I have it parse in a separate thread now.

[sergeyshulga]

x9k3 not dropping packets when i try to use some my junk code before "seg_file = f"seg{self.seg_num}.ts"":
del_file = f"seg{(self.seg_num-MEDIA_SLOTS)}.ts"
del_uri = self.mk_uri(self.output_dir, del_file)
try:
os.remove(del_uri)
except OSError as error:
print("File can not be removed:"+del_uri)
But its duct tape for my case ))

[futzu]

Don't call your code junk, I like how you did that actually.

I was thinking about how to clear up segments left by previous runs,
something like

import os


def  del_segs():
    with open('index.m3u8') as m3u8:
        active = [line.strip() for line in m3u8.readlines() if line[0] not in ('#','\n')]
        avail = [f for f in os.listdir() if f.endswith('.ts')]
        [os.unlink(f) for f in avail if f not in active]

[futzu]

I think I need to make a segment class, make a new instance when I start a new segment, and put instances in X9K3.queue. that would make it easier and help me clean up the code a bit.

I'll get the delete in there for you, just give me a minute.

Have you tried the 0.0.97? I put it up yesterday.I replaced threading with multiprocess,
that way it uses multiple cores. It works pretty well.

[sergeyshulga]

Hmm, build 97 not working. Tested on file and udp.

[futzu]

No big deal, I saw something about pickling and multiprocessing on Windows,
You see on the last line where it says cant pickle <function C9K3.<locals>.reader ......

it calls functools.partial, which I believe uses pickling to cache function args

      def readr():  # 340425  ~64MB
            if not self._find_start():
                return

            for chunk in iter(
              
               #  functools.partial call here    

                partial(self._tsdata.read, self._PACKET_SIZE * self._NUM_PKTS), b""
            ):
                work_queue.put(chunk)
            return

Give me a minute and I'll put up another version without pickling.

[futzu]

0.0.98 -> Back to threads, added --delete.

git pull and tell me how it goes man.

[futzu]

FYI,
I modeled the delete after how you did it.
I just used self.queue[0][1].rsplit(",")[1].strip()
instead of f"seg{(self.seg_num-MEDIA_SLOTS)}.ts"
so I could keep deleting the segment in sync with removing
the segment data from self.queue.

( I really need to rename self.queue because I import the queue module. )

 def _write_manifest(self):
        if self.live:
            if len(self.queue) > MEDIA_SLOTS:
                if self.delete_segs:
                    drop = self.mk_uri(
                        self.output_dir, self.queue[0][1].rsplit(",")[1].strip()
                    )
                    os.unlink(drop)
                    print(f"deleting {drop}")
                self.queue = self.queue[1:]
            self.start_seg_num = self.queue[0][0]

           < snip >

[sergeyshulga]

Hello.. Tested b.98, its working ok with TS file, but not work with UDP.

C:\PyPy>pypy3.exe x9k3.py --live -delete -i udp://192.168.0.20:1234
Traceback (most recent call last):
File "x9k3.py", line 457, in
x9k3.exp()
File "x9k3.py", line 399, in exp
readr()
File "x9k3.py", line 389, in readr
for chunk in iter(
File "C:\PyPy\Lib_functools.py", line 81, in call
return self._func(*(self._args + fargs), **fkeywords)
File "C:\PyPy\Lib\socket.py", line 704, in readinto
return self._sock.recv_into(b)
OSError: [Errno 10040].....

MSDN about that error: Message too long.
A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram was smaller than the datagram itself.

[futzu]

Change _NUM_PKTS to 57425 and see if that works

https://github.com/futzu/x9k3/blob/main/x9k3.py#L57

_NUM_PKTS = 57425

[futzu]

Do me a favor and run this while x9k3 is running

tcpdump udp port <port_num>

replace <port_num> with the port of the udp stream

What you're looking for is the length

18:07:04.992922 IP 192.168.1.66.1046 > 239.255.255.250.8082: UDP, length 876   <--- length
18:07:04.992927 IP 192.168.1.65.1039 > 239.255.255.250.8082: UDP, length 1637

[sergeyshulga]

After changing _NUM_PKTS = 57425 its working. But what that value mean?
Option --delete also looks working OK...

TCPDump show length 1316 bytes of each packet.
19:18:03.258022 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316
19:18:03.261023 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316
19:18:03.264023 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316

[futzu]

NUM_PKTS is the number of mpegts packets x9k3 reads at a time. That's what I changed to get rid of your original problem.
I was trying to lower it in 0.0.98 to reduce initial lag of segment creation,

If that is your stream is on port 1234, it looks like multicast, you may want to try it with udp://@ instead of udp:// in the uri.

[futzu]

Man the bitrate on that stream is almost twice what it should be, you should really consider running it through ffmpeg first to reduce the bitrate and then piping it to x9k3. Cut it down from 6000 to about 3500, and your hls playback with improve dramatically without a loss in quality.

[sergeyshulga]

I use high bitrate only for test, currently it ~3Mbps. No difference in x9k3 behavior.
With _NUM_PKTS = 57425 its worked some time (not too long) and crash with error 10040. I try reduce NUM_PKTS, less value -> crash earlier with the same error 10040.
Current workaround - not use NUM_PKTS value in chunk reading.
I replaced string
partial(self._tsdata.read, self._PACKET_SIZE * self._NUM_PKTS), b""
to
partial(self._tsdata.read, self._PACKET_SIZE), b""
and its seems working fine.

But sometime have error and thread stops:
C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234
seg120.ts start: 888.053322 duration: 5.300
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\PyPy\Lib\threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "x9k3.py", line 383, in workr
self._parse(item[i : i + self._PACKET_SIZE])
File "x9k3.py", line 365, in _parse
self._mk_segment(pid)
File "x9k3.py", line 268, in _mk_segment
self._write_manifest()
File "x9k3.py", line 240, in _write_manifest
os.unlink(drop)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: './CUE-O
UT=YES\n#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf/9+AAAAAAAAAAAAAD4xJP8="'

C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234
Splice Insert
Splice Point Splice Insert@159.429322
seg44.ts start: 159.193322 duration: 0.244
deleting ./seg34.ts
seg45.ts start: 159.437322 duration: 2.576
deleting ./seg35.ts
seg46.ts start: 162.013322 duration: 2.280
deleting ./seg36.ts
seg47.ts start: 164.293322 duration: 2.360
deleting ./seg37.ts
seg48.ts start: 166.653322 duration: 2.160
deleting ./seg38.ts
Splice Insert
seg49.ts start: 168.813322 duration: 2.240
deleting ./seg39.ts
seg50.ts start: 171.053322 duration: 2.060
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\PyPy\Lib\threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "x9k3.py", line 383, in workr
self._parse(item[i : i + self._PACKET_SIZE])
File "x9k3.py", line 365, in _parse
self._mk_segment(pid)
File "x9k3.py", line 268, in _mk_segment
self._write_manifest()
File "x9k3.py", line 240, in _write_manifest
os.unlink(drop)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: './CUE-OUT=YES\n#E
XTINF:0.92'

C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234
seg43.ts start: 157.053322 duration: 2.140
deleting ./seg33.ts
Splice Insert
Splice Point Splice Insert@159.429322
seg44.ts start: 159.193322 duration: 0.244
deleting ./seg34.ts
seg45.ts start: 159.437322 duration: 2.576
deleting ./seg35.ts
seg46.ts start: 162.013322 duration: 2.280
deleting ./seg36.ts
seg47.ts start: 164.293322 duration: 2.360
deleting ./seg37.ts
seg48.ts start: 166.653322 duration: 2.160
deleting ./seg38.ts
Splice Insert
seg49.ts start: 168.813322 duration: 2.240
deleting ./seg39.ts
seg50.ts start: 171.053322 duration: 2.060
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\PyPy\Lib\threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "x9k3.py", line 383, in workr
self._parse(item[i : i + self._PACKET_SIZE])
File "x9k3.py", line 365, in _parse
self._mk_segment(pid)
File "x9k3.py", line 268, in _mk_segment
self._write_manifest()
File "x9k3.py", line 240, in _write_manifest
os.unlink(drop)The filename, directory name, or volume label syntax is incorrect: './CUE-OUT=YES\n#E
XTINF:0.92'

[futzu]

This will make it easier.

get v.0.0.99 and use ffmpeg to get the stream

• --live with -- delete

# replace input.ts with your input stream

ffmpeg  -re -copyts -i input.ts -map 0 -c copy -f mpegts - | python3 x9k3.py --delete 

• --delete implies --live

• --output-dir will now create the directory if it does not exist.

[futzu]

You can use anything, curl, multicat, ffmpeg, tsduck, whatever you like and pipe it into x9k3.
--live and --delete and --output_dir all work

This issue is resolved.

[sergeyshulga]

Hello. Sorry for delay with answers. I'm not use file for test, always use UDP live stream with --live option. No difference unicast or multicast in x9k3 behavior.
I tested v.0.0.99, and have some issues - dont know is it related to Windows or anything else...
Sometime its crashes - but i need more testing to understand when it happened.
Some time its skips SCTE35 messages or not correctly write to manifest(but when i catch UDP stream it with TSDuck - all messages seen).
Example - i have live stream with clips and only 2 messages - CUE-OUT and CUE-IN immediatly.
In firrst time its writtent to m3u Ok, but second occurence x9k3 not correctly write CUE-OUT command to m3u (Command totally the same as first entry).
Sorry if i too annoying ))

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:1

Splice Insert

Splice Immediate

Splice Point @ 223.853333

#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf/9+AAAAAAAAAAAAAD4xJP8=",CUE-OUT=YES
#EXTINF:2.248,
seg1.ts
#EXTINF:5.752,
seg2.ts

Splice Insert

Splice Immediate

Splice Point @ 231.757333

#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf39+AAAAAAAAAAAAAMgDgSY=",CUE-IN=YES
#EXTINF:2.152,
seg3.ts
#EXTINF:5.768,
seg4.ts
#EXTINF:5.28,
seg5.ts
#EXTINF:5.28,
seg6.ts

Splice Insert

Splice Immediate

#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf/9+AAAAAAAAAAAAAD4xJP8="
#EXTINF:5.28,
seg7.ts

Splice Insert

Splice Immediate

Splice Point @ 255.213333

#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf39+AAAAAAAAAAAAAMgDgSY=",CUE-IN=YES
#EXTINF:1.848,
seg8.ts
#EXTINF:5.992,
seg9.ts
#EXTINF:5.28,
seg10.ts
#EXTINF:5.64,
seg11.ts
#EXTINF:5.28,
seg12.ts