[BUG] Scrambler fails with UDP multicast input because bitrate can't be calculated

[GeckoGB]

Bug description:

When using IP UDP multicast input, with either a full transponder MPTS or a single service SPTS source, scrambler fails at start with error message

Error: scrambler: unknown bitrate, cannot schedule crypto-periods

because none of the various bitrate calculations succeed.
Neither source tsAbstractDatagramInput, or tsInputExecutor bitrate calculation from PCR/DTS are able to compute it, so bitrate remains equals to 0. And so when scrambler wants to start to scramble data, without known bitrate it is unable to schedule crypto-periods.

How to reproduce:

Using tsecmg as ECMG:
tsp -I ip 239.1.1.1:1234 -P scrambler 1 -e 127.0.0.1:2222 --ecmg-scs-version 3 -s 0x01120000 --pid-ecm 0x0041 -O drop

Multicast source can be any MPTS/SPTS, for example a full satellite/terretrial tranponder, a single service.

Expected behavior:

Should be able to scramble SPTS/service from MPTS when using multicast source. Tested with SPTS at around 5 Mbits/s bitrate and MPTS at around 25 Mbits/s.
As a comparison, if the same multicast source is first stored in a file, and then tsp used with file input instead of IP multicast input, scrambler works correctly.

Errors and logs:

We can first see

* Debug: ip: initial buffer load: 7 packets, 1,316 bytes
  * ip: unknown initial input bitrate

and then a little later

* Debug: scrambler: adding standards MPEG to none
* scrambler: found service id 0x0001 (1), PMT PID is 0x0020 (32)
* Error: scrambler: unknown bitrate, cannot schedule crypto-periods
* Debug: scrambler: plugin requests termination
* Debug: scrambler: packet processing thread terminated after 1,970 packets, 1 passed, 0 dropped, 1,920 nullified

Environment:

• OS: Linux, macOS (tested), but probably Windows too
• OS version: Ubuntu 20.10, macOS 12.5
• TSDuck full version: 3.31-2761
• Installation type: both official binary or rebuilt

Additional information:

I've tried to explore source code and add more logs to spy various variables and try to understand what is going on. My understanding may be wrong, and/or i can have missed some details, but here is my analysis.

In tsInputExecutor, initAllBuffers first try to fill half of the data buffer and log the result in Debug mode. Then, initAllBuffers tries to get bitrate by calling internally « getBitrate » and outputs the result to log.

* Debug: ip: initial buffer load: 7 packets, 1,316 bytes
* ip: unknown initial input bitrate

We can first see that despite beeing called with a larger available buffer (up to 40 packets large), only a 7 packets are really received. And it remains true for all following calls to tsAbstractDatagramInputPlugin::receive : no more than 7 packets are received at a time.

The "unknown initial input bitrate" message is because call to tsAbstractDatagramInputPlugin::getBitrate returned 0, Which means that input plugin has not been able to calculate bitrate.

Then, tstspInputExecutor::initAllBuffers will try to calculate bitrate from PCR or DTS. But here again, both method fail, so bitrate remains = 0.

But anyway, loaded packets are made available to the next packet processor, and input bitrate is supposed to be propagated to next processors. But since it’s 0, not really.

Then, scrambler processor receives data, and it keeps asking repeatedly for bitrate (from input executor) each time it processes new incoming packets, but it stills get only 0.

So, when it finally starts to receive packets it wants to scramble (after PAT/PMT have arrived), it finally fails:

* Error: scrambler: unknown bitrate, cannot schedule crypto-periods

And processing is then terminated.

So the main reason scrambling fails is because bitrate never manage to be calculated before data to scramble arrived.

I've also tried to understand why bitrate can't be calculated.

In AbstractDatagramInputPlugin, getBitrate() tries to calculate bitrate only if tracked reception timestamps are different (_start_0 and _start_1). But when logging them, i can see this:

* Debug: ip: AbstractDatagramInput::getBitrate called with start0=1970/01/01 00:00:00.000 and start1=1970/01/01 00:00:00.000

So, _start_0 and _start_1 are both equals to "epoch".

And TsInputExecutors::initAllBuffers tries to call AbstractDatagramInputPlugin::getBitrate() only once at the very first packets reception, which fails to return bitrate.

Since this try failed, TsInputExecutors::getBitrate is supposed to then try to use PCR or DTS to calculate bitrate itself.

But PCR part is skipped, probably because if some packets are arriving before PAT and PMT, since PCR PID is still unknown, it's impossible to get it yet.

So the last resort is DTS, but since video's PID is also still unknown before PAT/PMT are received, it's also impossible to use it yet.

After that, and despite TsInputExecutors::main is supposed to call again getBitrate sometimes, there's no more call to getBitrate (probably because some conditions aren't satisfied, but i've not checked that). So Input plugin has no more chance to try to calculate bitrate.

Packets continue to arrive, and scrambler is still waiting for PAT/PMT to know what to scramble.

But as soon as PAT/PMT are discovered, scrambler immediately wants to schedule crypto-periods, and so needs to know bitrate. But since it has never been calculated, scrambler immediately aborts processing.

At this point, i've realized that IP input plugin has an --evaluation_time option to specify to force periodic re-evaluation instead of trying to use PCR/DTS. So i tried to use it with value of 1 to 5s.
The only change is that now _start_0 and _start_1 are no more equal to epoch, but they still have exactly the same value:

* Debug: ip: AbstractDatagramInput::getBitrate called with start0=2022/10/09 17:58:41.038 and start1=2022/10/09 17:58:41.038

My guess is that since only a very few packets are received at a time, reception is so fast that the time interval is less than a millisecond, rendering the bitrate evaluation impossible.

Also, input plugin's getBitrate is still called only once.

So i've tried to crudly patch TsInputExecutors::main to add systematic call to getBitrate, but this doesn't solve the problems:

• Input plugin still doesn't calculate bitrate because _start_0 and _start_1 are always identical.
• and PCR/DTS still can't be used

So it seems to me that the only possible solution would be to calculate UDP source's bitrate differently.

I hope my analysis is not to bad and could help you...

Thanks for your great work!

Gerald Bastelica
www.geckoandco.com

[lelegard]

Hi @GeckoGB

Congratulation for your deep dive into the code. We need to fix this.

To supplement your analysis, TSDuck never relies on the wall clock time to determine a bitrate because it is supposed to work indifferently with online and offline sources (eg. UDP and files). Bitrates are always computed from the distance between two PCR's (or DTS when PCR are not available). It works well with CBR streams, less with VBR streams.

Since many plugins (scrambler included) need an evaluation of the bitrate when they process their first packet, we try to estimate the bitrate of the TS before sending the first packet to the other plugins. This is why the first read operation tries to fill half of the global buffer, usually providing enough packets with PCR to estimate the bitrate.

But there are several observations here:

With stream-oriented inputs (files, DVB reception, most of them in fact), the first read operation actually completes when the requested number of bytes are read. With datagram-oriented inputs (UDP), a read operation completes when one datagram is received. And one UDP datagram usually contains 7 TS packets to fit into the MTU of most LAN's. This is why your command works with file input (the first read fills half of the buffer) and not with UDP: the first read returns 7 packets and there is absolutely no way to evaluate a bitrate from 7 packets, neither using PCR (there is at most 1) nor based on wall clock (the 7 TS packets arrive at the same time by definition).

There are two possibilities to fix this. The first one is generic, the second one is specific to the scrambler plugin.

First solution: Force the input executor thread to actually fill half of the buffer, reading more than once if necessary, to have enough packets to evaluate the bitrate. However, if an input plugin returns small amounts of packets at a time, there may be a reason. By default, the buffer size is 16 MB, half of the buffer is 8 MB = 64 Mb. With a realtime SPTS @ 2 Mb/s, it takes 32 seconds to fill half of the buffer. This would create a latency of 32 seconds in the transmission. Acceptable or not?

Second solution: Modify the scrambler plugin to patiently wait for the bitrate to be evaluated before starting to schedule the crypto periods. Less generic but maybe more convenient.

Note that, as a workaround, you can always specify the bitrate by hand using tsp --bitrate XX. But you need to know it in advance. Not always applicable.

[GeckoGB]

Hi,

Thanks for your very quick reply and your additional informations: i'm glad i'm not completely off the mark!

one UDP datagram usually contains 7 TS packets to fit into the MTU of most LAN's

So obvious i didn't realized it!

About your proposed fixes:

Force filling the buffer:
Beside the possibly large latency for low bitrate sources (which could be a problem), i wonder if it would work well in any case. Especially when dealing with live streams, when you will start receiving them anywhere in the stream, and especially with high bitrate ones (either SPTS or MPTS): it's possible that, either with half of the buffer fully filled, PAT/PMT and also enough PCR PID packets may not have arrived yet.
Of course, one could use --buffer-size-mb to use a bigger buffer, but since with random live sources, you don't really know in advance how large it should be to safely receive needed data, it would use too much memory, adding unneeded latency, so i believe it would be impractical.
It would be better to dump and refill buffer until you really have all that you need in it.

Modify the scrambler to wait for the bitrate to be evaluated:
Probably cleaner, and would finally be, more or less, similar to first solution if buffer is repeatdly filled/dumped until necessery data is here and bitrate evaluated.
But:

Since many plugins (scrambler included) need an evaluation of the bitrate when they process their first packet

That would imply that all those plugins should also be modified to wait, not only scrambler.

What do you think?

Gérald Bastelica
www.geckoandco.com

[lelegard]

with high bitrate ones [...] it's possible that, either with half of the buffer fully filled, [...] enough PCR PID packets may not have arrived yet.

Never seen that but still possible if 64Mb (half default buffer size) represent less than 500 ms of content, meaning the TS bitrate is over 128 Mb/s. And as you mentioned, it is still possible to adjust the buffer size with extreme streams.

It would be better to dump and refill buffer until you really have all that you need in it.

Not acceptable in the general case. When you process a file, you want the whole file to be processed. You don't want tsp to discard the beginning of the file.

Modify the scrambler to wait for the bitrate to be evaluated:
Probably cleaner, and would finally be, more or less, similar to first solution if buffer is repeatdly filled/dumped until necessery data is here and bitrate evaluated.

Not the right approach. The bitrate is only necessary to compute the amount of TS packets per crypto-period. In the startup phase, it is possible to start scrambling without knowing the crypto-period.

The duration of the crypto-period is not always respected, for instance when an ECMG does not respond or disconnects and the scrambler enters a degraded mode, extending the current crypto-period until all ECMG's are back and fully operational. This is the standard behaviour of many production MUX/scrambler and the TSDuck scrambler plugins simply mimics this. So, the scrambler plugin can work without precisely knowing the end of the current crypto-period.

Since many plugins (scrambler included) need an evaluation of the bitrate when they process their first packet

That would imply that all those plugins should also be modified to wait, not only scrambler.

Yes, and there is no general method for that. The behaviour of the scrambler plugin before the bitrate is know is very specific. Other plugins probably require a very different behaviour.

Generally speaking, this is the problem with very generic frameworks such as TSDuck: you cannot have everything working fine, automatically adjusting, in all configurations, without effort, and for free. Either you have specialized equipment which does one thing extremely well or you have a very generic tool which works with variable success in some combinations of situations.

[GeckoGB]

Never seen that [...] meaning the TS bitrate is over 128 Mb/s.

You're right, seems quite unlikely. I was just thinking (too quickly) that if PAT/PMT/PCR took a few seconds to arrive, buffer could fill up. But since PAT/PMT are occuring quite frequently, they will be there probably under a second in most cases. And usual transponders will take at worst case (50/60 Mb/s) around 1s to fill half of the default buffer. So yes, stupid remark, sorry.

you want the whole file to be processed. You don't want tsp to discard the beginning of the file.

I was thinking that usually, you have to wait for PAT/PMT before being able to do anything with other PIDs/packets, so most of the time, packets arriving before are of no use and could be dropped. But in some cases, like if already know which PIDs you want to process, especially for some more or less exotic streams without PMT, you can't discard data like that.
So, as a general purpose toolset, discarding data should not be done, except if explicitely allowed. So not a good idea here.

So about how to modify the scrambler, you said:

Modify the scrambler plugin to patiently wait for the bitrate to be evaluated before starting to schedule the crypto periods.

So do you mean starting to scramble anyway, and evaluate crypto-period/number of packets later when bitrate finally known using the PCR?

So how would you achieve that? Is it a modification you would consider to make?

[lelegard]

So do you mean starting to scramble anyway, and evaluate crypto-period/number of packets later when bitrate finally known using the PCR?

Yes

So how would you achieve that?

By writing some code 😃

Is it a modification you would consider to make?

Yes, but I cannot promise any date

[GeckoGB]

By writing some code 😃

I was considering some animal or human sacrifice, but it's probably a better solution! 🧐

Thanks!

[lelegard]

The problem has been fixed. When the bitrate is not initially known, the scrambling starts (with a warning) and the crypto-period scheduling starts when the bitrate becomes known.

[GeckoGB]

Great news, thanks a lot! Will test that ASAP...

[lelegard]

Automatically closed after 178 days without update and "close pending" label set.