Re-read old manifest after crash

[peyoh]

Hello,
on edash-packager restart, it starts to write new manifest file. Then the old video/audio segments are
not reachable. Is there a way after restart to re-read the old manifest and then continue to work without
lose old data? Or is too complicated?

[kqyang]

It is complicated to maintain states across packager restarts. Even if it is supported, there will still be some gaps, which might not be desirable.

[peyoh]

Yes, there will be some gaps, but this will be a player's decision how to deals with them.
BRS/
Peyo

[peyoh]

Hi KQ,
do you have idea how this can be achieved? Because (as I mentioned before) for some reasons in streams from different sources is possible to have DTS defects or some other issues. Then the packager asserts. After the restart the old manifest is overwritten and all timeshifted content is lost.

BRS/
Peyo

[kqyang]

I have some thoughts on maintaining states across packager restarts. It is a complicated issue and difficult to handle it properly.

I am also interested in the packager asserts and DTS defects you mentioned. If there is some problem in the content, even if we restart the packager with states carried over, we may run into the same problems again.

You have probably mentioned it before, can you list all the stream defects you have seen? We can go through these defects to see if there is another way out of it.

[peyoh]

Hi,
The most popular error is bellow. Other errors are very rare and I haven't seen them from a long time:

[0406/100857:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (34359961568 vs. 34360760768)
#0 0x000000773f1e base::debug::StackTrace::StackTrace()
#1 0x0000006b09d2 logging::LogMessage::~LogMessage()
#2 0x000000af3e32 edash_packager::media::mp2t::EsParserH264::EmitFrame()
#3 0x000000af2f6f edash_packager::media::mp2t::EsParserH264::ParseInternal()
#4 0x000000af242f edash_packager::media::mp2t::EsParserH264::Parse()
#5 0x000000aea666 edash_packager::media::mp2t::TsSectionPes::ParseInternal()
#6 0x000000ae8613 edash_packager::media::mp2t::TsSectionPes::Emit()
#7 0x000000ae83ee edash_packager::media::mp2t::TsSectionPes::Parse()
#8 0x000000ad9e8c edash_packager::media::mp2t::PidState::PushTsPacket()
#9 0x000000adb456 edash_packager::media::mp2t::Mp2tMediaParser::Parse()
#10 0x0000006614d0 edash_packager::media::Demuxer::Parse()
#11 0x000000661b84 edash_packager::media::Demuxer::Run()
#12 0x0000004125a4 edash_packager::media::RemuxJob::Run()
#13 0x0000006f2a21 base::SimpleThread::ThreadMain()
#14 0x0000006f1e3a base::(anonymous namespace)::ThreadFunc()
#15 0x7f83b3529dc5 start_thread
#16 0x7f83b325728d __clone

[kqyang]

Ok, the decoding timestamp was moved back by about 8.88 seconds... Any idea how often does it occur?

[peyoh]

KQ,
this issue happens when from example

1. the primary transcoder fails and the secondary change it's state to active.
2. Another reason is when the primary source starts to increment continuity counter errors and the monitoring system switches to another more stable source.
3. And sometimes - we've just received such packet without any reasons.

In case with DVB-S, the second case is frequent because environment reasons.
In short - I've seen this assert twice per hour and twice per week.

[kqyang]

I see. Am I correctly to say that for both case 1 and case 2, the timestamp difference should be small, a couple of seconds or even less?

If that is the case, we might just ignore these frames until the timestamp catches up. For example,

Frame Number	1	2	3	4	5	6	7	8
Original frames with timestamp	X	X+1	X+2	X+3
Switched to another source					X+2	X+3	X+4	....

We can drop frames 5 and 6. Do you think it will work?

[peyoh]

KQ,
I think the idea is ok.
BTW,
The latest logs are:

[0331/231020:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (8590276832 vs. 8595084272)
[0331/231524:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (1422000 vs. 18302400)
[0331/231754:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (284400 vs. 14122801)
[0331/232526:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (180000 vs. 20660400)
[0402/111247:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (291840 vs. 5196480)
[0406/100857:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (34359961568 vs. 34360760768)
[0408/095706:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (8590157792 vs. 8593570592)
[0408/095706:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (8590161632 vs. 8593574432)
[0408/170607:FATAL:es_parser_h264.cc(320)] Check failed: media_sample->dts() > pending_sample_->dts() (277440 vs. 592326240)

[kqyang]

The timestamp difference in the last one is significant...

I'll prepare a testing code for you later.

[kqyang]

I've reopened #50. Let's track this problem there.

[peyoh]

Hi,
do you have ideas for this case?
If the restart is fast it is possible to wait for MpdBuilder.adaptation_sets_ to be filled again and next to reread the old manifest to set the old values in the newly created list elements. At least for SegmentTemplate startNumber and segmentTimeline list.
availabilityStartTime can be also set as the old one, even the earliest_presentation_time it can be recalculated from the XML format.
But of course such "hacks" can affect timeshift correctness then. Or the idea is stupid?

[kqyang]

As mentioned in #50, the solution we are going to implement is to start a new segment when seeing timestamp rolling back.

Unfortunately, we don't have resource to work on this yet. We may look into it late this year.

[peyoh]

Hi KQ,
your idea is to generate something like this?

        <SegmentTemplate timescale="90000" initialization="edash-video-1080.mp4" media="edash-video-1080-$Number$.mp4" startNumber="2427">
          <SegmentTimeline>
            <S t="8817032752" d="900000" r="4320"/>
            <S t="4118878160" d="900000" r="75"/>
          </SegmentTimeline>
        </SegmentTemplate>

Because this crashes exoplayer and not so much shaka-player:)

[kqyang]

Yes. May consider putting them in two different periods instead of the same period - which will be more complicated.