Packaging webvtt subtitles into live stream #380
Comments
|
@jakubvojacek Don't worry about the multiple issues. We appreciate your participation in helping improve the packager. Just looking at the Next week I will can add some code that will classify the |
vaage
added
type: bug
|
The problem is that the vtt file is being generated by 3rd party tool and we have no control over it, therefore, I cannot remove STYLE tag. For VODs, we can just pre-process the vtt file but for live it's trickier. Thank you |
|
We have managed to update the subtitle generator to remove the STYLE tag. However, it is still not working. The MPD can be found on https://edge1.motv.eu/channels/ct24-9/live/ct24-9.mpd, when I attempt to play it in shaka player, it says it cannot find text segment. Without subtitles, it is playing without issues, so I am guessing it has something to do with the subtitles. The packager command is: Subtitle file is here https://edge1.motv.eu/channels/ct24-9/live/test.vtt and is growing (around 50% of events has subtitles available). What can be the issue? Is it correct, that the vtt file is having start times from zero and not from the current timestamp? Or does packager ignores the timestamps in file and just assumes that the subtitle should be shown at the time its appended to the file? Thank you |
|
@jakubvojacek Sorry I have not gotten to this yet. The packager uses the timestamps in the content. If the timestamps in the text does not align with the timestamps of the other assets, then there will be an issue. Looking at the content you provided and based on what you say, the text is not aligned with the video and audio. Which would explain why the shaka player can't find the segments. So going forward, the question is what should be done? It would be possible for us to use an offset to re-align the text with video and audio. However that would require you to know ahead of time (when writing the command) to know what offset you would need. Does a solution like that work for you? @kqyang Do you have any other thoughts on this? |
|
@vaage it would be tricky to use with an offset I guess. I would have to calculate the offset in before running packager and somehow manage to turn on the subtitle extractor right on time. But that will not happen, because before it starts to output data, it analyses the stream which takes variable time. The tool for extracting subtitles (ccextractor) supports timing either from 0 or unix timestamp (something like 423401:22:56.720 --> 423401:22:58.760), but I guess that is not supported by shaka packager as of now. The easiest solution would be if ccextractor team could add support for stream timestamp, I'll ask at their github whether they could consider that |
|
@jakubvojacek I think ccextractor already supports stream timestamp. This is what I get from their documentation: https://www.ccextractor.org/doku.php?id=public:general:command_line_usage |
|
@kqyang Thank you, I have tried these options but they do not seem to be working. No error is thrown but the output is still same
|
To ensure that we can parse content with style and region blocks, this change updates the parser to skip those blocks so that we can still parse the cues from a file. Full style and region support will be added later this year. Issue #380 Change-Id: I11b8fd862a108c27a5c67b15d4703532b44a1214
|
@jakubvojacek Have you talked to ccextractor team about this? |
|
@kqyang I tried, with no response unfortunately - CCExtractor/ccextractor#975 |
|
Hello, we have managed to update & fix the ccextractor ourselves and its now generating correct timestamps in vtt file. But still, it does not work with the packager. The command is The MPD is located here - https://goo.gl/R1wFti, but cannot guarantee how long it will work before it crashes. Attaching how currently the MPD looks like - ct24-9.mpd.txt Loos all the segments have same duration but theyre not groupped using the On the last line, I think, it's showing that it finalized and ended subtitles? But that should not happen for live, right? It should be neverending processing. This is how the files look like The problem when testing is that the subtitles are only broadcasted like 50% of time, not always. But I have been checking the file ( Thank you |
That is really weird. There must be something wrong. Do you see any logging on the command line?
That is a bit misleading, but it just means the init segment is written, so nothing alerting here.
No. The timestamp is not correct. It starts from zero which does not align with other streams and that is why it does not show up during playback. |
No, in the post, I have attached the whole output of the packager's command line. I have just restarted the packager and it's doing the same, the MPD has already over 4k lines in 11 minutes
Are you sure? As per the packagers output that I posted in the previous post, why did it took 2 minutes for the subtitle int segment to be written and why it did not write this for other streams? I see this message for other streams when the packager is stopping (killing the ffmpeg that pipes data into the packager). I did this just now to the packager that I started last night - see bellow the output please The
The link to the used vtt file is here https://goo.gl/Z4PLVs but seems pretty aligned to me, at least the way I understand it should be aligned in.
ffprobe is showing timestamp 36941.209611 which translates to a little bit over 10 hours, which is what the ccextractor is showing too. Is there something else that should be aligned? Thank you |
|
The finalized message appeared in the packager log again it happened about 20 minutes into packaging. After this point, no more m4s files were generated while other segmetns (audio & video) are still being created Current server time Subtitle folder Audio folder |
|
@jakubvojacek Ah, you are right. Thanks for your detail report! I looked into the issue again. There are three different problems.
|
|
I have opened a separate issue about text starting at zero and added some background about why we do that (#416). @kqyang could you jump over there and add your thoughts on it. I feel that this is going to be harder than we would like to solve. @jakubvojacek if you have any thoughts, please chime in on #416 as well. |
@vaage would love to help but not much of a C programmer. But happy to test anything that you guys prepare in real live production where subtitles support is actually required by law :) |
|
@jakubvojacek In Linux, pipe can be read and written like a normal file, so it may just work with ccextractor, i.e. you can create the pipe using |
|
@kqyang just tested using pipe. Was able to make ccextractor into pipe. But the packager only starts to read from the pipe when the ccextractor starts after packager. If I start ccextractor first, packager ignores the subtitles (and produces MPD without text track without throwing an error or waiting for the pipe). Upon restarting ccextractor subtitles started to appear in the MPD (but again without repeating Would prefer to not use pipe as an output from the ccextractor though. Was planning to use the generated vtt file to feed packager that packages the recorded EPG event. In case we would have to use pipe, we would have to have 2 instances of ccextractor running at all time, one for live writing into a pipe and one for catchup writing into a file, which isn't ideal. |
|
@jakubvojacek Glad to know pipe works for you. I understand that it is not ideal, but there does not seem to be a perfect solution here. An alternative solution is that we can add a flag to indicate the "input" will never terminate, forcing packager to keep retrying when reaching end of the file. I am not in favor of it as it is not very clean. We are planning to have native support for subtitles in TS sometime in Q3 / Q4 - but it will take some time.
Filed #417. Will have a fix soon. |
|
@kqyang will be happy to provide a sample file but since its live udp (pipe) input not sure what I should share. I have a short recorded stream that has subtitles in it that I could share (https://goo.gl/4hQHBf) that you can play in a loop + subtitles here In case you would like me to share anything else, please let me know |
|
@jakubvojacek It looks like all three issues in @kqyang post have been addressed. Are you okay with us closing this issue now? |
|
@vaage which has Also, if counting correctly, that would be around 120 000 m4s files in 4 hours long live buffer which is quite a lot of files for one directory. Can you please point out what I am doing wrong? Hope that it does not make any difference but the subtitles were fed to the packager as text file instead of pipe (see below my comment about pipes, please).
I somehow struggle with finding a way to make it work reliably. If I start the ccextractor that writes into the pipe before packager, the packager will not event start generating subtitles. If I restart the ccextractor when the packager is already running, sometimes it does kill the package with an error When writing into a regular file, I never have these problems, the only problem is the race condition that the packager after some random time finalizes the subtitles. Cannot the packager make an assumption, that whenever the input of audio / video is UDP, that it's a live content and that it should never finalize the subtitles? That would solve this case and there would be not need to add another flag for this, what do you think? |
|
@jakubvojacek You are right, there should be approximately the same number of audio/video segments as text and that number of text seems way too large. Thank you for checking this for us. Looking at the times in the segment templates:
So there is something weird with the text start time compared to the audio/video start times (@kqyang am I reading that right)? If I am reading the manifest correctly, it would mean that the text is starting well after the audio/video which would explain why the players would not be able to show any subtitles. |
|
@vaage now that I think about it again, this might be a mistake on my side. I was extracting subtitles from incorrect multicast. I was extracting them from the input multicast instead of from the one I transcode with ffmpeg that I feed to packager afterwards. But still unable to test really du to the another issue I mentioned in the previous test. When piping subtitles, it wont start in packager, when using file to provide the subtitles, the packager finalizes them right away, therefore I cannot confirm whether it's working now. |
|
@jakubvojacek Reply to your previous post, we think having a separate flag is better because
Anyway, we'll give it more thoughts before picking an implementation. For your use case, would it work if you redirect the output from ccextractor to UDP, possibly using netcat (https://serverfault.com/questions/386405/netcat-udp-file-transfer)? |
|
@kqyang just tested using netcat and if I understand what you're suggesting, it does not seem to be supported results in error Did you mean something else other then to use UDP as input for packager? Because that does not seem to be supported. |
|
That is what I meant, but you are right, it is not supported yet.. We'll think about either make text for UDP work or make it possible to not terminate when reaching end of file. |
|
Thank you, that will be very helpful, both options sounds nice, but I still would prefer the flag. I was trying to browse the source code to find if I could insert a return somewhere in before the finalize method, to avoid closing the file - so that packager would not stop reading the file and we could test that both ccextractor and packager can work together well and for long without crashing. Would something like this possible? Before you manage to come with a clean solution. |
|
@jakubvojacek Yes, certainly, here is the change you need: --- c/packager/file/threaded_io_file.cc
+++ w/packager/file/threaded_io_file.cc
@@ -174,12 +174,14 @@ void ThreadedIoFile::RunInInputMode() {
while (true) {
int64_t read_result =
internal_file_->Read(&io_buffer_[0], io_buffer_.size());
- if (read_result <= 0) {
+ if (read_result < 0) {
NoBarrier_Store(&eof_, read_result == 0);
NoBarrier_Store(&internal_file_error_, read_result);
cache_.Close();
return;
}
+ if (read_result == 0)
+ continue;
if (cache_.Write(&io_buffer_[0], read_result) == 0) {
return;
} |
|
@kqyang Thank you, hope I managed to apply the patch (here https://github.com/google/shaka-packager/compare/master...jakubvojacek:jakubvojacek-live-subtitles?expand=1) but it appears to not work the way it should. I will try to describe how it behaves now. Using applied changes, I no longer get the finalized message of subtitle track in the packager output. This is how the output looks like: The audio and video segments are being created correctly as usual but the subtitle files are not. The number of subtitles segments generated changes from run to run (probably depends on what is the timing of the first subtitle compare to start of the input / package), once it was 6, once 38, once 1000. But the number does not get higher afterwards. It almost seems like if the race condition was still happening? Or am I doing something wrong? Also, when there were no subtitles at the time when packager starts (the vtt file is empty, just the header in there), the packager will not even start the subtitle segment timeline and the MPD is then unplayable in both dashjs and shaka player. Even when some subtitle becomes available, it does not start. This scenario can happen quite often as the subtitles are available rarely on some channels. Attaching how the generated MPD looks like |
|
@jakubvojacek Looks like there is something wrong. Will investigate. As for
I am curious what is the problem you are seeing. I thought it was working in dashjs but didn't work in shaka player before the fix in shaka-project/shaka-player@64d72fe. Is that not the case? |
|
@kqyang that was the case with packaged VOD, not live content. For live it just ends up with I have prepared a working sample MPD here https://goo.gl/51EYWm. |
|
@kqyang thought of testing with https://nightly-dot-shaka-player-demo.appspot.com/demo and it does play, seems like the fix has resolved it for both live and vod. I guess I should report this at dashjs repo. Anyway, the problem is that the |
|
@jakubvojacek Thanks. Can you try if it works in Shaka Player nightly: https://nightly-dot-shaka-player-demo.appspot.com/demo/? |
|
Ooops, I posted the above comment before seeing your new comment.
Good to know! Yes, please report the issue to dashjs and keep us posted! As a side note, I wonder if ccextractor supports generating empty cues, which will allow packager to generate segments (albeit empty) for text. This will be another way to workaround the problem.
Look into it now! |
Thought of it as well, but could not it lead to another, yet unlikely, race condition? (not sure how does ccextractor work internally and when it writes to file). But when both packager and ccextrator would start right after each other
|
|
@kqyang please ignore my last comment regarding the packager not creating subtitle segments, this was an issue on my side, was checking a wrong commit in the glient sync command. It does now play and show subtitles on dashjs (https://edge1.motv.eu///channels/ct24-9/live/ct24-9.mpd). Thank you and I am sorry for the misunderstanding I created. Seems like only one issue to tackle is when there are no subtitles available. I will do some test with ccextractor generating empty cues tomorrow |
|
@kqyang after the vtt header, however, that did not help. Is it possible that packager is throwing away cues that are older than current stream time? If so, then it is hard to add an empty cue by the ccextractor, isnt it?Even if the ccextractor would be capable of adding an empty cue to the stream time + small offset (so that it would not interfere with real cues, if any), the packager would have to start in whatever the offset was, right? If would take longer, the cue would be older than current stream time and would be discarded. That is also a race condition, isn't it? Not sure I made myself clear, this is somewhat hard to explain. Can you please explain further about the empty cues as well as what timing should they have in order for the packager to work? I might have misunderstood something. Thank you |
|
@jakubvojacek ccextractor needs to generate continuous empty cues - not just one, with the correct timestamp extracted from the input. I don't know if it is supported in ccextractor right now, but it should be possible to do. Something like this: |
|
@kqyang the ccextracot does not support this, we tried adding the feature ourselves but it isnt ideal. To add empty cues (with just a space for example) would be doable for VOD. But for live, you never know when subtitle is going to appear in the stream. And generating empty cues would result in overlap and shifting subtitles in the player - the player tries to show all subtitles, even empty and shifts them down. Also, I talked to dashsj team and they're saying that the MPD generated is not even valid as per dash specs (Dash-Industry-Forum/dash.js#2708 (comment)). Do you think it would be hard to fix this error this in shaka packager? To somehow "emulate" one empty subtitle cue at the beginning, and that should be enough, right? After the first cue, the packager will keep on creating even empty subtitle files. Thank you |
|
A cue with payload '.' is not really empty and it does affect the display. There is a packager bug that it does not support 'real' empty cue, i.e. a cue without payload, right now: #433. When the issue is addressed, you'll be able to generate cues without affecting the display during playback.
It is actually pretty difficult. The problem is that packager does not know what timestamp should the empty subtitle cue be without a reference timestamp. And for the same reason, packager does not know how much empty subtitle files to create without a reference timestamp. One possible solution is to synchronize the text stream with A/V streams internally during packaging and get the current timestamps from A/V streams. That requires a lot of work and it is also difficult to implement correctly. I probably prefer supporting CEA-608/708 (I assume that is the subtitle format you have now) parsing directly in packager then we'll be able to get timestamps from the container. That also requires some amount of work, so it cannot be supported in the near term.
I agree with DASH IF folks that it is out of spec :). Unfortunately, we do not have much options here as we don't have any segments available. We can address #422 by skipping writing the AdaptationSet/Representation with empty SegmentTimeline if it is needed. Let us know what you think! |
|
Was hoping it would be possible to take the timestamp from audio / video stream (whichever is first added into the packager), that would resolve it I believe. Another option could be to add a parameter to the segment template, which would reference from which stream (file and or udp + index) to take the timestamp from? It is not just CEA-608/708 but other as well, right now I am testing mostly on dvb_teletext and later we're planning on supporting dvb_subttiles via OCR as well (not really familiar with what the standards are called). I guess that when #433 will be implemented, we might be able to make it work via doing what you suggested earlier - generating lots of empty cues periodically inside ccextractor and hoping that the packager, for whatever reason will not start before ccextractor. And also hope that players on all platforms will be able to handle the empty cues without moving/replacing existing cues down on the screen Thank you |
|
Doing some more tests and probably doing something wrong. I used the patch you shared like a week ago to never stop finalizing subtitle file (https://github.com/jakubvojacek/shaka-packager/commit/3d716b615e85a00dce1fe17225efe54105d3466c). I updated ccextractor to always include at the very beginning of file this: up to 59 seconds. I can start with zero timestamps because I am extracting the subtitles from ffmpeg that starts at the same time as the packager and ccextractor (there is race condition if one would start way faster than the other but lets disregard that for now). The packager picks up the subtitles and starts processing them, the output of packager is no message about finalizing the subtitle track so it should be processing indefinitely, right. But the problem is that only 17 subtitle segments were created in the subtitle folder and no more. Do you please know what the issue could be? My understanding was that once the packager starts processing the subtitles (picks up at least one valid cue), it will continue generating them indefintely. The MPD looks like. Also, how will packager react to a case where I will have 59 empty cues up to 59 seconds and after that the ccextractor will write some cue at second 10. Will it crash or will it ignore it? If ignore, thats better for use. We can live without having correct subtitles for 59 seconds after start for live content. If it would crash, would it help if we were had 1 very long cue ( |
[1] Unfortunately it is not a one-time thing. It needs to get the time from an audio / video stream periodically and continuously. There are a lot of challenges. Will think about it more to see if it is possible to make it work.
Don't worry about it. Packager will strip all the empty cues unless there are cues in the segment then the whole segment is an empty cue. This ensures that the actual cues are always correctly displayed no matter how it is handled in the players.
That is expected. WebVTT allows overlapping cues, so packager will hold the last cue (59th cue) until seeing the next cue, no segments will be created during this time frame. So it is important to generate continous non-stopping empty cues for it to work.
I believe packager will return with an error as time is not allowed to move back. The time for empty cues should reflect the time encoded in the media, which ccextractor has access to. That is one of the challenges of the solution [1] above as packager does not have text time information when there is no text cues; the time hint from audio/video may not align with the time from text exactly. |
Ok, now I guess I finally understand, thought that it just needs subtitles at the beginning. Will see what we can do for that in ccextractor, not an ideal solution but any solution is perfect now :) Thank you |
|
@jakubvojacek That is great! I am curious how do you get the timestamps though in the python script? |
System info
Operating System: latest debian
Shaka Packager Version: packager version daac686-release
Issue and steps to reproduce the problem
Trying to supply our live stream with subtitles that we continuously generate into vtt file
Packager Command:
Extra steps to reproduce the problem?
(1) this is the test.vtt test.vtt.zip. This file is growing along with the stream (once we detect subtitles from dvb stream, we add them into the test.vtt file)
What is the expected result?
Having live mpd manifest with subtitles
What happens instead?
It ends with an error
[0412/205943:ERROR:packager_main.cc(473)] Packaging Error: 14 (INTERNAL_ERROR): Failed to parse WebVTT source. See log for details..Is there a way to make this work? I am sorry for multiple questions regarding subtitles but the documentation is quite brief on this matter. Once I grasp this concept, I can try to make a PR to the doc.
Thank you
The text was updated successfully, but these errors were encountered: