Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PROPOSAL] Process closed captions and burned-in subtitles in one pass #726

Closed
cfsmp3 opened this issue Mar 22, 2017 · 27 comments
Closed

[PROPOSAL] Process closed captions and burned-in subtitles in one pass #726

cfsmp3 opened this issue Mar 22, 2017 · 27 comments
Labels
difficulty: medium GSoC-related HardsubX Issues related to extraction of HardsubX OCR

Comments

@cfsmp3
Copy link
Contributor

cfsmp3 commented Mar 22, 2017 •
edited
Loading

There's some subs that have both closed captions and burned-in subtitles. This happens typically when part of the audio is in English (which is closed captions so the viewers can enable/disable them) while part of the audio is in a different language, which is subtitled for everyone.

A very good example is the Americans, in which a good portion of the audio is in Russian.

This task gives ENOUGH points to qualify for GSoC. If you're proposal is good and your code to get this done gets merged, you're in.
@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Apr 12, 2017 •
edited
Loading

A sample video:

@saurabhshah0410
Copy link
Contributor

saurabhshah0410 commented Dec 21, 2017 •
edited
Loading

Is this issue still open?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Dec 22, 2017 via email

@thealphadollar
Copy link
Contributor

@cfsmp3 I would like to work on this. Can you please give me some tips for the solution?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 9, 2018

@thealphadollar First download the sample file we have for it, extract the subtitles in two passes (one for burned-in and one for closed captions), and then figure out how to do everything in one pass :-)

@saurabhshah0410
Copy link
Contributor

@cfsmp3 After running ccextractor once on files like these, and having extracted the closed captions as well as hard subtitles, can you clarify if the result should be a single file with both closed captions and hard subtitles, or two separate files which contain the closed captions and hard subtitles respectively?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 26, 2018 via email

@saurabhshah0410
Copy link
Contributor

Thanks for clarifying this @cfsmp3 . :-)

@saurabhshah0410
Copy link
Contributor

I think we should add a new option, something like hcc (hard and closed captions) or whatever name you suggest, which when given would process hard subs and closed captions in one pass. What do you think?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 27, 2018 via email

@saurabhshah0410
Copy link
Contributor

Thanks, just wanted to be sure about it. Just to play the devil's advocate: If a user gives both the switches: -hardsubx and -hcc, should we process both hard subs and closed captions? Basically, does -hcc takes precedence over anything else(like -hardsubx)?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 27, 2018 via email

@saurabhshah0410
Copy link
Contributor

Yeah okay. That makes sense, thanks!

@saurabhshah0410
Copy link
Contributor

@cfsmp3 What should be written to the output file if in a particular interval of time from t1 to t2, the video contains both hard subtitles and closed captions?

Although such a scenario would be very rare, as you wouldn't normally have closed captions as well as hard subs having overlapping time intervals, but if it happens, what should be done in case of an overlap between the burned in and closed captions?

Should it be like:

1
t1 --> t2
<closed captions>
<hardsubs output>

or

1
t1 --> t2
<hardsubs output>
<closed captions>

Or something else?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Mar 5, 2018

Maybe check if they are the same (likely) and then display it just once.
If they're different display in any order, not really important.

@thealphadollar
Copy link
Contributor

@cfsmp3 I would request you to provide another sample for the same since the sample provided gives error, log report here.

I'll look into the error by myself and, if needed, file an issue for the same.

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Apr 18, 2018 via email

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Dec 27, 2018

Update: This task gives ENOUGH points to qualify for GSoC. If you're proposal is good and your code to get this done gets merged, you're in.

@samrat2825
Copy link

Does a approach where in the background two .srt files, for both cc and burned-in, are independently generated and are combined in the end using timestamp, via a single command from terminal, is outputted as a single file qualify?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Mar 28, 2021

Does a approach where in the background two .srt files, for both cc and burned-in, are independently generated and are combined in the end using timestamp, via a single command from terminal, is outputted as a single file qualify?

No. That's the lazy approach :-)

@PunitLodha PunitLodha added the HardsubX Issues related to extraction of HardsubX label Jan 28, 2022
@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 18, 2022

Video link again:
https://drive.google.com/file/d/0B_61ywKPmI0TeU5rOTlCMWxKbW8/view?usp=sharing&resourcekey=0-nZ9e7M3yhB6099__xP8ofw

@shashwat1002
Copy link
Contributor

Hi, I tried running the compiled source code on the examples with the command (in the build directory)

./ccextractor -hardsubx -subcolor yellow ~/Downloads/the.americans.s05.e06-original.unedited.from.hdhomerun.ts -conf_thresh 50 -ocr_mode word

and I get the following set of failures

CCExtractor 0.94, Carlos Fernandez Sanz, Volker Quetschke.
Teletext portions taken from Petr Kutalek's telxcc
--------------------------------------------------------------------------
HardsubX (Hard Subtitle Extractor) - Burned-in subtitle extraction subsystem
Input : /home/mrcreator/Downloads/the.americans.s05.e06-original.unedited.from.hdhomerun.ts
Subtitle Color : Yellow
OCR Mode : Word-wise
OCR Confidence Threshold : 50.00
OCR Italic Detection : Off
Minimum subtitle duration : 0.5 seconds (Default)
FFMpeg Media Information:-
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpeg2video @ 0x55bdef3fc280] Invalid frame dimensions 0x0.
[mpegts @ 0x55bdee66c140] start time for stream 3 is not set in estimate_timings_from_pts
[mpegts @ 0x55bdee66c140] PES packet size mismatch
[mpegts @ 0x55bdee66c140] Packet corrupt (stream = 1, dts = 2115777097).
[mpegts @ 0x55bdee66c140] PES packet size mismatch
[mpegts @ 0x55bdee66c140] Packet corrupt (stream = 2, dts = 2115777961).
[mpegts @ 0x55bdee66c140] Could not find codec parameters for stream 4 (Unknown: none (ETV1 / 0x31565445)): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[mpegts @ 0x55bdee66c140] Could not find codec parameters for stream 5 (Unknown: none (ETV1 / 0x31565445)): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[mpegts @ 0x55bdee66c140] Could not find codec parameters for stream 6 (Unknown: none ([192][0][0][0] / 0x00C0)): unknown codec
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, mpegts, from '/home/mrcreator/Downloads/the.americans.s05.e06-original.unedited.from.hdhomerun.ts':
  Duration: 01:10:00.91, start: 19308.731378, bitrate: 2438 kb/s
  Program 2111 
  Stream #0:0[0xf51]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, top first), 720x480 [SAR 8:9 DAR 4:3], Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Side data:
      cpb: bitrate max/min/avg: 18000000/0/0 buffer size: 1835008 vbv_delay: N/A
  Stream #0:1[0xf52](eng): Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, 5.1(side), fltp, 384 kb/s
  Stream #0:2[0xf53](spa): Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, stereo, fltp, 192 kb/s
  Stream #0:3[0xf54]: Data: scte_35
  Stream #0:4[0xf55]: Unknown: none (ETV1 / 0x31565445)
  Stream #0:5[0xf56]: Unknown: none (ETV1 / 0x31565445)
  Stream #0:6[0xfee]: Unknown: none ([192][0][0][0] / 0x00C0)
[1]    46029 segmentation fault (core dumped)  ./ccextractor -hardsubx -subcolor yellow  -conf_thresh 50 -ocr_mode word

It seems to be a segmentation fault.

I haven't yet looked at the source code and was just looking at basic input and output. Before I dive deep, is there something I'm doing very obviously wrong in running it on the test file?

Any advice is appreciated.

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 19, 2022

Before I dive deep, is there something I'm doing very obviously wrong in running it on the test file?

No, a segfault is never the user's fault :-)
Go dive! :-)

@shashwat1002
Copy link
Contributor

shashwat1002 commented Feb 20, 2022 •
edited
Loading

Hi, I ran a debug build on two different samples and received a segmentation fault in the same place.

Given that I can't work on this issue until the segmentation faults are resolved, would it be more appropriate to start a new issue?

@cfsmp3

Edit: interestingly enough, the fault happens at

ccx_common_logging.debug_ftn(CCX_DMT_708, "[CEA-708] dtvcc_writer_init\n");

and when that line is commented

ccx_common_logging.debug_ftn(CCX_DMT_708, "[CEA-708] dtvcc_writer_init: "
						  "[%s][%d][%d]\n",
				     base_filename, program_number, service_number);

causes the segmentation fault.

both of which seem to be debug logs related.

is it possible that my build is at fault here?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Feb 20, 2022

Given that I can't work on this issue until the segmentation faults are resolved, would it be more appropriate to start a new issue?

Well, you can work on fixing that segfault, which seems trivial :-)

I'd recommend to build it with debug info and run with valgrind. It will tell exactly what's going on.

@shashwat1002 shashwat1002 mentioned this issue Feb 22, 2022
Merged
10 tasks
@shashwat1002
Copy link
Contributor

shashwat1002 commented Mar 7, 2022 •
edited
Loading

Hello,

I feel that this proposal will require making a struct that is in some way a general form of struct lib_ccx_ctx and struct lib_hardsubx_ctx with relevant attributes from both.

Does that sound feasible right now?
Would it be wise for me to go ahead with it?

@cfsmp3
Copy link
Contributor Author

cfsmp3 commented Mar 8, 2022

Hello,

I feel that this proposal will require making a struct that is in some way a general form of struct lib_ccx_ctx and struct lib_hardsubx_ctx with relevant attributes from both.

Maybe you can just pass both as needed instead of creating a new one?

Does that sound feasible right now? Would it be wise for me to go ahead with it?

Sure. Give it a go - there's several approaches, go with the one you feel more comfortable with, we can always iterate once you have something working.

@shashwat1002 shashwat1002 mentioned this issue Mar 14, 2022
Merged
9 tasks
@cfsmp3 cfsmp3 closed this as completed Mar 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
difficulty: medium GSoC-related HardsubX Issues related to extraction of HardsubX OCR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cfsmp3 @saurabhshah0410 @thealphadollar @shashwat1002 @samrat2825 @PunitLodha