[twitch] Embedding subs (chat) failing

[za3k]

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2020.05.08
• I've checked that all provided URLs are alive and playable in a browser
  A bit of background: Twitch has "Video on demand"=VOD, basically whole archived streams, and "clips", 30-second segments. VOD has chat, I think clips don't. Since VOD disappears after a month, I archive it--but also you may need to find a new sample URL to test on depending on how long it takes to get to this, please don't just close the bug because the URL is expired.
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones
  I did survey existing bugs. This may be related to #18593, I can't tell if that issue was fixed and just not cleaned up, but this is a new error message so I'm filing it as a new bug. I didn't see any other bugs that have this error message and the word "twitch", open or closed.

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://twitch.tv/videos/627004856', u'--no-playlist', u'--add-metadata', u'--all-subs', u'--embed-subs', u'--download-archive', u'ARCHIVE', u'-o', u'videos/twitch_%(id)s.%(ext)s']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2020.05.08
[debug] Python version 2.7.16 (CPython) - Linux-4.19.0-8-amd64-x86_64-with-debian-10.3
[debug] exe versions: ffmpeg 4.1.4-1, ffprobe 4.1.4-1
[debug] Proxy map: {}
[twitch:vod] 627004856: Downloading vod info JSON
[twitch:vod] 627004856: Downloading vod access token
[twitch:vod] 627004856: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: videos/twitch_v627004856.rechat.json
[debug] Invoking downloader on u'https://dqrpb9wgowsf5.cloudfront.net/9077bff1bbd844a1ea14_skizzleman_38182849504_1455302010/chunked/index-dvr.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 911
[download] Destination: videos/twitch_v627004856.mp4
[download] 100% of 3.58GiB in 06:22.12MiB/s ETA 00:00042:04
[debug] ffmpeg command line: ffprobe -show_streams 'file:videos/twitch_v627004856.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "videos/twitch_v627004856.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:videos/twitch_v627004856.mp4' -c copy -f mp4 '-bsf:a' aac_adtstoasc 'file:videos/twitch_v627004856.temp.mp4'
[ffmpeg] Adding metadata to 'videos/twitch_v627004856.mp4'
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:videos/twitch_v627004856.mp4' -c copy -metadata 'date=20200520' -metadata 'purl=https://twitch.tv/videos/627004856' -metadata 'artist=Skizzleman' -metadata 'title=Solo Skizz Sitcom' 'file:videos/twitch_v627004856.temp.mp4'
[ffmpeg] Embedding subtitles in 'videos/twitch_v627004856.mp4'
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:videos/twitch_v627004856.mp4' -i 'file:videos/twitch_v627004856.rechat.json' -map 0 -c copy -map '-0:s' -map '-0:d' '-c:s' mov_text -map '1:0' '-metadata:s:s:0' 'language=rechat' 'file:videos/twitch_v627004856.temp.mp4'
ERROR: file:videos/twitch_v627004856.rechat.json: Invalid data found when processing input
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2065, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 426, in run
    self.run_ffmpeg_multiple_files(input_files, temp_filename, opts)
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 235, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError: file:videos/twitch_v627004856.rechat.json: Invalid data found when processing input

Description

As best I can tell, the flags --all-subs --embed-subs causes videos with chat to fail download on twitch.

Edit: Oh also, I believe the expected behavior on twitch is that "subs" should some version of twitch chat, embedded as a subtitle track, which isn't super obvious :)

Thanks everyone for maintaining this great program.

[vxbinaca]

Because Twitch subs are in a format that can't be embedded.

[za3k]

Oh, okay. I thought this was working at some point, must be mistaken.

Is there any other way to shove them in the same file, like is done with youtube metadata?

Anyway feel free to close, although I do think it would be a plus if there was a better error message for unsupported operations like this.

[vxbinaca]

Oh, okay. I thought this was working at some point, must be mistaken.

Is there any other way to shove them in the same file, like is done with youtube metadata?

Anyway feel free to close, although I do think it would be a plus if there was a better error message for unsupported operations like this.

You can close the issue since you opened it. Just click the button.

To answer your first question, you'd have to manually transcode the JSON subs format to SRT, then embed it into the file, but that won't look pretty like you think it will and I don't even think you can transcode from JSON to SRT.

[za3k]

On 2020-06-06 20:26, Paul Henning wrote: > Anyway feel free to close, although I do think it would be a plus if > there was a better error message for unsupported operations like > this. You can close the issue since you opened it. Just click the button.

I don't care to close this because I think there's still an action item (print a better error message). One contributor should please take charge and decide whether that should be done (to avoid lack of responsibility), and close it as wontfix if not :).

> Oh, okay. I thought this was working at some point, must be > mistaken. > > Is there any other way to shove them in the same file, like is done > with youtube metadata? > To answer your first question, you'd have to manually transcode the JSON subs format to SRT, then embed it into the file, but that won't look pretty like you think it will and I don't even think you can transcode to that format.

By "shove them in the same file", I meant that for youtube, --add-metadata adds youtube's JSON metadata into the .mkv somehow after download (I assume as a metadata track, although it looks like there might be an xattr option? I really need to learn more about how this works). Sorry for being a bit vague the first time around. It seems like that would be useful here, even if actual subtitle embedding isn't possible. However this is a poorly-thought-out idea, and definitely a separate issue, I was just thinking aloud.

[vxbinaca]

JSON metadata is different from subtitles, they don't even describe the same things.

[za3k]

Yes. I was suggesting to use the same mechanism to include the 'XXXrechat.json' as a track as is used to include the youtube metadata JSON as a track. And I'm retracting my suggestion, in any case.

Feel free to add a check for the incompatible: twitch, --all-subs, and --embed-subs with a nice error message. I feel like this conversation isn't going anywhere productive so I'm closing this.