Could someone kindly add a function to download CC subtitle from site "www.tagesschau.de"?

[snowmfx]

Checklist

• [x ] I'm reporting a site feature request
• I've verified that I'm running youtube-dl version 2019.06.08
• [x ] I've searched the bugtracker for similar site feature requests including closed ones

Description

WRITE DESCRIPTION HERE

Some video in this site "www.tagesschau.de" has subtitle.
Eg. https://www.tagesschau.de/multimedia/sendung/ts-31835.html .
But I can't download it with latest youtube-dl tool.
I ask a favor.

[mkg20001]

I've found that it downloads https://www.tagesschau.de/multimedia/video/untertitel-35735.xml (seems like TTS format) the ID comes from https://www.tagesschau.de/multimedia/sendung/ts-31835.html but can be dynamically aquired from JS using window.location.host + '/' + mc.getSubtitleUrl()....

...which gets it from https://www.tagesschau.de/multimedia/video/video-554109~mediajson_broadcastType-TS2000.json property _subtitleUrl

No clue where the JSON comes from, though. Seems to be loading some iframe where it load that JSON

[snowmfx]

Thanks.
I searched word "untertitel" in element - Network. I got this subtitle address.

[mkg20001]

@snowmfx That should work as a workarround. I'd suggest you to leave this issue open until it's implemented in the extractor. (I tried to add it, but it seems like the current extractor isn't even using the JSON. I'll try again once I get the time to do it)

[basicmaster]

(seems like TTS format)

It is TTML or rather the EBU-TT-D-Basic-DE profile used by the German public broadcasters. If renamed to *.ttml , recent VLC versions render it.

[Reino17]

For the time being you could have a look at Xidel, a HTML/XML/JSON parser (using CSS, XPath, XQuery, JSONiq and pattern templates).
Extract the json-url:

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -e 'json(//@data-ctrl-iframe)/action/default/src'
/multimedia/video/video-554109~ardplayer_autoplay-true_broadcastType-TS2000.html

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -e 'json(//@data-ctrl-iframe)/action/default/replace(src,"(.+~).+(_broadcast.+\.).+","$1mediajson$2json")'
/multimedia/video/video-554109~mediajson_broadcastType-TS2000.json

"Follow" (-f)/open the json-url, parse as json (because it's Content-Type: text/html instead of Content-Type: application/json) and extract the subtitle-url:

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -f 'json(//@data-ctrl-iframe)/action/default/replace(src,"(.+~).+(_broadcast.+\.).+","$1mediajson$2json")' -e 'json($raw)/_subtitleUrl'
/multimedia/video/untertitel-35735.xml

If you want to convert to *.srt (subrip), then Xidel can do this too:

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -f 'json(//@data-ctrl-iframe)/action/default/replace(src,"(.+~).+(_broadcast.+\.).+","$1mediajson$2json")' -f 'json($raw)/_subtitleUrl' --xquery 'for $x at $i in //tt:p[tt:span] return ($i,replace(concat($x/@begin," --> ",$x/@end),"\.",","),$x/tt:span,"")'
1
00:00:03,040 --> 00:00:06,720
Hier ist das Erste Deutsche Fernsehen
mit der tagesschau.

2
00:00:06,880 --> 00:00:11,120
Herzlich willkommen zur Live-
Untertitelung des NDR (16.06.2019)

3
00:00:15,200 --> 00:00:17,240
Heute im Studio: Jan Hofer

[...]

257
00:15:45,320 --> 00:15:47,400
Copyright Untertitel: NDR 2019

[basicmaster]

The problem with SubRip here is that the speaker colors all are discarded.

[Reino17]

Then we just have to update the extraction query.
Prettified query:

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" \
-f '
  json(//@data-ctrl-iframe)/action/default/replace(
    src,
    "(.+~).+(_broadcast.+\.).+",
    "$1mediajson$2json"
  )
' \
-f 'json($raw)/_subtitleUrl' \
--xquery '
  let $a:={|
    //tt:style[@tts:color]/{
      @xml:id:@tts:color
    }
  |}
  for $x at $i in //tt:p[tt:span]
  return (
    $i,
    replace(
      concat(
        $x/@begin,
        " --> ",
        $x/@end
      ),
      "\.",","
    ),
    $x/tt:span/(
      if (@style="textWhite") then
        .
      else
        concat(
          "<font color="",
          $a(@style),
          "">",
          .,
          "</font>"
        )
    ),
    ""
  )
'
1
00:00:03,040 --> 00:00:06,720
Hier ist das Erste Deutsche Fernsehen
mit der tagesschau.

2
00:00:06,880 --> 00:00:11,120
<font color="#0000FF">Herzlich willkommen zur Live-</font>
<font color="#0000FF">Untertitelung des NDR (16.06.2019)</font>

3
00:00:15,200 --> 00:00:17,240
Heute im Studio: Jan Hofer

[...]

257
00:15:45,320 --> 00:15:47,400
<font color="#0000FF">Copyright Untertitel: NDR 2019</font>

---

--xquery 'let $a:={|//tt:style[@tts:color]/{@xml:id:@tts:color}|} return $a'
{
  "textBlack": "#000000",
  "textRed": "#FF0000",
  "textGreen": "#00FF00",
  "textYellow": "#FFFF00",
  "textBlue": "#0000FF",
  "textMagenta": "#FF00FF",
  "textCyan": "#00FFFF",
  "textWhite": "#FFFFFF"
}

When we come across <tt:span style="textBlue"> for instance, with $a(@style) this json object is consulted for the value of the "textBlue" attribute.

---

Minified query:

$ xidel -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -f 'json(//@data-ctrl-iframe)/action/default/replace(src,"(.+~).+(_broadcast.+\.).+","$1mediajson$2json")' -f 'json($raw)/_subtitleUrl' --xquery 'let $a:={|//tt:style[@tts:color]/{@xml:id:@tts:color}|} for $x at $i in //tt:p[tt:span] return ($i,replace(concat($x/@begin," --> ",$x/@end),"\.",","),$x/tt:span/(if (@style="textWhite") then . else concat("<font color="",$a(@style),"">",.,"</font>")),"")'

Minified query (Windows):

xidel.exe -s "https://www.tagesschau.de/multimedia/sendung/ts-31835.html" -f "json(//@data-ctrl-iframe)/action/default/replace(src,'(.+~).+(_broadcast.+\.).+','$1mediajson$2json')" -f "json($raw)/_subtitleUrl" --xquery "let $a:={|//tt:style[@tts:color]/{@xml:id:@tts:color}|} for $x at $i in //tt:p[tt:span] return ($i,replace(concat($x/@begin,' --> ',$x/@end),'\.',','),$x/tt:span/(if (@style='textWhite') then . else concat('<font color="',$a(@style),'">',.,'</font>')),'')"