Support decoding Coub videos (was: Downloaded Coub video file broken)

[arisboch]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.07.23. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2017.07.23

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

casval@casval-X6815 /tmp % youtube-dl http://coub.com/view/eq9pa -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'http://coub.com/view/eq9pa', u'-v']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.23
[debug] Python version 2.7.13 - Linux-4.10.0-28-generic-x86_64-with-Ubuntu-17.04-zesty
[debug] exe versions: avconv 3.2.4-1build2, avprobe 3.2.4-1build2, ffmpeg 3.2.4-1build2, ffprobe 3.2.4-1build2, rtmpdump 2.4
[debug] Proxy map: {}
[Coub] eq9pa: Downloading JSON metadata
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'http://s3.storage.akamai.coub.com/get/b130/p/coub/simple/cw_file/30cf7df5d5e/aa6e6b21fbdf78b11538d/muted_mp4_big_size_1473445250_muted_big.mp4'
[download] Destination: Spergberb-eq9pa.mp4
[download] 100% of 1.77MiB in 00:00
casval@casval-X6815 /tmp % mpv Spergberb-eq9pa.mp4
Playing: Spergberb-eq9pa.mp4
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
Failed to recognize file format.

Exiting... (Errors when loading file)

---

When I try to download a Coub video, the resulting video file is unplayable (I tried VLC, as well) and judging by the URL is also tried to download the muted version of the video.

[dstftw]

This is what coub serves.

[catdog2]

I found something interesting inside their js:

MediaElementStrategy.prototype.decode = function(buf) {
      var x;
      x = new Uint16Array(buf, 0, 2);
      if (x[0] === 19392) {
        return x[0] = 0;
      }
    };

And indeed, zeroing out the first two bytes with a hex editor makes the files playable.

[ozburo]

@catdog2 I see that

    MediaElementStrategy.prototype.setSrc = function(src, size) {
      var MS, minChunkSize;
      this.mimeCodec = this.getCodec();
      MS = window.MediaSource;
      if (MS && MS.isTypeSupported(this.mimeCodec)) {
        this.size = size;
        minChunkSize = this.kind === 'video' && 262144 || 131072;
        if (!this.isHD) {
          minChunkSize /= 2;
        }
        this.chunkSize = this.kind === 'video' ? Math.max(Math.ceil(this.size / 4), minChunkSize) : minChunkSize;
        return this.setMediaSource(src);
      } else { 
        console.log("[PLAYER]", "media", this.kind, "fallback to media element");
        return this.setSource(src);
      }
    };

Looking deeper I see that this decode method is called when using setMediaSource which in turn is called in setSrc where there is a check for "MediaSource" browser support, if there isn't any, then it reverts to basic setSource. Maybe there's a way to trigger this by mimicking an old browser?

I tried making a request and setting 'User-Agent' header to something like IE8 but that didn't work, not sure if I'm doing it right, just thought I'd share.

[niranjanshr13]

@ozburo @arisboch @catdog2
If you guys still have problem, try this:
curl -wL $(youtube-dl -g 'http://coub.com/view/uilvc') | grep -o "http.*mp4" | sed 's/muted_//g'

[arisboch]

@niranjanshr13 Thx, I'm gonna incorporate that into my (primitive) wrapper script till a proper solution is implemented.

[Kagami]

@niranjanshr13 seems like that doesn't work anymore too. It worked for me few days ago.

[Kagami]

Here is simple script to construct videos that should looks similar to the ones played on the site. I didn't try to make it fancy, just the minimal proof of concept.

#!/usr/bin/env bash
set -e
youtube-dl -o 1.mp4 "$1"
youtube-dl -f html5-audio-high -o 1.mp3 "$1"
printf '\x00\x00' | dd of=1.mp4 bs=1 count=2 conv=notrunc
for i in `seq 1 100`; do echo "file '1.mp4'" >> 1.txt; done
ffmpeg -hide_banner -f concat -i 1.txt -i 1.mp3 -c copy -shortest -movflags faststart out.mp4
rm 1.mp4 1.mp3 1.txt

[affinityv]

Okay, here's my take on that shell script:

#!/bin/bash

set -e

TMPFILE=$(mktemp 'XXXXXXXX.mp4')

filename_format+='%(upload_date)s'
filename_format+='_%(title)s'
filename_format+='_%(id)s'

FILE_NAME=$(
	youtube-dl --get-filename \
		--restrict-filenames -o "${filename_format}" "${1}"
)

youtube-dl -f html5-video-high -o "${FILE_NAME}.mp4" "$1"
youtube-dl -f html5-audio-high -o "${FILE_NAME}.mp3" "$1"
printf '\x00\x00' | dd of="${FILE_NAME}.mp4" bs=1 count=2 conv=notrunc
i=0;export i
for i in {1..100}
do
	echo "file '${FILE_NAME}.mp4'" >> "${FILE_NAME}.txt"
done
ffmpeg -y -hide_banner -f concat -i "${FILE_NAME}.txt" \
	-i "${FILE_NAME}.mp3" -c copy -shortest -movflags faststart "${TMPFILE}"
rm "${FILE_NAME}"*
mv "${TMPFILE}" "${FILE_NAME}.mp4"

[arisboch]

@Kagami @affinityv Is there a way to download just the video and not the 55 second loop of the video?

[niranjanshr13]

@arisboch here is the code without any loop.

#!/bin/bash
url="http://coub.com/view/g8j3b"
youtube-dl $url -o 1.mp4
printf '\x00\x00' | dd of=1.mp4 bs=1 count=2 conv=notrunc

[therealkrispet]

I've written a Python based tool for this a while ago: https://git.strongds.hu/mrtoth/coub-dl
It properly calculates how many times the shorter stream should be looped and outputs videos just like coub does in a browser. It used the curl workaround by @niranjanshr13 for a while, and now it uses this "just zero the first 2 bytes" magic (first posted by @catdog2), so thanks!

[arisboch]

@niranjanshr13 But it's muted ;-(

[niranjanshr13]

@arisboch it is muted, if you want the audio then try Kagami code, basically his code loop the video until the audio ends.

#!/usr/bin/bash
set -e
youtube-dl -o 1.mp4 "$1"
youtube-dl -f html5-audio-high -o 1.mp3 "$1"
printf '\x00\x00' | dd of=1.mp4 bs=1 count=2 conv=notrunc
for i in `seq 1 100`; do echo "file '1.mp4'" >> 1.txt; done
ffmpeg -hide_banner -f concat -i 1.txt -i 1.mp3 -c copy -shortest -movflags faststart out.mp4
rm 1.mp4 1.mp3 1.txt

[affinityv]

@niranjanshr13 Don't use /usr/bin/env to find a program (or shell)..... you should know exactly where the program (or shell) is on your system and it should be in a safe directory. I'm sure you can find other advice, but quite simply, don't use /usr/bin/env ... EVER.

[niranjanshr13]

@affinityv this is not my code, i copied from @Kagami code for the @arisboch question.

[niranjanshr13]

for the python user.

#!/usr/bin/python3
import youtube_dl
import requests
link='http://coub.com/view/zoxf2'
ytdl = youtube_dl.YoutubeDL({'outtmpl': '%(id)s%(ext)s'})
result = ytdl.extract_info(link, download=False)
r = requests.get(result['url'])
with open( result['title'] + '.mp4','wb') as f:
    f.write(b'\x00\x00' + r.content[2:])

[Hrxn]

FWIW, I just tested this again and it is still not working with the latest version of youtube-dl.
Dunno, maybe either incorporate a fix along those examples listed here, or take Coub off of the list of supported sites in the meantime.

[Hrxn]

FYI, the Python example listed in the comment above still works, although it's only the video, obviously.
Extraction of the audio track still works with youtube-dl as well..

[piotrmil]

Yeah, can confirm, downloaded videos from coub seems to be broken. Fix it pls.

[arisboch]

In a few months this gonna be 2 years old and still not fixed.

[tmfck]

Still need proper Coub downloading.

[CaViCcHi]

@niranjanshr13 Don't use /usr/bin/env to find a program (or shell)..... you should know exactly where the program (or shell) is on your system and it should be in a safe directory. I'm sure you can find other advice, but quite simply, don't use /usr/bin/env ... EVER.

I disagree with this. if you move scripts often between platform this is useful.
the only place where env is not available is inside an initram, as long as you know.

[CaViCcHi]

This would download the mute video

youtube-dl -f 'html5-video-med[ext=mp4]' --exec "printf '\x00\x00' | dd of={} bs=1 count=2 conv=notrunc" -o "%(title)s-%(id)s.%(ext)s" <URL>

I was hoping the --exec would work as by documentation it says "Execute a command on the file after downloading" But it's a lie :P because it executes AFTER merging.

because otherwise you could use -f 'html5-video-med[ext=mp4]+html5-audio-high[ext=mp3]' then exec on the mp4, but instead the exec doens't happen as it's on the final output file which fails cause the mp4 is broken. :(

so you still need 3 lines of code to download a coub... life has no meaning :)

[CaViCcHi]

#!/usr/bin/env bash
[ -z "$1" ] && exit 88
URL=$1
[ -e "$(youtube-dl --restrict-filenames -f 'html5-video-med[ext=mp4]' --get-filename -o "%(title)s-%(id)s.%(ext)s" "$URL")" ] && echo -e "Already downloaded $URL" &&  exit 0
if youtube-dl --list-formats "${URL}" | /bin/grep html5-audio &>/dev/null; then
    youtube-dl -q --restrict-filenames -f 'html5-video-med[ext=mp4]+html5-audio-high[ext=mp3]' --merge-output-format FAKE -o "%(title)s-%(id)s.%(ext)s" "$URL" &>/dev/null
    printf '\x00\x00' | dd of=$(youtube-dl --restrict-filenames -f 'html5-video-med[ext=mp4]' --get-filename -o "%(title)s-%(id)s.f%(format_id)s.%(ext)s" "$URL") bs=1 count=2 conv=notrunc &>/dev/null
    youtube-dl  --restrict-filenames --prefer-ffmpeg --postprocessor-args '-shortest' -f 'html5-video-med[ext=mp4]+html5-audio-high[ext=mp3]' --exec 'touch {}' -o "%(title)s-%(id)s.%(ext)s" "$URL" &>/dev/null
else
    youtube-dl -q --restrict-filenames --prefer-ffmpeg -f 'html5-video-med[ext=mp4]' --exec "printf '\x00\x00' | dd of={} bs=1 count=2 conv=notrunc" -o "%(title)s-%(id)s.%(ext)s"  "$URL" &>/dev/null
fi

re-edited because if the coub is video only it will only get the fragment. now it gets the video correctly with or without audio.

the first -e is because if you try to download the same url it will create fragments and then say it already made the file and exit.
I use FAKE to force an error, but it would fail anyway, so it's probably irrelevant, still...
and the --exec touch is something I do cause I don't want the date to be the video's but I want now.

that's the smallest I could do, I did it this way because if I download the pieces separately then you don't have the name and you're still going to need to call --get-filename for each "fragment"

if --exec would really run after download this could be 1 line :(

youtube-dl --restrict-filenames --ffmpeg-location /root/bin/ -f 'html5-video-med[ext=mp4]+html5-audio-high[ext=mp3]' --exec "[[ {} =~ \.mp4$ ]] && printf '\x00\x00' | dd of={} bs=1 count=2 conv=notrunc || true" --merge-output-format mp4 -o "%(title)s-%(id)s.%(ext)s" <URL>

but it doesn't work cause it will try to execute on the post-processed, but it doesn't reach it as ffmpeg won't recognize the video as a video.

[shinji2009]

hi. can anybody make last script in windows .bat to work with youtube-dl.exe please? if it only works. even if i must paste url in it every time it will be ok