Vimeo: Chokes on --add-metadata

[vxbinaca]

youtube-dl  --ignore-config --verbose  --download-archive ~/.ytdlarchive --no-overwrites --call-home --continue --write-info-json --write-description --write-thumbnail --merge-output-format mkv--all-subs --sub-format srt --convert-subs srt --write-sub --add-metadata https://vimeo.com/70668043 https://vimeo.com/70666333

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--ignore-config', u'--verbose', u'--download-archive', u'/home/vxbinaca/.ytdlarchive', u'--no-overwrites', u'--call-home', u'--continue', u'--write-info-json', u'--write-description', u'--write-thumbnail', u'--merge-output-format', u'mkv--all-subs', u'--sub-format', u'srt', u'--convert-subs', u'srt', u'--write-sub', u'--add-metadata', u'https://vimeo.com/70668043', u'https://vimeo.com/70666333']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.27
[debug] Python version 2.7.10 - Linux-4.2.0-25-generic-x86_64-with-Ubuntu-15.10-wily
[debug] exe versions: ffmpeg 2.7.5-0ubuntu0.15.10.1, ffprobe 2.7.5-0ubuntu0.15.10.1, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Public IP address: 76.101.221.232
[vimeo] 70668043: Downloading webpage
[vimeo] 70668043: Extracting information
[vimeo] 70668043: Downloading webpage
[vimeo] 70668043: Downloading JSON metadata
[vimeo] 70668043: Downloading m3u8 information
[info] Video description is already present
[info] Video description metadata is already present
[vimeo] 70668043: Thumbnail is already present
[debug] Invoking downloader on u'https://10-lvl3-hls.vimeocdn.com/1453990861-28223b02a7d6053983227f4b64333f85d0240957/01/4133/2/70668043/178317076.mp4.m3u8'
[download] Ask Ash No. 1-70668043.mp4 has already been downloaded
[download] 100% of 9.83MiB
[ffmpeg] Adding metadata to 'Ask Ash No. 1-70668043.mp4'
[debug] ffmpeg command line: ffmpeg -y -i 'file:Ask Ash No. 1-70668043.mp4' -c copy -metadata 'comment=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
Ask Ash | thenakedissue@junnnktank.com' -metadata 'description=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
Ask Ash | thenakedissue@junnnktank.com' -metadata artist=JUNNNKTANK -metadata 'title=Ask Ash No. 1' -metadata date=20130719 -metadata purl=https://vimeo.com/70668043 'file:Ask Ash No. 1-70668043.temp.mp4'
ERROR: Conversion failed!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1737, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 395, in run
    self.run_ffmpeg(filename, temp_filename, options)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 159, in run_ffmpeg
    self.run_ffmpeg_multiple_files([path], out_path, opts)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 155, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError

[vxbinaca]

Looks like it's failing on metadata embed, it's choking on --add-metadata

[yan12125]

Here's the output of the ffmpeg command:

$ ffmpeg -y -i 'file:Ask Ash No. 1-70668043.mp4' -c copy -metadata date=20130719 -metadata 'description=More | junnnktank.com/thenakedissue/faq 
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}.
Ask Ash | thenakedissue@junnnktank.com' -metadata 'title=Ask Ash No. 1' -metadata artist=JUNNNKTANK -metadata purl=https://vimeo.com/70668043 -metadata 'comment=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}.
Ask Ash | thenakedissue@junnnktank.com' 'file:Ask Ash No. 1-70668043.temp.mp4'
ffmpeg version 2.8.5 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mpegts, from 'file:Ask Ash No. 1-70668043.mp4':
  Duration: 00:01:05.50, start: 0.700000, bitrate: 1259 kb/s
  Program 1 
    Stream #0:0[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720, 24 fps, 24 tbr, 90k tbn, 48 tbc
    Stream #0:1[0x102]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 144 kb/s
[mp4 @ 0x559d96690260] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x559d96690260] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:Ask Ash No. 1-70668043.temp.mp4':
  Metadata:
    date            : 20130719
    description     : More | junnnktank.com/thenakedissue/faq
                    : f. Ash twitter.com/ashvandeesch
                    : This is Ash. She's from Holland. She's a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
                    : Ask Ash | thenakedissue@junnnktank.com
    title           : Ask Ash No. 1
    artist          : JUNNNKTANK
    purl            : https://vimeo.com/70668043
    comment         : More | junnnktank.com/thenakedissue/faq
                    : f. Ash twitter.com/ashvandeesch
                    : This is Ash. She's from Holland. She's a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
                    : Ask Ash | thenakedissue@junnnktank.com
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720, q=2-31, 24 fps, 24 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, 144 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame=    2 fps=0.0 q=-1.0 Lsize=       3kB time=00:00:00.13 bitrate= 151.7kbits/s    
video:1kB audio:3kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

These lines are relevant:

[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)

With NativeHlsFD, -bsf:a aac_adtstoasc is not applied in downloading. FFMpeg can't handle the downloaded file. @remitamine I see 'm3u8_native' is added in your commit e5c209a. Do you have any reason for that change? I've tested a few Vimeo videos HlsFD using ffmpeg works.

[vxbinaca]

If it helps, I downloaded that entire channel and encountered what I think may be a similar error but I just re-ran the rip (this is easy since I have it not clobber or retry already downloaded videos) and it passed over the problem video but this particular one wouldn't do it without dropping the --add-metadata flag.

92/94 videos on that channel completed without error.

[yan12125]

@vxbinaca The problematic line is this one: https://github.com/rg3/youtube-dl/blob/ed7cd1e859cf97e975a28a5e8c58a1d1aca819fe/youtube_dl/extractor/vimeo.py#L459
Change the following lines in youtube_dl/extractor/vimeo.py

            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))

to:

            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))

Works for me. However, the parameter 'm3u8_native' is added by another developer, so I have to ask his/her original intention before making changes.

[remitamine]

However, the parameter 'm3u8_native' is added by another developer, so I have to ask his/her original intention before making changes.

#7126 (comment)

[yan12125]

So let's call @dstftw: using NativeHlsFD by default is causing problems on ffmpeg like this. (OK with avconv v12_dev0-2305-g77a44f5). Is using ffmpeg to download HLS streams causing a different problem?

[dstftw]

So do you suggest switching all m3u8_native to m3u8 in all (~45) extractors?

[yan12125]

The problem is: if m3u8_native is specified by the extractor, there's no simple way to force HlsFD. Maybe adding an option --hls-prefer-ffmpeg solves all the problems? If we keep current framework, replacing all m3u8_native occurrences is necessary for this case.

[dstftw]

Presence of --hls-prefer-ffmpeg does not really solve the problem for average user who use defaults, but it would be useful.

[remitamine]

adding -bsf:a aac_adtstoasc to the command that add metadata if the protocol is m3u8_native will fix the problem(created PR for this).

Do you have any reason for that change?

in general hls native is better for two reasons(at least):

• can continue the download
• progress bar

[vxbinaca]

I'll wait for the push. I get my script from pip anyway and check twice daily for new copies. I also have another issue on Vimeo.