Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

youtube-dl 2018.02.26: "urlopen error unknown url type: npo+drm" for `youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590` #15742

Closed
jpluimers opened this issue Mar 2, 2018 · 1 comment
Closed

youtube-dl 2018.02.26: "urlopen error unknown url type: npo+drm" for `youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590` #15742

jpluimers opened this issue Mar 2, 2018 · 1 comment
Labels
duplicate

Comments

@jpluimers
Copy link

@jpluimers jpluimers commented Mar 2, 2018

Please do NOT close this issue until a release that undoes the regression: this way it is easier to track for everyone involved.

This issue will document how to binary search a homebrew installation for the last working and first failing version so it is easier to find the actual cause of a failure.

  • I've verified and I assure that I'm running youtube-dl 2018.02.26
  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser
  • Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

Output of youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-k', 'https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.02.26
[debug] Python version 3.6.4 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.2, ffprobe 3.4.2
[debug] Proxy map: {}
[npo] BV_101386590: Downloading JSON metadata
[npo] BV_101386590: Downloading token
[npo] BV_101386590: Downloading formats JSON
[npo] BV_101386590: Downloading Adaptive stream JSON
[npo] BV_101386590: Downloading m3u8 information
[npo] BV_101386590: Downloading Hoog stream JSON
[npo] BV_101386590: Downloading Normaal stream JSON
[npo] BV_101386590: Downloading Laag stream JSON
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'npo+drm://internetvod.omroep.nl/BV_101386590'
ERROR: unable to download video data: <urlopen error unknown url type: npo+drm>
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1896, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1835, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 340, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2199, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 549, in _open
    'unknown_open', req)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1388, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: npo+drm>

Description of your issue, suggested solution and other information

NPO downloads fail with an ERROR: unable to download video data: <urlopen error unknown url type: npo+drm>

It looks like NPO recently changed their format as this used to work two weeks ago, but fails now with the same error youtube-dl -k https://www.npo3.nl/het-is-hier-autistisch/15-02-2018/BV_101386592

These URL formats now fail but did not in the past:

Since #15725 indicates it is a regression from around 2018.02.04, I've dug deeper using a binary search homebrew (see below) pinpointing the failure between ... and ...

Finding the breaking release

This is based on the excellent article Zoltan Altfatter: Install a specific version of a formula with homebrew

pushd "$(brew --repo homebrew/core)"
git log Formula/youtube-dl.rb | grep "\(^commit \|youtube-dl 2018\)"

This got me these relevant commits:

commit 75a68825ec487ea888017b2f9340718ae3a8c5a7
    youtube-dl 2018.02.26 (#24529)
commit 9d4eb3de3364d7cad20b31ef3c518abb56a5dc13
    youtube-dl 2018.02.22 (#24440)
commit bbfa68ab39d5cafd3e42bda6030f8dd44c1db05b
    youtube-dl 2018.02.11
commit 7ff05a1d5bc5e387afce4f8e5047fa40c02a16cf
    youtube-dl 2018.02.08
commit 78a7cec7944640b15098c2207ce09aeba9221364
    youtube-dl 2018.02.04 (#23650)

Comparing to https://github.com/rg3/youtube-dl/releases, this one is missing but hopefully not relevant: https://github.com/rg3/youtube-dl/releases/tag/2018.02.25

These were the ones already on my system:

$ brew list youtube-dl --versions
youtube-dl ... 2018.01.27 2018.02.11 2018.02.22 2018.02.26

This meant I needed to install these ones:

commit 7ff05a1d5bc5e387afce4f8e5047fa40c02a16cf
    youtube-dl.2018.02.08
commit 78a7cec7944640b15098c2207ce09aeba9221364
    youtube-dl 2018.02.04 (#23650)

This got them:

pushd "$(brew --repo homebrew/core)"
git checkout -b youtube-dl.2018.02.08 7ff05a1d5bc5e387afce4f8e5047fa40c02a16cf
brew unlink youtube-dl
HOMEBREW_NO_AUTO_UPDATE=1 brew install youtube-dl
git checkout -b youtube-dl.2018.02.04 78a7cec7944640b15098c2207ce09aeba9221364
brew unlink youtube-dl
HOMEBREW_NO_AUTO_UPDATE=1 brew install youtube-dl
git checkout master
git branch -d youtube-dl.2018.02.08
git branch -d youtube-dl.2018.02.04

as the new version list shows:

$ brew list youtube-dl --versions
youtube-dl ... 2018.01.27 2018.02.04 2018.02.08 2018.02.11 2018.02.22 2018.02.26

Binary search

Fails:

brew switch youtube-dl 2018.02.26
youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590

Succeeds:

brew switch youtube-dl 2018.02.04
youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590
rm Het\ is\ hier\ Autistisch\ -\ Onderwijs-BV_101386590.m4v

Succeeds:

brew switch youtube-dl 2018.02.11
youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590
rm Het\ is\ hier\ Autistisch\ -\ Onderwijs-BV_101386590.m4v

Fails:

brew switch youtube-dl 2018.02.22
youtube-dl -v -k https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590

Failure:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-k', 'https://www.npo.nl/het-is-hier-autistisch/01-03-2018/BV_101386590']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.02.22
[debug] Python version 3.6.4 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.4.2, ffprobe 3.4.2
[debug] Proxy map: {}
[npo] BV_101386590: Downloading JSON metadata
[npo] BV_101386590: Downloading token
[npo] BV_101386590: Downloading formats JSON
[npo] BV_101386590: Downloading Adaptive stream JSON
[npo] BV_101386590: Downloading m3u8 information
[npo] BV_101386590: Downloading Hoog stream JSON
[npo] BV_101386590: Downloading Normaal stream JSON
[npo] BV_101386590: Downloading Laag stream JSON
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'npo+drm://internetvod.omroep.nl/BV_101386590'
ERROR: unable to download video data: <urlopen error unknown url type: npo+drm>
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1896, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1835, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 340, in real_download
    establish_connection()
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2199, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 549, in _open
    'unknown_open', req)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.6.4_3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1388, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: npo+drm>

My gut feeling is that one of these is the culprit as they are all marked [npo]:

  • b8adcec: [npo] Capture and output error message
  • c4e7496: [npo] Relax _VALID_URL (closes #14987, closes #14994)
  • ee706f1: [npo] Improve quality metadata extraction

If time permits, I will try getting individual commits and try to binary search even further.
@dstftw dstftw closed this Mar 2, 2018
@dstftw dstftw added the duplicate label Mar 2, 2018
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 2, 2018

  1. Stop posting duplicates and wait for the next release.
  2. There is no regression. Breakage is caused by changes at npo side.
@ytdl-org ytdl-org locked and limited conversation to collaborators Mar 2, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
duplicate
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@dstftw @jpluimers
You can’t perform that action at this time.