Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: fix for password protected videos #28695

Closed
wants to merge 1 commit into from

Conversation

mmuman
Copy link
Contributor

@mmuman mmuman commented Apr 7, 2021

It seems accessing password-protected videos have changed. Checking one from vimeo.com I noticed it was POSTing to player.vimeo.com for check-password and noticed the only codepath doing it in the extractor was not taken.
It also happens for videos already in the tests:

For videos like http://vimeo.com/68375962 we had:
Unable to download JSON metadata: HTTP Error 403: Forbidden

I'm not yet sure if it's required every time or if it should be conditional to some URL match though…

For videos like http://vimeo.com/68375962 we had:
Unable to download JSON metadata: HTTP Error 403: Forbidden
@remitamine
Copy link
Collaborator

remitamine commented Apr 8, 2021

the tests pass without a problem both in CI and locally without any changes:

python test_download.py TestDownload.test_Vimeo_3
[vimeo] 68375962: Downloading webpage
[vimeo] 68375962: Verifying the password
[vimeo] 68375962: Downloading JSON metadata
[vimeo] 68375962: Downloading JSON metadata
[vimeo] 68375962: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 68375962: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 68375962: Downloading fastly_skyfire m3u8 information
[vimeo] 68375962: Downloading fastly_skyfire m3u8 information
[vimeo] 68375962: Downloading akfire_interconnect_quic MPD information
[vimeo] 68375962: Downloading akfire_interconnect_quic MPD information
[vimeo] 68375962: Downloading fastly_skyfire MPD information
[vimeo] 68375962: Downloading fastly_skyfire MPD information
[info] Writing video description metadata as JSON to: test_Vimeo_3_68375962.info.json
[debug] Invoking downloader on 'https://vod-progressive.akamaized.net/exp=1617876075~acl=%2A%2F171158304.mp4%2A~hmac=7a5d9d5baa6cc2d5a9049a9b8e5f73fbe97b2f5af7e6fde6b9f6a69c8ed60fa5/vimeo-prod-skyfire-std-us/01/3675/2/68375962/171158304.mp4'
[download] Destination: test_Vimeo_3_68375962.mp4
[download] 100% of 10.00KiB in 00:01
.
----------------------------------------------------------------------
Ran 1 test in 20.226s

OK
python test_download.py TestDownload.test_Vimeo_10
[vimeo] 68375962: Downloading webpage
[vimeo] 68375962: Verifying the password
[vimeo] 68375962: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 68375962: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 68375962: Downloading fastly_skyfire m3u8 information
[vimeo] 68375962: Downloading fastly_skyfire m3u8 information
[vimeo] 68375962: Downloading akfire_interconnect_quic MPD information
[vimeo] 68375962: Downloading akfire_interconnect_quic MPD information
[vimeo] 68375962: Downloading fastly_skyfire MPD information
[vimeo] 68375962: Downloading fastly_skyfire MPD information
[info] Writing video description metadata as JSON to: test_Vimeo_10_68375962.info.json
[debug] Invoking downloader on 'https://vod-progressive.akamaized.net/exp=1617876129~acl=%2A%2F171158304.mp4%2A~hmac=fd3aea4aa519e1d14b5980577654338d27b3604315dc14e24d6b07b40c69a9bd/vimeo-prod-skyfire-std-us/01/3675/2/68375962/171158304.mp4'
[download] Destination: test_Vimeo_10_68375962.mp4
[download] 100% of 10.00KiB in 00:00
.
----------------------------------------------------------------------
Ran 1 test in 5.955s

OK

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

Here (in France, maybe they started screwing up only in France?) I have this (and same with python2) (on Debian Sid) using 545d6cb:

$ python3 test_download.py TestDownload.test_Vimeo_3
[vimeo] 68375962: Downloading webpage
[vimeo] 68375962: Verifying the password
[vimeo] 68375962: Downloading JSON metadata
[vimeo] 68375962: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/vimeo.py", line 747, in _real_extract
    source_format = self._extract_original_format(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/vimeo.py", line 234, in _extract_original_format
    download_data = self._download_json(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 895, in _download_json
    res = self._download_json_handle(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 874, in _download_json_handle
    res = self._download_webpage_handle(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 654, in _request_webpage
    self._downloader.report_warning(errmsg)
  File "/mnt/data/devel/youtube-dl/youtube-dl/test/helper.py", line 271, in _report_warning
    real_warning(w)
  File "/mnt/data/devel/youtube-dl/youtube-dl/test/test_download.py", line 52, in report_warning
    raise ExtractorError(message)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

E
======================================================================
ERROR: test_Vimeo_3 (__main__.TestDownload):
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/vimeo.py", line 747, in _real_extract
    source_format = self._extract_original_format(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/vimeo.py", line 234, in _extract_original_format
    download_data = self._download_json(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 895, in _download_json
    res = self._download_json_handle(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 874, in _download_json_handle
    res = self._download_webpage_handle(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 667, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/extractor/common.py", line 654, in _request_webpage
    self._downloader.report_warning(errmsg)
  File "/mnt/data/devel/youtube-dl/youtube-dl/test/helper.py", line 271, in _report_warning
    real_warning(w)
  File "/mnt/data/devel/youtube-dl/youtube-dl/test/test_download.py", line 52, in report_warning
    raise ExtractorError(message)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/devel/youtube-dl/youtube-dl/test/test_download.py", line 157, in test_template
    res_dict = ydl.extract_info(
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 799, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    self.report_error(compat_str(e), e.format_traceback())
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 628, in report_error
    self.trouble(error_message, tb)
  File "/mnt/data/devel/youtube-dl/youtube-dl/youtube_dl/YoutubeDL.py", line 598, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

----------------------------------------------------------------------
Ran 1 test in 5.337s

FAILED (errors=1)

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

What the…

I tried to test with proxies outside of France but none of them seem to work anyway, then I tested from another ISP using a SOCKS proxy through SSH… And it works.

So basically it only fails from a connection at Free, while it works fine from Orange. Either my ISP is doing some weirder things that I knew they do, or Vimeo decided to piss off Free in some way…

I thought it was why the password is asked twice (once on the page, then once again in the player on the black background), but no, it also happens from the Orange connection.

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

Asked others on Mastodon and it seems to confirm the case…

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

Another explanation would be not all their frontal servers have the same version and depending on the IP you use to connect you get one or the other… someone reported they heard an issue with Orange as well…

@remitamine
Copy link
Collaborator

when I try to access password-protected videos using a french proxy, Vimeo is asking for the password twice, this is clearly a bug that should be addressed by Vimeo on their website.
after entering the password for the first time:
Screenshot from 2021-04-08 14-30-29
Vimeo asks for the password again:
Screenshot from 2021-04-08 14-30-57
the user is not supposed to enter the password multiple times, so i suggest that you address this problem to the Vimeo team.

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

Except the test actually works from another connection where the web browser does ask the password twice… go figure.

@mmuman
Copy link
Contributor Author

mmuman commented Apr 8, 2021

Couldn't find their bug tracker so asked them on twitter, let's see…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants