Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extractor/npo.py API query now needs to be POST, not GET #6398

Closed
11 tasks done
VDK opened this issue Mar 2, 2023 · 6 comments · Fixed by #8413
Closed
11 tasks done

extractor/npo.py API query now needs to be POST, not GET #6398

VDK opened this issue Mar 2, 2023 · 6 comments · Fixed by #8413
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website

Comments

@VDK
Copy link

VDK commented Mar 2, 2023

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

Netherlands

Provide a description that is worded well enough to be understood

npo.py line 240 has a json request that needs to be done with a POST method, where it previously was a GET.

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt-dlp https://www.npostart.nl/vpro-boeken/06-01-2019/VPWON_1298036 -vU
[debug] Command-line config: ['https://www.npostart.nl/vpro-boeken/06-01-2019/VPWON_1298036', '-vU']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.10.0 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1l  24 Aug 2021)
[debug] exe versions: ffmpeg 5.1.2-full_build-www.gyan.dev (setts), ffprobe 5.1.2-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2021.10.08, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[npo] Extracting URL: https://www.npostart.nl/vpro-boeken/06-01-2019/VPWON_1298036
[npo] VPWON_1298036: Downloading token
[npo] VPWON_1298036: Downloading player JSON
[npo] VPWON_1298036: Downloading hls profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1298036: Downloading dash-widevine profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1298036: Downloading dash-playready profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1298036: Downloading smooth profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1298036: Downloading embed page
ERROR: [npo] VPWON_1298036: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
Traceback (most recent call last):
  File "C:\Users\Vera\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1502, in wrapper
    return func(self, *args, **kwargs)
  File "C:\Users\Vera\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1599, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "C:\Users\Vera\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1658, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "C:\Users\Vera\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 2599, in process_video_result
    self.raise_no_formats(info_dict)
  File "C:\Users\Vera\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 1043, in raise_no_formats
    raise ExtractorError(msg, video_id=info['id'], ie=info['extractor'],
yt_dlp.utils.ExtractorError: [npo] VPWON_1298036: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
@VDK VDK added site-bug Issue with a specific website triage Untriaged issue labels Mar 2, 2023
@VDK VDK changed the title NPO.py API query now needs to be POST, not GET extractor/npo.py API query now needs to be POST, not GET Mar 2, 2023
@october262
Copy link

the video is DRM protected.

@VDK
Copy link
Author

VDK commented Mar 2, 2023

no the way how the API works has changed since monday. This Kodi video add-on was updated yeseterday

@pukkandan pukkandan added DRM The referred content is DRM protected and removed triage Untriaged issue labels Mar 3, 2023
@bashonly
Copy link
Member

bashonly commented Mar 5, 2023

OP's example URL is DRM-protected, but @VDK is correct that the API is requiring POST requests even for non-DRM content, e.g. https://www.npo3.nl/the-genius/21-11-2022/VPWON_1341105

diff --git a/yt_dlp/extractor/npo.py b/yt_dlp/extractor/npo.py
index 40fee24d0..9150b91c5 100644
--- a/yt_dlp/extractor/npo.py
+++ b/yt_dlp/extractor/npo.py
@@ -245,7 +245,7 @@ def _real_extract(self, url):
                     'quality': 'npoplus',
                     'tokenId': player_token,
                     'streamType': 'broadcast',
-                })
+                }, data=b'')
             if not streams:
                 continue
             stream = streams.get('stream')

with the above patch applied:

$ yt-dlp -F 'https://www.npo3.nl/the-genius/21-11-2022/VPWON_1341105'
[npo] Extracting URL: https://www.npo3.nl/the-genius/21-11-2022/VPWON_1341105
[npo] VPWON_1341105: Downloading token
[npo] VPWON_1341105: Downloading player JSON
[npo] VPWON_1341105: Downloading hls profile JSON
[npo] VPWON_1341105: Downloading m3u8 information
[npo] VPWON_1341105: Downloading dash-widevine profile JSON
[npo] VPWON_1341105: Downloading MPD manifest
[npo] VPWON_1341105: Downloading dash-playready profile JSON
[npo] VPWON_1341105: Downloading MPD manifest
[npo] VPWON_1341105: Downloading smooth profile JSON
[npo] VPWON_1341105: Downloading ISM manifest
[npo] VPWON_1341105: Downloading embed page
[info] Available formats for VPWON_1341105:
ID                  EXT  RESOLUTION │   FILESIZE  TBR PROTO │ VCODEC       VBR ACODEC      ABR ASR MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
dash-audio=128001-0 m4a  audio only │ ~ 41.53MiB 128k dash  │ audio only       mp4a.40.2  128k 48k DASH audio, m4a_dash
dash-audio=128001-1 m4a  audio only │ ~ 41.53MiB 128k dash  │ audio only       mp4a.40.2  128k 48k DASH audio, m4a_dash
hls-136             mp4  audio only │ ~ 44.13MiB 136k m3u8  │ audio only       mp4a.40.2  128k
mss-audio-128       isma audio only │ ~ 41.53MiB 128k ism   │ audio only       AACL       128k 48k
mss-video-453       ismv 640x360    │ ~146.98MiB 453k ism   │ AVC1        453k video only
dash-video=453222-0 mp4  640x360    │ ~147.05MiB 453k dash  │ avc1.42C01E 453k video only          DASH video, mp4_dash
dash-video=453222-1 mp4  640x360    │ ~147.05MiB 453k dash  │ avc1.42C01E 453k video only          DASH video, mp4_dash
hls-617             mp4  640x360    │ ~200.19MiB 617k m3u8  │ avc1.66.30  453k mp4a.40.2  128k
mss-video-762       ismv 960x540    │ ~247.24MiB 762k ism   │ AVC1        762k video only
dash-video=762183-0 mp4  960x540    │ ~247.30MiB 762k dash  │ avc1.4D401F 762k video only          DASH video, mp4_dash
dash-video=762183-1 mp4  960x540    │ ~247.30MiB 762k dash  │ avc1.4D401F 762k video only          DASH video, mp4_dash
hls-944             mp4  960x540    │ ~306.29MiB 944k m3u8  │ avc1.77.31  762k mp4a.40.2  128k

without:

$ yt-dlp -F 'https://www.npo3.nl/the-genius/21-11-2022/VPWON_1341105'
[npo] Extracting URL: https://www.npo3.nl/the-genius/21-11-2022/VPWON_1341105
[npo] VPWON_1341105: Downloading token
[npo] VPWON_1341105: Downloading player JSON
[npo] VPWON_1341105: Downloading hls profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1341105: Downloading dash-widevine profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1341105: Downloading dash-playready profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1341105: Downloading smooth profile JSON
WARNING: [npo] Unable to download JSON metadata: HTTP Error 405: Method Not Allowed
[npo] VPWON_1341105: Downloading embed page
ERROR: [npo] VPWON_1341105: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

this is likely not a complete fix though, as some of the other tests in the extractor still fail

@pukkandan pukkandan added patch-available There is patch available that should fix this issue. Someone needs to make a PR with it and removed DRM The referred content is DRM protected labels Mar 5, 2023
@ttencate
Copy link

ttencate commented Aug 2, 2023

I can confirm that the patch works.

@dirkf
Copy link
Contributor

dirkf commented Aug 2, 2023

Seems like the playlist IEs inherit from NPOIE for no reason?

@dirkf
Copy link
Contributor

dirkf commented Aug 4, 2023

Also:

  • the live radio page has changed URL and structure
  • do radio fragments still exist?
  • when handling npo:NPO_ID either from command-line or from redirection by a subsidiary extractor, the npo:... URL can't be passed in the API requests (HTTP 500).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
patch-available There is patch available that should fix this issue. Someone needs to make a PR with it site-bug Issue with a specific website
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants