New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[extractor/tencent]: Add more formats and info #5950
Conversation
Can you give an example with DRM? |
4K and HDR videos are available on the Tencent Video website (v.qq.com) in China, but they will only be enabled if the browser supports HEVC decoding (which is supported by the latest Chrome). Here's an example: https://v.qq.com/x/cover/lqoxvyv6a23jo9o/b00440jumct.html |
There was a typo in my comment 🤦. I meant to ask for an example with DRM, sorry |
f49f600
to
95bc926
Compare
* Use HEVC codec * Supports 4K and HDR * More information
95bc926
to
b72208d
Compare
Please avoid force pushing if possible. The merge will be squashed, so take as many commits as you need. |
Ok, I get it. |
e1e226d
to
62bd9c2
Compare
yt_dlp/extractor/tencent.py
Outdated
def _extract_all_video_formats_and_subtitles(self, url, video_id, series_id): | ||
first_api_response = self._get_video_api_response(url, video_id, series_id, 'vtt', 'hls', 'hd') | ||
self._check_api_response(first_api_response) | ||
api_responses = [first_api_response] | ||
quality = self._extract_all_video_quality(first_api_response) | ||
for q in quality: | ||
if q != 'sd' and q != 'hd': | ||
api_response = self._get_video_api_response( | ||
url, video_id, series_id, 'vtt', 'hls', q) | ||
self._check_api_response(api_response) | ||
api_responses.append(api_response) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current yt-dlp is able to get 480p direct links, now those are missing
yt-dlp 2022.10.04
[debug] Command-line config: ['https://wetv.vip/play/7csb8h9c79p8idi/a0041yw44yk', '-F', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f27] (debian*) API
[debug] Python 3.10.7 (CPython 64bit) - Linux-5.19.0-26-generic-x86_64-with-glibc2.36 (glibc 2.36)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-108931-g4dda3b1653-20221104 (setts), ffprobe N-108931-g4dda3b1653-20221104, phantomjs 140337668638656, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, pyxattr-0.7.2, secretstorage-3.3.3, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] [wetv:episode] Extracting URL: https://wetv.vip/play/7csb8h9c79p8idi/a0041yw44yk
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for a0041yw44yk:
ID EXT RESOLUTION │ PROTO │ VCODEC ACODEC
───────────────────────────────────────────
0 mp4 864x362 │ https │ unknown unknown
1 mp4 864x362 │ https │ unknown unknown
2 mp4 864x362 │ https │ unknown unknown
3 mp4 864x362 │ https │ unknown unknown
4 mp4 1280x536 │ m3u8 │ unknown unknown
5 mp4 1280x536 │ m3u8 │ unknown unknown
6 mp4 1280x536 │ m3u8 │ unknown unknown
7 mp4 1280x536 │ m3u8 │ unknown unknown
Your branch
[debug] Command-line config: ['https://wetv.vip/play/7csb8h9c79p8idi/a0041yw44yk', '--verbose', '-F']
[debug] User config: []
[debug] System config: []
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.02 [d83b0ad80] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 62bd9c276
[debug] Python 3.10.7 (CPython x86_64 64bit) - Linux-5.19.0-26-generic-x86_64-with-glibc2.36 (OpenSSL 3.0.5 5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg N-108931-g4dda3b1653-20221104 (setts), ffprobe N-108931-g4dda3b1653-20221104, phantomjs broken, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1756 extractors
[wetv:episode] Extracting URL: https://wetv.vip/play/7csb8h9c79p8idi/a0041yw44yk
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading webpage
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[wetv:episode] a0041yw44yk: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for a0041yw44yk:
ID EXT RESOLUTION FPS │ PROTO │ VCODEC ACODEC MORE INFO
──────────────────────────────────────────────────────────
0 mp4 256x108 0 │ m3u8 │ unknown unknown ld (144P)
1 mp4 256x108 0 │ m3u8 │ unknown unknown ld (144P)
2 mp4 864x362 0 │ m3u8 │ unknown unknown hd (480P)
3 mp4 864x362 0 │ m3u8 │ unknown unknown hd (480P)
4 mp4 1280x536 0 │ m3u8 │ unknown unknown shd (720P)
5 mp4 1280x536 0 │ m3u8 │ unknown unknown shd (720P)
6 mp4 1280x536 0 │ m3u8 │ unknown unknown shd (720P)
7 mp4 1280x536 0 │ m3u8 │ unknown unknown shd (720P)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is expected because the list of qualities returned by the API will not include 4K and HDR if the API is requested without the hls format.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should call both API? Removing existing formats is a regression and should only be done when there is no other choice
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pukkandan, the extraction speed for those outside of China is pretty bad, especially if you are using proxy to unblock content. In my daily use, with proxy, it takes 15–30 seconds to extract all AVC formats, in which case, extracting AVC and HEVC would double, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My opinion is to add all formats by default, and have an extract-arg to limit it. I think the current video_codec
is too limited. We still have to download multiple m3u8s for a single codec. Something like formats=hevc-hd,avc-fhd
would be better imo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, we'd better not fetch both avc and hevc, even in China, it's a burden, and too many calls to the API might get blocked.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright. What do you think of the suggested change to extractor-arg?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean change video_codec
to formats
? Looks good, but doesn't add much value, and I think very few people need to download videos with both encoders.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused. Calling the API is expensive, but an option to reduce number of m3u8 downloads doesn't add value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, my fault, I misunderstood. It does have value, and I'll consider it when I have time.
By the way, I actually want to disassemble the implementation of vqq and WeTV. Although they are very similar at the API level, they are still different in terms of content. This can also reduce maintenance costs, and Chinese users can only focus on vqq. |
7c83c61
to
091bbc8
Compare
The Japanese proxy I used was tested on WeTV, so there is no guarantee that it will work in all countries. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are all tests up to date? I get an error on WeTvEpisode extractor
python3 test/test_download.py TestDownload.test_WeTvEpisode
[debug] Loaded 1756 extractors
[wetv:episode] Extracting URL: https://wetv.vip/en/play/air11ooo2rdsdi3-Cute-Programmer/v0040pr89t9-EP1-Cute-Programmer
[wetv:episode] v0040pr89t9: Downloading webpage
[wetv:episode] v0040pr89t9: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] v0040pr89t9: Downloading 1 format(s): 3
[info] Writing video metadata as JSON to: test_WeTvEpisode_v0040pr89t9.info.json
[debug] Invoking http downloader on "https://apd-vlive.apdcdn.tc.qq.com/wetvvlive.qqvideo.tc.qq.com/uwMRJ_z-r5pAYaQXGdGnFWde8_uKQ6p7LL8hY22lcu2s4A/svp_50125/gzc_1000117_0b53jyaaaaaatiaonjkd5nq42twdabbaabka.f2.mp4?vkey=B96FD54DD4DD4C114A919FAE44447567AE2F3FA0993D6D7106DF924AB8BD05E2077D207866808D7AD3CE3023B06A792AD4F0618130D78D81C95AD11EDA98297F826C9AC984520A296B54F34EA281BD3C28BDBDF0668F22F9D215652D3C545D1472DE3767FDB5868AE68B4EBEF89356E897B0CCD8DBD78F6B25798A649E2EF7EB"
[download] Destination: test_WeTvEpisode_v0040pr89t9.mp4
[download] 100% of 10.00KiB in 00:00:00 at 55.39KiB/s
F
======================================================================
FAIL: test_WeTvEpisode (__main__.TestDownload):
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/amish/Documents/Programming/yt-dlp/test/test_download.py", line 215, in test_template
expect_info_dict(self, tc_res_dict, tc.get('info_dict', {}))
File "/home/amish/Documents/Programming/yt-dlp/test/helper.py", line 233, in expect_info_dict
expect_dict(self, got_dict, expected_dict)
File "/home/amish/Documents/Programming/yt-dlp/test/helper.py", line 186, in expect_dict
expect_value(self, got, expected, info_field)
File "/home/amish/Documents/Programming/yt-dlp/test/helper.py", line 178, in expect_value
self.assertEqual(
AssertionError: 'shd' != 'hd'
- shd
? -
+ hd
: Invalid value for field format_id , expected 'shd', got 'hd'
----------------------------------------------------------------------
Ran 1 test in 1.710s
FAILED (failures=1)
@elyse0 Yes, All Test is update. My test results:
|
Excellent extractor improvement, really appreciated. |
@D0LLYNH0 That's a good idea, I'll look at how to add parameters to extractor later. |
This reverts commit 8c5821e.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have reverted the extractor-arg for now. Once you confirm I haven't broken anything, we can merge this PR. You can make a separate PR with the arg later
Still waiting for confirmation |
I'll test it when I have time, and wait for some time. |
👍 Just wanted to make sure the message was seen |
54efafd
to
630a7ed
Compare
Nothing is broken. But I made some small changes to make the format_note display the same as the web page. But I don't know why the GitHub Action check fails. |
We don't want it to be same as website. The field is used for adding additional data that doesn't fit in the other fields |
630a7ed
to
0a74b60
Compare
Okay, that's no more questions, it can be merged. I will do PR for arg when I have time. |
Authored by: Hill-98
Description of your pull request and other information
Template
Before submitting a pull request make sure you have:
In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:
What is the purpose of your pull request?