Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ie/tiktok] Fix API extraction #9548

Merged
merged 9 commits into from Mar 29, 2024
Merged

Conversation

bashonly
Copy link
Member

@bashonly bashonly commented Mar 28, 2024

The mobile API feed endpoint that is used to extract high-resolution/no-watermark formats has begun checking/enforcing the iid (install id) query param for requests made to it.

A unique install ID is assigned to the mobile app installation by the TT API upon device registration. This PR hardcodes 3 of them that were donated for use in the extractor. The app versions fallback/cache loop has been replaced by an install IDs fallback/cache loop (since having multiple app versions hasn't been useful for quite some time).

The PR also adds an iid extractor-arg so that users can bring their own install ID if they so wish (or if the hardcoded iids are failing). This also necessitated an app_name extractor-arg,

The PR adds an app_info extractor-arg, which allows one or more strings in the format of <iid>/[app_name]/[app_version]/[manifest_app_version]/[aid] to be passed, since any given install ID is tethered to the app variant it was assigned to (trill for KR/PH/TW/TH/VN or musical_ly for rest of world) and each variant has its own versioning and aid (app ID).

Closes #9506

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense

What is the purpose of your pull request?

  • Fix or improvement to an extractor (Make sure to add/update tests)

@bashonly bashonly added the site-bug Issue with a specific website label Mar 28, 2024
yt_dlp/extractor/tiktok.py Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@bashonly bashonly added pending-fixes PR has had changes requested pending-review PR needs a review and removed pending-fixes PR has had changes requested labels Mar 28, 2024
@pukkandan pukkandan removed the pending-review PR needs a review label Mar 29, 2024
Authored by: bashonly
@bashonly bashonly merged commit cb61e20 into yt-dlp:master Mar 29, 2024
6 checks passed
@JailSeed
Copy link

Sorry if I'm writing in the wrong topic, but this fix downloads some videos with an strange codec bytevc2 if using -S res:1080,vcodec:avc1,ext:mp4:m4a , which leads to the fact that the video cannot be played.

Formats:

user@host:~$ python3 -m yt_dlp -F https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[TikTok] 7351538939712359713: Downloading video feed
[info] Available formats for 7351538939712359713:
ID                    EXT RESOLUTION │   FILESIZE  TBR PROTO │ VCODEC  ACODEC MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
download_addr-0       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked (API)
download_addr-1       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
download_addr-2       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
h264_540p_986746-0    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video (API)
h264_540p_986746-1    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
h264_540p_986746-2    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
bytevc1_540p_263555-0 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video (API)
bytevc1_540p_263555-1 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_263555-2 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_344910-0 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video (API)
bytevc1_540p_344910-1 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_344910-2 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_507345-0 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video (API)
bytevc1_540p_507345-1 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc1_540p_507345-2 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc2_720p_616180-0 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video (API)
bytevc2_720p_616180-1 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc2_720p_616180-2 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc1_720p_595186-0 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video (API)
bytevc1_720p_595186-1 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video
bytevc1_720p_595186-2 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video

Example:

user@host:~$ python3 -m yt_dlp -vU -S res:1080,vcodec:avc1,ext:mp4:m4a https://vm.tiktok.com/ZMMPDNEJL/
[debug] Command-line config: ['-vU', '-S', 'res:1080,vcodec:avc1,ext:mp4:m4a', 'https://vm.tiktok.com/ZMMPDNEJL/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.03.10 from yt-dlp/yt-dlp [615a84447]
[debug] Lazy loading extractors is disabled
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-101-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1807 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.03.10 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.03.10 from yt-dlp/yt-dlp)
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[debug] [TikTok] iid=7351149742343391009
[TikTok] 7351538939712359713: Downloading video feed
[debug] Sort order given by user: res:1080, vcodec:avc1, ext:mp4:m4a
[debug] Sort order given by extractor: quality, codec, size, br
[debug] Formats sorted by: hasvid, ie_pref, res:1080(1080.0), vcodec:avc1(7), vext:mp4(6), aext:m4a(8), quality, acodec, size, br, lang, fps, hdr:12(7), channels, asr, proto, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 7351538939712359713: Downloading 1 format(s): bytevc2_720p_616180-2
[debug] Invoking http downloader on "https://v16m.byteicdn.com/366e989edb75b43b4e545b0f6f94180e/6607a32c/video/tos/useast2a/tos-useast2a-ve-0068-euttp/oIEDMCMe2rmeIExnh6mGk70AEkerG1aIRgfFj6/?a=0&bti=OHYpOTY0Zik3OjlmOm01MzE6ZDQ0MDo%3D&ch=0&cr=13&dr=0&lr=all&cd=0%7C0%7C0%7C&cv=1&br=1202&bt=601&cs=5&ds=3&ft=teSL~8QLodzR12NvvEh3hIxR34DaRq_45SY&mime_type=video_mp4&qs=14&rc=aTwzOGdkOGRnMzw1NDQ1OUBpam9pbng5cjx4cjMzZjczM0AvL18yYmBgXjMxXy4tLy5eYSNsa2FnMmRrZy9gLS1kMWNzcw%3D%3D&vvpl=1&l=20240329232904ECF8D33243F1CE142F0B&btag=e00088000&cc=10"
[download] Destination: Оригинал) [7351538939712359713].mp4
[download] 100% of  794.82KiB in 00:00:00 at 7.49MiB/s

@bashonly
Copy link
Member Author

@JailSeed can you open a new issue please?

@JailSeed
Copy link

@JailSeed can you open a new issue please?

Sure

haccy pushed a commit to haccy/yt-dlp that referenced this pull request Mar 30, 2024
Closes yt-dlp#9506
Authored by: bashonly, Grub4K

Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024
Closes yt-dlp#9506
Authored by: bashonly, Grub4K

Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
@bashonly bashonly deleted the fix/tiktok-api branch May 10, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
site-bug Issue with a specific website
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TikTok] Failed to parse JSON/ No video formats found
5 participants