Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Youtube API broken getting channel playlist, leading to duplicated videos and many pages #6392

Closed
9 of 11 tasks
JeffreyMercado opened this issue Mar 2, 2023 · 1 comment
Closed
9 of 11 tasks

Youtube API broken getting channel playlist, leading to duplicated videos and many pages #6392

JeffreyMercado opened this issue Mar 2, 2023 · 1 comment
Labels
duplicate This issue or pull request already exists site-bug Issue with a specific website

Comments

@JeffreyMercado
Copy link

JeffreyMercado commented Mar 2, 2023 •
edited

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

  • I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

Region

United States

Provide a description that is worded well enough to be understood

Youtube has been making a lot of changes breaking the UI on desktop, but it appears it is broken in the API as well. After reaching the end of the list, it will loop and repeat all pages, and this appears to happen indefinitely. https://www.youtube.com/@lexfridman demonstrates this for example. While it will eventually stop and proceed to download, it'll take way too long to wait for it to finish. You can go into youtube right now and scroll to the bottom of a page and it will go on, and on, and on, and so on and you can see the duplicated videos.

It seems to be a problem when there are more than 25 or so pages by my guess. (have a channel with 24 that doesn't have the problem: https://www.youtube.com/@Pikamee/streams) Only noticed when archiving a channel that it reported over 200k videos which is not correct.

This will probably be fixed in Youtube eventually, but is there something that could be done to catch this and mitigate the problem? Maybe add an option to detect if there are pages of duplicated ids and stop processing after a threshold?

Provide verbose output that clearly demonstrates the problem

  • Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
  • If using API, add 'verbose': True to YoutubeDL params instead
  • Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

$ yt-dlp --no-config -vU -s https://www.youtube.com/@lexfridman
[debug] Command-line config: ['--no-config', '-vU', '-s', 'https://www.youtube.com/@lexfridman']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-108549-g3d6d127cd0-20221007 (setts), ffprobe N-108549-g3d6d127cd0-20221007, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[youtube:tab] Extracting URL: https://www.youtube.com/@lexfridman
[youtube:tab] @lexfridman: Downloading webpage
[debug] [youtube:tab] Selected tab: 'videos' (videos), Requested tab: ''
[youtube:tab] Downloading all uploads of the channel. To download only the videos in a specific tab, pass the tab's URL
[youtube:tab] @lexfridman/streams: Downloading webpage
[debug] [youtube:tab] Selected tab: 'streams' (live), Requested tab: 'streams'
[youtube:tab] @lexfridman/shorts: Downloading webpage
[debug] [youtube:tab] Selected tab: 'shorts' (shorts), Requested tab: 'shorts'
[youtube:tab] Downloading as multiple playlists, separated by tabs. To download as a single playlist instead, pass https://www.youtube.com/playlist?list=UUSHZKyawb77ixDdsGog4iWA
[download] Downloading playlist: Lex Fridman
[youtube:tab] Playlist Lex Fridman: Downloading 3 items of 3
[download] Downloading item 1 of 3
[download] Downloading playlist: Lex Fridman - Videos
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 1: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 2: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 3: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 4: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 5: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 6: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 7: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 8: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 9: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 10: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 11: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 12: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 13: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 14: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 15: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 16: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 17: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 18: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 19: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 20: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 21: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 22: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 23: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 24: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 25: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 26: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 27: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 28: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 29: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 30: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 31: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 32: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 33: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 34: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 35: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 36: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 37: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 38: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 39: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 40: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 41: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 42: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 43: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 44: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 45: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 46: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 47: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 48: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 49: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 50: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 51: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 52: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 53: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 54: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 55: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 56: Downloading API JSON
[youtube:tab] UCSHZKyawb77ixDdsGog4iWA page 57: Downloading API JSON

ERROR: Interrupted by user

$ yt-dlp --no-config -vU -s https://www.youtube.com/@Pikamee/streams
[debug] Command-line config: ['--no-config', '-vU', '-s', 'https://www.youtube.com/@Pikamee/streams']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-108549-g3d6d127cd0-20221007 (setts), ffprobe N-108549-g3d6d127cd0-20221007, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[youtube:tab] Extracting URL: https://www.youtube.com/@Pikamee/streams
[youtube:tab] @Pikamee/streams: Downloading webpage
[debug] [youtube:tab] Selected tab: 'streams' (live), Requested tab: 'streams'
[download] Downloading playlist: 天野ピカミィ. Pikamee - Live
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 1: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 2: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 3: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 4: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 5: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 6: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 7: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 8: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 9: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 10: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 11: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 12: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 13: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 14: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 15: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 16: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 17: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 18: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 19: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 20: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 21: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 22: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 23: Downloading API JSON
[youtube:tab] UCajhBT4nMrg3DLS-bLL2RCg page 24: Downloading API JSON
[youtube:tab] Playlist 天野ピカミィ. Pikamee - Live: Downloading 745 items of 745

ERROR: Interrupted by user
@JeffreyMercado JeffreyMercado added site-bug Issue with a specific website triage Untriaged issue labels Mar 2, 2023
@coletdjnz
Copy link
Member

Think this is the same issue as #5555

Seems similar to the issue we had for comments too: #6290

Wonder what youtube is doing 🤔

@pukkandan pukkandan closed this as not planned Won't fix, can't repro, duplicate, stale Mar 2, 2023
@pukkandan pukkandan added duplicate This issue or pull request already exists and removed triage Untriaged issue labels Mar 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate This issue or pull request already exists site-bug Issue with a specific website
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JeffreyMercado @pukkandan @coletdjnz