Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[soundcloud] Reduce pagination limit to fix 502 Bad Gateway errors when listing a user's tracks. #26557

Merged
merged 1 commit into from
Sep 12, 2020

Conversation

tfvlrue
Copy link
Contributor

@tfvlrue tfvlrue commented Sep 9, 2020

Please follow the guide below

  • You will be asked some questions, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your pull request (like that [x])
  • Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

Per the documentation here https://developers.soundcloud.com/blog/offset-pagination-deprecated the maximum limit value is actually 200, however 2000 seems to work fine. The exact upper limit I found experimentally was 2847 (anything above the returns a 502 Bad Gateway error). I figured 2000 would provide some leeway in case the actual limit comes down again.

This limit was previously modified by b334732 after many users reported the 502 errors back in May. It seems to have cropped up again, and reducing the limit further seems to fix the problem. @remitamine might be interested in this since they merged in the last fix.

Before:

$ youtube-dl https://soundcloud.com/mediasanctuary
[soundcloud:user] mediasanctuary: Downloading user info
[soundcloud:user] 164222112: Downloading track page 1
ERROR: Unable to download JSON metadata: HTTP Error 502: Bad Gateway (caused by <HTTPError 502: 'Bad Gateway'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

After:

$ youtube-dl https://soundcloud.com/mediasanctuary
[soundcloud:user] mediasanctuary: Downloading user info
[soundcloud:user] 164222112: Downloading track page 1
[soundcloud:user] 164222112: Downloading track page 2
[soundcloud:user] 164222112: Downloading track page 3
[soundcloud:user] 164222112: Downloading track page 4
[download] Downloading playlist: MediaSanctuary (All)
[soundcloud:user] playlist MediaSanctuary (All): Collected 4483 video ids (downloading 4483 of them)
[download] Downloading video 1 of 4483
...

edit: Reduced limit to 200 to match SoundCloud's documentation.

youtube_dl/extractor/soundcloud.py Outdated Show resolved Hide resolved
blackjack4494 added a commit to blackjack4494/youtube-dlc that referenced this pull request Sep 9, 2020
@remitamine
Copy link
Collaborator

Add a comment about the limit mentioned in the api documentation and prefix the commit message with [soundcloud].

…en listing a user's tracks.

Per the documentation here https://developers.soundcloud.com/blog/offset-pagination-deprecated the maximum limit is 200, so let's respect that (even if a higher value sometimes works).
@tfvlrue tfvlrue changed the title Reduce SoundCloud pagination limit to fix 502 Bad Gateway errors when listing a user's tracks. [soundcloud] Reduce pagination limit to fix 502 Bad Gateway errors when listing a user's tracks. Sep 12, 2020
@tfvlrue
Copy link
Contributor Author

tfvlrue commented Sep 12, 2020

All set.

@blackjack4494
Copy link

good luck @tfvlrue
I gave up that my PRs like this one #26265 will ever be merged. So I am holding back changes on my own fork until there is more activity on here.

@remitamine remitamine merged commit aa27253 into ytdl-org:master Sep 12, 2020
@tfvlrue tfvlrue deleted the soundcloud_limit branch September 12, 2020 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants