Internal Server Error 500 on Linux VPS but OK elsewhere

[avniir]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.10.05. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.10.05

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl -g --restrict-filenames -v https://soundcloud.com/3lau/rosesandmoney

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-g', u'--restrict-filenames', u'-v', u'https://soundcloud.com/3lau/rosesandmoney']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2018.10.05
[debug] Python version 2.7.12 (CPython) - Linux-2.6.32-042stab112.15-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.14-0ubuntu0.16.04.1, ffprobe 2.8.14-0ubuntu0.16.04.1, rtmpdump 2.4
[debug] Proxy map: {}

ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 604, in _request_webpage
    return self._downloader.urlopen(url_or_request)

  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 2211, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)

  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)

  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)

  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)

  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)

  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

---

Description of your issue, suggested solution and other information

Youtube-dl (and in general Curl) can't access soundcloud.com, it returns the Internal Server Error code 500. So the JSON info can't be downloaded by ytdl. All this happens only on my Virtual Private Server (VPS). All works like a charm on Windows or a virtual Ubuntu machine on VirtualBox.

What could be the solution? I've tried putting a proxy with --proxy, doesn't help.
Maybe I need to add some custom headers? If yes, what needs to be the content of the headers?
Need your help. Thanks!

[dstftw]

Most likely your VPS' IP/IP range is banned by soundcloud. Can you watch it in browser from VPS?

[avniir]

Tried command w3m www.soundcloud.com and yep.. My VPS seems to be banned.
Question is, if I buy a new VPS, how to avoid being banned once again by SC? Maybe I need to parse HTTP headers with my commands to SC?

[rautamiekka]

VPS companies have a static IP range given to them, so unless you can buy an outside IP which ain't banned by them or be able to convince them to unban you it won't matter where you buy your VPS from. But since the IPv4 range has been allocated down to the last number, that might not be possible.

[avniir]

Ouch.. Alright, didn't know about that, thanks for the info.
I'll try using a VPN on my VPS to bypass that