soundcloud HTTP Error 403: Forbidden

[hedii]

Hi, I get a 403 error when i try to download this soundcloud track: https://soundcloud.com/roywoodsofficial/drama-feat-drake

youtube-dl --verbose  https://soundcloud.com/roywoodsofficial/drama-feat-drake
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'https://soundcloud.com/roywoodsofficial/drama-feat-drake']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.11.02
[debug] Python version 2.7.6 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: ffmpeg 2.8.1-static, ffprobe 2.8.1-static
[debug] Proxy map: {}
[soundcloud] roywoodsofficial/drama-feat-drake: Resolving id
[soundcloud] roywoodsofficial/drama-feat-drake: Downloading info JSON
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 329, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1873, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[hedii]

maybe i need to update python to 3.x?
which python version is recommended?

[hedii]

ok, my ip is banned, i close

[hedii]

I re open my old issue, because i have investigated a little more, on several machine/servers.
I don't know what are soundcloud way of rejecting a connexion, but i tested with plenty of servers, ips and computers. If the source machine has never connected to soundcloud before (eg: visit soundcloud website with a browser), soundcloud always return 403 forbiden, even with if accessed with curl, not just youtube-dl.

I tested with 4 servers, and dozens of ips in each server.

When i test with ips that were used with youtube-dl for more than a year, no problem. With new ips, i get 403. Soundcloud should have made this restriction recently, and the old ips do not have to proove that they belong to a regular user (maybe i am fantasizing?...).

I know it is not so related to youtube-dl, and that this issue tracker is not a forum, but there could be a workaround to avoid these 403. This problem prevent youtube-dl with soundcloud to work on new machines/servers that never accessed soundcloud.com with a browser.
Anybody has experienced that before? anybody has a trick to avoid that?

youtube-dl --dump-json --verbose https://soundcloud.com/kevingates/really-really
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--dump-json', u'--verbose', u'https://soundcloud.com/kevingates/really-really']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.02.01
[debug] Python version 2.7.6 - Linux-3.14.32-xxxx-grs-ipv6-64-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: ffmpeg N-52512-g22ee0a5-static, ffprobe N-52512-g22ee0a5-static
[debug] Proxy map: {}
ERROR: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 352, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1903, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

curl https://soundcloud.com/kevingates/really-really
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <title>403 - Forbidden</title>
    </head>
    <body>
        <h1>403 - Forbidden</h1>
    </body>
</html>

[hedii]

i messed with closing/reopening the issue, i though it would be back on top of opened ones.
@dstftw @phihag @yan12125 @jaimeMF do you have any idea?
sorry to ping you...

[yan12125]

If you get a 403 error with curl, there's nothing much youtube-dl can do. Modern browsers, including recent Firefox and Chrome, provide the functionality to generate a curl command to mimic the behavior of an HTTP(S) request. A possible approach is copying the cURL command from a working machine and running it on the problematic machine. If you've found some header critical for the first time accessing, please propose it here. The -v flag of curl can be useful for debugging purposes. For example:

$ curl -v https://soundcloud.com/kevingates/really-really
*   Trying 68.232.44.127...
* Connected to soundcloud.com (68.232.44.127) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*    subject: OU=Domain Control Validated; CN=*.soundcloud.com
*    start date: Apr 22 16:52:12 2014 GMT
*    expire date: Apr  8 10:08:48 2016 GMT
*    subjectAltName: soundcloud.com matched
*    issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Domain Validation CA - SHA256 - G2
*    SSL certificate verify ok.
> GET /kevingates/really-really HTTP/1.1
> Host: soundcloud.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Content-Type: text/html
< Date: Tue, 02 Feb 2016 18:51:37 GMT
< Server: am/2
< Set-Cookie: sc_anonymous_id=31379-454414-575424-74182; path=/; expires=Fri, 30 Jan 2026 18:51:36 GMT; domain=.soundcloud.com
< Via: sssr
< X-Frame-Options: SAMEORIGIN
< Content-Length: 30985
< 
<!DOCTYPE html>

<html lang="en">
<head>
  <meta charset="utf-8">

  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
(...remaining HTML contents omitted...)

As you've said, it's not a problem of youtube-dl. I'm closing this issue now, leaving it for discussion only.

[tanduong]

@hedii Did you manage to get over the issue? I got 403 error with Soundcloud now. I understand that this is not youtube-dl's problem. But I didn't find any dicussions about SoundCloud API issue out there.

[yan12125]

@tanduong Works for me. Please make sure that you're using the latest version (2016.08.06). If the problem is still, paste the output of the following command:

$ youtube-dl -v "https://soundcloud.com/roywoodsofficial/drama-feat-drake" 

[hedii]

@tanduong the only solution i found is changing the server where youtube-dl is running or changing its ip addresses.