SoundCloud secret_token for private tracks is dropped #7872

i336 · 2015-12-15T06:07:35Z

Some SoundCloud URLs are private; these are in the form https://soundcloud.com/publisher/trackname/s-NWnsv. youtube-dl needs to grab the characters off the end of the URL and submit them as secret_token in all API requests.

This is what's happening instead:

We get:

header: Location: https://api.soundcloud.com/tracks/12345678.json?client_id=02gUJC0hH2ct1EGOcYXQIzRFU91c72Ea&secret_token=s-NWnsv

aaand then we go to

send: u'GET /tracks/12345678.json?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK&secret_token=s-NWnsv HTTP/1.1 ....'

...so far so good

send: u'GET /i1/tracks/12345678/streams?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK&secret_token=s-NWnsv HTTP/1.1 ...'

uh oh

send: u'GET /tracks/12345678/download?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK HTTP/1.1 ...'

no that's not going to work

[soundcloud] 12345678: download video format URL is invalid, skipping

You don't say?!

I'm all good though; my shell hack is currently trundling along file 5 of 10 at 100Mbps ^^

url='.......'; X="$(youtube-dl --print-traffic --dump-json "$url")"; tid="$(echo "$X" | grep -o '"display_id": "[^"]\+"' | sed 's/.*: "//;s/"$//')"; cid="$(echo "$X" | grep -o 'client_id=[A-Za-z0-9]\+' -m1)"; filename="$(echo "$X" | grep -o '"_filename": "[^"]\+"' | sed 's/.*: "//;s/mp3"$/wav/')"; wget -vvvv "https://api.soundcloud.com/tracks/$tid/download?$cid&secret_token=${url##*/}" -O "$filename"

dstftw · 2015-12-15T14:40:41Z

Post the full output of youtube-dl when run with --verbose option.

i336 · 2015-12-16T02:13:54Z

Oh... right, whoops. Of course.

$ youtube-dl --verbose https://soundcloud.com/(removed)/(removed)/s-NWnsv
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'https://soundcloud.com/(removed)/(removed)/s-NWnsv']
[debug] Encodings: locale ISO-8859-1, fs ISO-8859-1, out ISO-8859-1, pref ISO-8859-1
[debug] youtube-dl version 2015.12.10
[debug] Python version 2.7.10 - Linux-4.1.6-i686-Intel-R-_Pentium-R-_M_processor_1.86GHz-with-slackware-14.1
[debug] exe versions: ffmpeg 2.8, ffprobe 2.8
[debug] Proxy map: {}
[soundcloud] (removed)/(removed): Resolving id
[soundcloud] (removed)/(removed): Downloading info JSON
[soundcloud] (removed): Downloading track url
[soundcloud] (removed): Checking download video format URL
[soundcloud] (removed): download video format URL is invalid, skipping
[soundcloud] (removed): Checking http_mp3_128_url video format URL
[debug] Invoking downloader on u'https://ec-media.sndcdn.com/aGVsbG8gd28K.128.mp3?YWYxNWE5NGEwNDE4YTdlZjY5YjAzZDUyMmNkNmRmZWU5Mzk5ZWViOWE1MjIwNDljNmRmYmI2MzRjOWZiMmRjMjFlZDMzZTJjMjIxODEyZTYyMTVhYTFjZjNmZjNlMzM2MDc3NzgK'
[download] Destination: (removed).mp3
^C
ERROR: Interrupted by user

That's not really that informative though, as it hides the bug. Here's a transcript with --print-traffic, showing how the download video URL drops the secret_token (as previously documented).

$ youtube-dl --print-traffic --verbose https://soundcloud.com/(removed)/(removed)/s-NWnsv
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--print-traffic', u'--verbose', u'https://soundcloud.com/(removed)/(removed)/s-NWnsv']
[debug] Encodings: locale ISO-8859-1, fs ISO-8859-1, out ISO-8859-1, pref ISO-8859-1
[debug] youtube-dl version 2015.12.10
[debug] Python version 2.7.10 - Linux-4.1.6-i686-Intel-R-_Pentium-R-_M_processor_1.86GHz-with-slackware-14.1
[debug] exe versions: ffmpeg 2.8, ffprobe 2.8
[debug] Proxy map: {}
[soundcloud] (removed)/(removed): Resolving id
[soundcloud] (removed)/(removed): Downloading info JSON
send: u'GET /resolve.json?url=http://soundcloud.com/(removed)/(removed)/s-NWnsv&client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK HTTP/1.1\r\nHost: api.soundcloud.com\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 302 Found\r\n'
...
send: u'GET /tracks/12345678.json?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK&secret_token=s-NWnsv HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: api.soundcloud.com\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
...
header: Connection: close
[soundcloud] 123456789: Downloading track url
send: u'GET /i1/tracks/12345678/streams?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK&secret_token=s-NWnsv HTTP/1.1\r\nHost: api.soundcloud.com\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
...
[soundcloud] 12345678: Checking download video format URL
send: u'GET /tracks/12345678/download?client_id=aGVsbG8gd29ybGQgdGVzdGluZyAxMjMK HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: api.soundcloud.com\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
...
[soundcloud] 12345678: download video format URL is invalid, skipping
[soundcloud] 12345678: Checking http_mp3_128_url video format URL
send: u'GET /aGVsbG8gd28K.128.mp3?YWYxNWE5NGEwNDE4YTdlZjY5YjAzZDUyMmNkNmRmZWU5Mzk5ZWViOWE1MjIwNDljNmRmYmI2MzRjOWZiMmRjMjFlZDMzZTJjMjIxODEyZTYyMTVhYTFjZjNmZjNlMzM2MDc3NzgK HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: ec-media.sndcdn.com\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
...
[debug] Invoking downloader on u'https://ec-media.sndcdn.com/aGVsbG8gd28K.128.mp3?YWYxNWE5NGEwNDE4YTdlZjY5YjAzZDUyMmNkNmRmZWU5Mzk5ZWViOWE1MjIwNDljNmRmYmI2MzRjOWZiMmRjMjFlZDMzZTJjMjIxODEyZTYyMTVhYTFjZjNmZjNlMzM2MDc3NzgK'
send: u'GET /aGVsbG8gd28K.128.mp3?YWYxNWE5NGEwNDE4YTdlZjY5YjAzZDUyMmNkNmRmZWU5Mzk5ZWViOWE1MjIwNDljNmRmYmI2MzRjOWZiMmRjMjFlZDMzZTJjMjIxODEyZTYyMTVhYTFjZjNmZjNlMzM2MDc3NzgK HTTP/1.1\r\nAccept-Encoding: identity\r\nAccept-Language: en-us,en;q=0.5\r\nHost: ec-media.sndcdn.com\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
...
[download] Destination: (removed).mp3
^C
ERROR: Interrupted by user

dstftw · 2015-12-16T03:22:57Z

You must provide concrete example URL.

i336 · 2015-12-16T04:10:40Z

Unfortunately I don't know of one :(

On 12/16/15, Sergey M. notifications@github.com wrote:

You must provide concrete example URL.

Reply to this email directly or view it on GitHub:
#7872 (comment)

dstftw · 2015-12-16T14:16:40Z

youtube-dl works fine with private links at least with those we have in tests:

>py26yt https://soundcloud.com/jaimemf/youtube-dl-test-video-a-y-baw/s-8Pjrp
[soundcloud] jaimemf/youtube-dl-test-video-a-y-baw: Resolving id
[soundcloud] jaimemf/youtube-dl-test-video-a-y-baw: Downloading info JSON
[soundcloud] 123998367: Downloading track url
[soundcloud] 123998367: Checking fallback video format URL
[soundcloud] 123998367: fallback video format URL is invalid, skipping
[soundcloud] 123998367: Checking http_mp3_128_url video format URL
[download] Youtube - Dl Test Video '' A-123998367.mp3 has already been downloaded
[download] 100% of 155.10KiB

>py26yt https://api.soundcloud.com/tracks/123998367?secret_token=s-8Pjrp
[soundcloud] 123998367: Downloading info JSON
[soundcloud] 123998367: Downloading track url
[soundcloud] 123998367: Checking fallback video format URL
[soundcloud] 123998367: fallback video format URL is invalid, skipping
[soundcloud] 123998367: Checking http_mp3_128_url video format URL
[download] Youtube - Dl Test Video '' A-123998367.mp3 has already been downloaded
[download] 100% of 155.10KiB

There is nothing we can do until you provide an concrete example URL that reproduces the problem.

i336 · 2015-12-16T23:53:13Z

Wait! Wait!

_YOU ARE EXPERIENCING THE SAME ISSUE WITH YOUR EXAMPLES!_

Look!!!

[soundcloud] 123998367: fallback video format URL is invalid, skipping

Please alter the code that fetches the download URL to include the secret_token as I have previously described. Just try it.

In my case, it was the difference between getting 128k MP3 and lossless WAV. :P

dstftw closed this Dec 16, 2015

ytdl-org / youtube-dl

SoundCloud secret_token for private tracks is dropped #7872

SoundCloud secret_token for private tracks is dropped #7872

i336 commented Dec 15, 2015

dstftw commented Dec 15, 2015

i336 commented Dec 16, 2015

dstftw commented Dec 16, 2015

i336 commented Dec 16, 2015

dstftw commented Dec 16, 2015

i336 commented Dec 16, 2015

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

SoundCloud secret_token for private tracks is dropped #7872

SoundCloud secret_token for private tracks is dropped #7872

Comments

i336 commented Dec 15, 2015

dstftw commented Dec 15, 2015

i336 commented Dec 16, 2015

dstftw commented Dec 16, 2015

i336 commented Dec 16, 2015

dstftw commented Dec 16, 2015

i336 commented Dec 16, 2015

Essential cookies

Always active

Analytics cookies