Investigation on command acknowledgment request

[devgianlu]

librespot-java receives commands from a Websocket endpoint called "dealer". These commands tell it what to do (load a playlist, pause/resume, change volume and more), but since 1.1.14 the client must send back "something" to let the server know that it has received the data correctly (I think). Not sending that ack results in receiving the command three times which is a bit annoying for next or previous actions, for example. I found some proto definitions inside the executable, that relate to my idea:

enum SendCommandResult {
	UNKNOWN_SEND_COMMAND_RESULT = 0;
	SUCCESS = 1;
	DEVICE_NOT_FOUND = 2;
	CONTEXT_PLAYER_ERROR = 3;
	DEVICE_DISAPPEARED = 4;
	UPSTREAM_ERROR = 5;
	DEVICE_DOES_NOT_SUPPORT_COMMAND = 6;
	RATE_LIMITED = 7;
}

message PostCommandResponse {
	optional string ack_id = 1;
}

There's also an options called command_acks in the device information (unluckily toggling it doesn't do anything):

message Capabilities {
	optional bool can_be_player = 2;
	optional bool restrict_to_local = 3;
	optional bool gaia_eq_connect_id = 5;
	optional bool supports_logout = 6;
	optional bool is_observable = 7;
	optional int32 volume_steps = 8;
	repeated string supported_types = 9;
	optional bool command_acks = 10;
	optional bool supports_rename = 11;
	optional bool hidden = 12;
	optional bool disable_volume = 13;
	optional bool connect_disabled = 14;
	optional bool supports_playlist_v2 = 15;
	optional bool is_controllable = 16;
	optional bool supports_external_episodes = 17;
	optional bool supports_set_backend_metadata = 18;
	optional bool supports_transfer_command = 19;
	optional bool supports_command_request = 20;
	optional bool is_voice_enabled = 21;
	optional bool needs_full_player_state = 22;
	optional bool supports_gzip_pushes = 23;
}

At this point, we need to find where the request should be sent which has been the major issue in a couple of weeks. Spotify seems to be working very hard on securing the desktop API: it uses TLS, certificate pinning and a Diffie-Hellman cipher.

There are two places where to search: the HTTPS API and the Hermes one. I've already inspected the second one and there's not a single call which may correlate to what I've said above.

The HTTPS is much harder to inspect, some calls are made with HTTP2 and can be decrypted with the SSLKEYLOGFILE technique, but others can't (I did not find any evidence in those ones that are being decrypted by Wireshark). I am now running out of ideas, but #105 won't be complete until this issue has been solved. The "dealer" is suspected to be used as the ack endpoint as a packet is sent right after receiving the command. ATM no technique can be used to intercept that traffic.

I am mainly asking for help, if someone knows anything more than me about TLS and how to decrypt or even how to patch the executable. Thank you for your help, ask below if there's anything unclear. ANY help is appreciated.

MITM ideas

The SSLKillSwitch tool used to intercept the HTTPS traffic on Mac doesn't work with WebSocket probably because the certificate verification doesn't use the OpenSSL api.

Patching ideas

The client uses the LWS library (https://libwebsockets.org/) on all clients. The library is statically linked inside the executable: patching is noticeably more difficult.

By starting Spotify with /Applications/Spotify.app/Contents/MacOS/Spotify --show-console, we get a pretty useful output which includes some "LWS got ..." corresponding to each LWS callback being called. We can confirm that the client sends something when receiving a command (LWS_CALLBACK_CLIENT_WRITEABLE appears, indicating data being written). The client apparently loads its own certificates as LWS_CALLBACK_OPENSSL_LOAD_EXTRA_CLIENT_VERIFY_CERTS is being called too.

For most users it's enough to pass the SSL certificate and key information by
giving filepaths to the info.ssl_cert_filepath and info.ssl_private_key_filepath
members when creating the vhost.

If you want to control that from your own code instead, you can do so by leaving
the related info members NULL, and setting the info.options flag
LWS_SERVER_OPTION_CREATE_VHOST_SSL_CTX at vhost creation time. That will create
the vhost SSL_CTX without any certificate, and allow you to use the callback
LWS_CALLBACK_OPENSSL_LOAD_EXTRA_SERVER_VERIFY_CERTS to add your certificate to
the SSL_CTX directly. The vhost SSL_CTX * is in the user parameter in that
callback.

[devgianlu]

The issue disappeared since Spotify 1.1.17. Thanks @crsmoro

Stopped working already.

[devgianlu]

@devgianlu did you get the dealer stuff working with the {"type":"ping"} requests at 30 second intervals? I had a look at #140 and the traffic that i intercept, there is nothing that looks like an ack_id for play.spotify.com wss service, and I see no dealer traffic from the desktop client, so not really sure where the dealer stuff might be coming from?

@sashahilton00 I have the ping/pong mechanism already figured out and also the messages/requests. I can assure you that the desktop client has a dealer connection because of the DNS requests (and further traffic) and the presence of related strings in the binary. The client simply refuses to connect the dealer via the proxy and bypasses it: I had the idea to place an "hardware proxy", but then the requests will be encrypted.

The last idea that came to my head is to patch the executable just like it has been done for mercury requests: they don't seem to use a library and that makes everything more difficult.

I also tried investigating through the help of an old Android device, but the device is kind of broken and I can't use it (could be a valid entry point).

[sashahilton00]

Hmmm, interesting. Will investigate further then. It sounds like patching and a nic proxy are the only options then. Just a thought, spin up macOS in VMWare and run Spotify in that (with the charles proxy cert installed and certificate pinning disabled, but without a system proxy), then use vmware to proxy all traffic from the vm through charles on the host machine. simulates a hardware proxy but without the messing around with extra hardware.

[devgianlu]

You can't do that because (on Windows) the VM connects directly to the network via an adapter.

[sashahilton00]

Ah right. VMWare on mac gives the option of a shared mode where all the traffic is run through the host os network stack. Under what situation does the client connect to dealer.spotify.com? is it only when using spotify connect or something?

[devgianlu]

Under what situation does the client connect to dealer.spotify.com? is it only when using spotify connect or something?

It connects upon opening the client.

[sashahilton00]

and you're sure it's not just a DNS lookup then nothing else? I've just captured using wireshark on the mac version the client startup, and there's nothing going to dealer.spotify.com or dealer-weu.spotify.com. just saw something going to global-dealer-ssl.spotify.com, investigating.

[devgianlu]

Filtering DNS requests:

Then filtering for IP 35.186.224.47:

[sashahilton00]

Ok can confirm the same behaviour on mac as well. weird. looks like it's going to be a case of an upstream proxy to get this working, will try the VM approach a bit later and see if we have any luck with that. I note that the dealer traffic appears not to exist on the iOS version of Spotify (at least when tested via Internet Sharing).

[sashahilton00]

No luck on the VM approach. I did manage to get the client to attempt to connect whilst running MITM proxy by running it in transparent mode with a few tweaks, but more bad news in that it didn't accept the mitmproxy certificate, whereas the other domains did, hence there's probably more patching needed to get the dealer traffic.

[devgianlu]

I note that the dealer traffic appears not to exist on the iOS version of Spotify (at least when tested via Internet Sharing).

Same on Android.

there's probably more patching needed to get the dealer traffic.

Yeah, that's what I am afraid of. It clearly uses another API of some sort.

At one point I thought that we could check every string that is created by the application and the message would be somewhere in there. Don't quite know how to achieve such result.

[devgianlu]

I've found a workaround. Just pretend that librespot-java is an Android device. (updated main post with more details).

[l3d00m]

It's still happening for me with 3088373, each command is received three times

[devgianlu]

@l3d00m Strange, I haven't seen it once since the workaround. Which client where you using? Version? Still happening.

[l3d00m]

I've built it myself on top of 3088373 and ran it with java -jar ./core/target/librespot-core-jar-with-dependencies.jar. I'm not really experienced with this tool yet, so take it with a grain of salt. It logs the following: Edit: removed the log because you already acknowledged it

[devgianlu]

It clearly uses another API of some sort.

@sashahilton00 Not true. They are using LWS (https://libwebsockets.org/) on Windows and Mac. This should be patchable.

Example client and Logging functions, will work on that tomorrow.

[sashahilton00]

It clearly uses another API of some sort.

@sashahilton00 Not true. They are using LWS (https://libwebsockets.org/) on Windows and Mac. This should be patchable.

Not sure which other API you're referring to, looks like a normal API to me. will look into it, at this point though I think we're definitely into patch territory :(

[devgianlu]

It clearly uses another API of some sort.

@sashahilton00 Not true. They are using LWS (https://libwebsockets.org/) on Windows and Mac. This should be patchable.

Not sure which other API you're referring to, looks like a normal API to me. will look into it, at this point though I think we're definitely into patch territory :(

I meant library, not API, sorry. At this point I think patching is more reliable when working.

[sashahilton00]

Have just done a bit of investigating, it looks like the reason behind the ssl certificate not being accepted is down to the fact that chromium embedded doesn't use the system keystore at all, instead they bundle all the root ca certificates with the framework. I think that we might be able to get around that by just creating a new root ca with the same parameters, then just swap it in for one of the existing ones.

[devgianlu]

I am not entirely sure how the chromium platform works and how to replace the certificates, but if you could create a script that autonomously patches the binary, that would be awesome.

[sashahilton00]

Will look into it. Am busy for the next few days though. CEF (chromium embedded framework) is what Spotify desktop uses; the app is basically a series of browser windows. CEF is just the chromium engine packaged up in library format. You can see the library in Spotify > Contents > Frameworks on macOS. A quick analysis of the binary reveals that all the root certificates are included in the CEF library, so what I’ll try and do is create a new root CA and swap it in, then mitm. Will let you know how it goes

…

-- Sent from Canary (https://canarymail.io)

On Monday, Oct 21, 2019 at 2:44 pm, Gianlu ***@***.*** ***@***.***)> wrote: I am not entirely sure how the chromium platform works and how to replace the certificates, but if you could create a script that autonomously patches the binary, that would be awesome. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub (#140?email_source=notifications&email_token=AA752EV7FTVH74ZR4HWDX33QPWP2TA5CNFSM4I2ER6BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB2F5PI#issuecomment-544497341), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AA752ETT4XWFEOJEKCW6DHTQPWP2TANCNFSM4I2ER6BA).

[devgianlu]

I really hope that your approach works because hooking LWS is more difficult than I thought since the library is statically linked and therefore the method names haven't been kept.

There are some strings that reveal some debugging features that aren't enabled by default, but I have no idea how to enable it.

[devgianlu]

I've run binwalk against it and have found quite a few certificates:

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Microsoft executable, portable (PE)
14832712      0xE25448        HTML document header
14832802      0xE254A2        HTML document footer
14841408      0xE27640        Unix path: /offline/v1/progress/total
14844392      0xE281E8        Unix path: /cpu-monitor/unstable/config/enable
14855420      0xE2ACFC        Unix path: /desktop/v1/upgrade/status
14855900      0xE2AEDC        Certificate in DER format (x509 v3), header length: 4, sequence length: 24064
14862296      0xE2C7D8        HTML document footer
14862352      0xE2C810        HTML document header
14866303      0xE2D77F        Unix path: /www.facebook.com/v2.3/dialog/oauth?client_id=174829003346&response_type=token&display=popup&redirect_uri=
14874540      0xE2F7AC        Unix path: /feedback/v1/feedback/like
14875914      0xE2FD0A        eCos RTOS string reference: "eCosmosBuilder"
14891578      0xE33A3A        eCos RTOS string reference: "eCosmosBuilder"
14892064      0xE33C20        Unix path: /v2/experimental/loop/start
14928080      0xE3C8D0        Unix path: /v2/settings/state/ad_state_endpoint
14937060      0xE3EBE4        Unix path: /player/v2/main/skip_next
14962771      0xE45053        Unix path: /list/tracks/all/play
14993684      0xE4C914        Copyright string: "copyrights"
14995851      0xE4D18B        Unix path: /spclient.wg.spotify.com/frecency/v1/play-contexts
15010188      0xE5098C        Unix path: /feedback/v1/feedback/dislike?uri=
15022364      0xE5391C        Copyright string: "copyrights"
15028079      0xE54F6F        Unix path: /spclient.wg.spotify.com/extended-metadata/v0/extended-metadata
15069008      0xE5EF50        Copyright string: "copyrights"
15089248      0xE63E60        Unix path: /playlistextender/ft/v1/extend-playlist
15145004      0xE7182C        Copyright string: "Copyright (C) 2016, Thomas G. Lane, Guido Vollbeding"
15162076      0xE75ADC        Unix path: /speakeasy/v2/natural-language/action
15165892      0xE769C4        Unix path: /connect-state/v1/player/command
15191168      0xE7CC80        Unix path: /tv-token-exchange/v1/token/code?
15196543      0xE7E17F        Unix path: /spclient.wg.spotify.com/device-auth/v1/refresh
15203200      0xE7FB80        Certificate in DER format (x509 v3), header length: 4, sequence length: 943
15209416      0xE813C8        Base64 standard index table
15214280      0xE826C8        Base64 standard index table
15217280      0xE83280        HTML document header
15217447      0xE83327        HTML document footer
15220264      0xE83E28        PEM RSA private key
15220328      0xE83E68        PEM EC private key
15221756      0xE843FC        PEM certificate
15251936      0xE8B9E0        SHA256 hash constants, little endian
15265472      0xE8EEC0        Base64 standard index table
15268304      0xE8F9D0        PEM certificate
15269610      0xE8FEEA        PEM certificate
15270916      0xE90404        PEM certificate
15271848      0xE907A8        PEM certificate
15272736      0xE90B20        PEM EC private key
15273128      0xE90CA8        PEM certificate
15273944      0xE90FD8        PEM EC private key
15274184      0xE910C8        PEM certificate
15275024      0xE91410        PEM EC private key
15275264      0xE91500        PEM certificate
15276576      0xE91A20        PEM RSA private key
15278384      0xE92130        PEM certificate
15279584      0xE925E0        PEM RSA private key
15281296      0xE92C90        PEM certificate
15282608      0xE931B0        PEM RSA private key
15284320      0xE93860        PEM certificate
15285632      0xE93D80        PEM certificate
15301584      0xE97BD0        Unix path: /product-state/v1/overrides/cappedondemand
15310527      0xE99EBF        Unix path: /login5/v2/challenges/code.proto
15311331      0xE9A1E3        Unix path: /login5/v2/challenges/hashcash.proto
15312375      0xE9A5F7        Unix path: /login5/v2/credentials/credentials.proto
15313047      0xE9A897        Unix path: /login5/v2/identifiers/identifiers.proto
15317383      0xE9B987        Unix path: /login5/v3/challenges/code.proto
15318187      0xE9BCAB        Unix path: /login5/v3/challenges/hashcash.proto
15319367      0xE9C147        Unix path: /login5/v3/credentials/credentials.proto
15320167      0xE9C467        Unix path: /login5/v3/identifiers/identifiers.proto
15680072      0xEF4248        Base64 standard index table
15722928      0xEFE9B0        SQLite 3.x database,, user version 626982913
15765762      0xF09102        Unix path: /../sandbox/win/src/broker_services.cc
15768659      0xF09C53        Unix path: /../sandbox/win/src/handle_closer_agent.cc
15772646      0xF0ABE6        Unix path: /../sandbox/win/src/interception.cc
15774153      0xF0B1C9        Unix path: /../base/threading/thread_local_storage.cc
15774579      0xF0B373        Unix path: /../base/metrics/statistics_recorder.cc
15782484      0xF0D254        CRC32 polynomial table, little endian
15783708      0xF0D71C        Unix path: /../base/threading/thread_task_runner_handle.cc
15797059      0xF10B43        Unix path: /workers/contexts/detached_context/isolate_0x?
15897112      0xF29218        CRC32 polynomial table, little endian
15901208      0xF2A218        CRC32 polynomial table, big endian
15905320      0xF2B228        Copyright string: "Copyright 1995-2017 Jean-loup Gailly and Mark Adler "
15905664      0xF2B380        Copyright string: "Copyright 1995-2017 Mark Adler "
15912194      0xF2CD02        Unix path: /www.w3.org/XML/1998/namespace
15912926      0xF2CFDE        Unix path: /www.w3.org/XML/1998/namespace
15922040      0xF2F378        SHA256 hash constants, little endian
16998336      0x1035FC0       Base64 standard index table
16998768      0x1036170       Copyright string: "Copyright (c) by P.J. Plauger, licensed by Dinkumware, Ltd. ALL RIGHTS RESERVED."
17803616      0x10FA960       PNG image, 256 x 256, 8-bit/color RGBA, non-interlaced
17874312      0x110BD88       XML document, version: "1.0"
17875414      0x110C1D6       Unix path: /schemas.microsoft.com/SMI/2005/WindowsSettings">true</dpiAware>
17875513      0x110C239       Unix path: /schemas.microsoft.com/SMI/2016/WindowsSettings">system</dpiAwareness>
21335709      0x1458E9D       Certificate in DER format (x509 v3), header length: 4, sequence length: 1353
21337066      0x14593EA       Certificate in DER format (x509 v3), header length: 4, sequence length: 1328
21338398      0x145991E       Certificate in DER format (x509 v3), header length: 4, sequence length: 951
21339353      0x1459CD9       Certificate in DER format (x509 v3), header length: 4, sequence length: 1328
21340685      0x145A20D       Certificate in DER format (x509 v3), header length: 4, sequence length: 1044
21341733      0x145A625       Certificate in DER format (x509 v3), header length: 4, sequence length: 1183

[sashahilton00]

Have tried a quick swap of the MITM proxy certificate into the framework, unfortunately no luck there. It looks like the client is pinning against a specific certificate issuer as opposed to anything in the CEF trust store. The next step in this is probably to just nuke all of the certificate pinning logic. There will be a function somewhere in Chromium that is basically an equivalence check that determines whether a certificate is valid or not, easiest approach would be to method swizzle that and have it always return true. Tried it with the usual suspects for macOS (SSLCreateContext, SSLHandshake, etc.) but no luck there. I don't have much time on my hands atm, so can't go digging through the Chromium source, but if someone else wants to then I'd imagine that would allow interception of the dealer.spotify.com requests. If someone finds the function, I can look into writing a patch if that's out of their comfort zone.

[devgianlu]

I have reverse engineered a bit the LWS library and can reliably intercept the so called callbacks, but I am yet to figure out how where the outgoing messages are in memory (due to lack of experience in assembly). I am afraid that openssl is statically linked inside the binary and therefore cannot intercept the calls to the most vulnerable functions.

As Spotify loads it's own certificates when setting up (updated the main post), we could exploit SSL_CTX_get_cert_store which is needed to add its certificates, but I have no idea where to go from here.

Currently my best bet is to use https://github.com/arvinddoraiswamy/slid.

[sashahilton00]

Just had a look at the LWS code, OpenSSL_client_verify_callback looks to be the function I was talking about. It just returns a 1 or 0 depending on whether the certificate is valid or not. Should be possible to patch that to return 1 regardless. Still not sure if it'll be enough though, depends on whether LWS is the only part that connects to dealer.spotify.com

[sashahilton00]

@devgianlu which callbacks are you referring to? have just tried patching out the pinning checks, though i'm not sure if they're using LWS on macOS?

[devgianlu]

@devgianlu which callbacks are you referring to? have just tried patching out the pinning checks, though i'm not sure if they're using LWS on macOS?

It is, but you can't patch the functions directly because the library is statically linked. If you start it with --show-console you'll see all the LWS callbacks that I am talking about.

[sashahilton00]

That's a neat trick. I missed the part where you mentioned it is statically linked, though you might have some luck patching using https://github.com/rentzsch/mach_override. Haven't tried with that yet, but it's normally the most effective as it patches the function in memory as opposed to rewriting the symbols table.

The way LWS implemented events is somewhat irritating, as it's 1 callback function that handles all events, so LWS_CALLBACK_OPENSSL_LOAD_EXTRA_CLIENT_VERIFY_CERTS is just one of the cases. Means we have to modify the original function as opposed to just overwriting it.

[devgianlu]

The command ack string is as following:

{"key":"5e5f17ad-5de4-45e0-b699-3c53cf11d9cc","payload":{"success":true},"type":"reply"}

Where key is provided by the request.

P.S. I've noticed another message being sent, not sure what's about:

{"connect_attempt_count":0,"conn_id_latency_us":2335830}

[sashahilton00]

@devgianlu how did you mitm it in the end? Might be useful to know for future work

[devgianlu]

No MITM, I had to debug step by step and look at the assembly to figure where the lws_write function was. I can reproduce this if needed in the future, but I don't think a patch can be written for this.

[sashahilton00]

Hmm, unfortunate. Let’s hope we don’t need to deal with more of this nonsense anytime soon.