New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_stream_url gives 403 with 'DEVICE_NOT_AUTHORIZED' for Mobileclient.FROM_MAC_ADDRESS #590

Open
stegzilla opened this Issue Dec 19, 2017 · 52 comments

Comments

Projects
None yet
@stegzilla

stegzilla commented Dec 19, 2017

Been using everything fine until a few hours ago, when any call to get_stream_url returns 403 with 'X-Rejected-Reason': 'DEVICE_NOT_AUTHORIZED' when using Mobileclient.FROM_MAC_ADDRESS

I'm able to do everything but get_stream_url. Using an actual Android ID seems to work, so Google may have changed something to stop us using MAC addresses? Makes no difference if 2FA is enabled or not.

This is probably a dupe of #584, sorry.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented Dec 19, 2017

Huh, that's the first I've heard of them actually introspecting device ids. Sounds like we may need to deprecate FROM_MAC_ADDRESS and make it easier for people to look up their existing device ids before logging in.

@stegzilla

This comment has been minimized.

stegzilla commented Dec 19, 2017

I just tried it again, and FROM_MAC_ADDRESS seems to be working now, so I don't really know what's happening.

I guess deprecating is probably a good idea, no idea when it will just stop working completely.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented Dec 19, 2017

Weird. Could be that they're in the process of rolling the new change out (though in the past I've usually seen that around Fridays, not the start of a week).

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented Dec 19, 2017

Or could just be Google servers being flaky as they have been so many times in the past : P

@fergyfresh

This comment has been minimized.

fergyfresh commented May 25, 2018

I just got a similar problem to this.

stack trace is basically this

[DEBUG]	2018-05-25T00:23:41.272Z	e02cc2ce-5fb1-11e8-a4dc-637d90e1a294	https://mclients.googleapis.com:443 "GET /music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa HTTP/1.1" 403 None
[2018-05-25 00:23:41,274] ERROR in app: Exception on /alexa/stream/Tbm4qc5ka36wvdcifyfryjalqx4 [GET]
Traceback (most recent call last):
File "/var/task/gmusicapi/protocol/shared.py", line 218, in perform
response.raise_for_status()
File "/var/task/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/task/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/var/task/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/task/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/task/flask/_compat.py", line 33, in reraise
raise value
File "/var/task/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/var/task/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/task/geemusic/controllers.py", line 40, in redirect_to_stream
stream_url = api.get_google_stream_url(song_id)
File "/var/task/geemusic/utils/music.py", line 148, in get_google_stream_url
return self._api.get_stream_url(song_id)
File "<decorator-gen-102>", line 2, in get_stream_url
File "/var/task/gmusicapi/utils/utils.py", line 293, in wrapper
return function(*args, **kw)
File "/var/task/gmusicapi/clients/mobileclient.py", line 388, in get_stream_url
return self._make_call(mobileclient.GetStreamUrl, song_id, device_id, quality)
File "/var/task/gmusicapi/clients/shared.py", line 84, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "/var/task/gmusicapi/protocol/shared.py", line 226, in perform
raise CallFailure(err_msg, call_name)
gmusicapi.exceptions.CallFailure: GetStreamUrl: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa

So I work on an Alexa app that is built off of this, which is awesome by the way and suddenly my deployment stopped working. So I have a serverless setup that depends on this that has been working for about two months now and it randomly stopped working today. It gets the song info but fails to stream the song. The only thing I can think of is that I'm using S3 to redirect (I've used it for almost 200 songs at this point.) the stream through because I'm using a serverless setup (ApiGateway and Lambda) cause its just so damn cheap.

Long story short, does anything glaringly wrong jump out in my stack trace? Like I said I haven't changed the code at all, it seems Google is blocking me, thats my only idea. Maybe I should try rotating the app password every so often? Maybe this is their way of telling me? I'm not sure.

@fergyfresh fergyfresh referenced this issue May 25, 2018

Merged

Zappa deploy #214

@fergyfresh

This comment has been minimized.

fergyfresh commented May 25, 2018

In order to use ApiGateway + Lambda with this skill we need to manually login to the api in a python shell from a computer with a good mac address and then use Mobileclient.get_registered_devices() to get the list of valid devices and then use one of those device ids hardcoded as the device id.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented May 25, 2018

Ah, yeah. I think geemusic defaults to from_mac_address, which doesn't work well in paas setups: stevenleeg/geemusic#183 (comment).

@fergyfresh

This comment has been minimized.

fergyfresh commented May 25, 2018

Any idea why it worked for about a month and then they finally cut me off? I did notice the stream taking longer and longer to play even from the native app.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented May 25, 2018

Hard to say. Could have been that you were creating lots of new devices without knowing, Google blacklisted the lambda ips, aws switched to new ip range that Google already had blocked, etc.

@fergyfresh

This comment has been minimized.

fergyfresh commented May 25, 2018

True. I just didn't know how much you knew about the inner workings of this stuff. It appeared to only block me because I wasn't using a device id, or does gmusicapi randomly generate one if you don't supply one. The reason why I'm asking is that it logged in on the PAAS, but couldn't download from get_stream_url.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented May 25, 2018

gmusicapi can generate one from the machine's mac address if requested, though it's not recommended. geemusic defaults to this behavior.

@fergyfresh

This comment has been minimized.

fergyfresh commented May 25, 2018

Ah ok, and since Lambda hits a different machine every time which has a different MAC Address, but has the same ARN, Google blocks it because I'm coming from the same address but with multiple different device ids.

I now see what you were saying about the creating lots of new devices.

@GallantJR

This comment has been minimized.

GallantJR commented May 25, 2018

I was using the same MAC address for each call for a month now... Today my main MAC address that i pass trough the API as an hardcoded string isn't working ! So i don't think the only issue is about using multiple Mac addresses.

@fizzybunk

This comment has been minimized.

fizzybunk commented May 26, 2018

I have also been using the same MAC address for each call for well over a month, something was changed on Googles end. I even have tried using a valid android device_id that is registered with the account, seems the only way I am able to get Mobileclient.get_stream_url() to work is being logged into the account on the mobile app with the device of the device_id at the same time. Otherwise it is throwing a 403 error. I am able to reproduce this with 2 different accounts using 2 different devices and device_id's on 2 completely separate networks; so it is not blocking or anything strange on Googles end other than the changes they made this last week.

@fizzybunk

This comment has been minimized.

fizzybunk commented May 27, 2018

After some testing I have found out that using the MAC address no longer will work. You need a valid android device_id that has been linked (authorized) to the account. You also can no longer use your regular login/password, you must use an app password after setting up 2FA. Once I met these two requirements everything started working again.

@dnikkel

This comment has been minimized.

dnikkel commented May 27, 2018

@fizzybunk I'm having this problem on my Raspberry Pi. Do you know what I need to do for a valid device_id?

@dnikkel

This comment has been minimized.

dnikkel commented May 27, 2018

@fizzybunk Actually, I tried plugging in one authorized Android device ID's and I'm back to streaming GMusic!

@fergyfresh

This comment has been minimized.

fergyfresh commented May 27, 2018

@butty2017

This comment has been minimized.

butty2017 commented May 27, 2018

@fergyfresh

Any idea why it worked for about a month and then they finally cut me off?

I think this problem has something to do with Google Play Music's integration to YouTube Music. Because I had the same problem just after the change (night May 22 UTC).

@simon-weber

This comment has been minimized.

Owner

simon-weber commented May 30, 2018

My CI broke too. I'll try to make some time for this over the weekend.

@fergyfresh

This comment has been minimized.

fergyfresh commented May 30, 2018

I'll try to help out too. I think I have a workaround, if you use FROM_MAC_ADDRESS to log in and get the valid device IDs and then logout and re-log with the valid device id it works. I know its a run around, but that's the quickest solution I could think of off the top of my head. Do you have any idea what way you'd go?

@simon-weber

This comment has been minimized.

Owner

simon-weber commented May 30, 2018

If it's just the format of the id that they care about, we might be able to fix the mac address generation. Otherwise, we can just make it easier to pick an existing id (I don't think one is needed during login, so we can pick it afterwards).

@fergyfresh

This comment has been minimized.

fergyfresh commented May 30, 2018

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented May 30, 2018

Just an FYI: when I was poking at this in the IRC channel with someone, I noticed that my Pixel 2 XL's associated ID wasn't its Android Device ID, but rather its Google Service Framework ID. Seems that it is the same kinda format, digits and uppercase letters. But they are different.

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented May 30, 2018

If it's just the format of the id that they care about, we might be able to fix the mac address generation.

From my testing, this is not the case : (

Appears to require the ID of a linked Android device.

@simon-weber

This comment has been minimized.

Owner

simon-weber commented Jun 3, 2018

An interesting finding: I can get stream urls with a device id that's been registered to another account. There were a few hangups, though:

  • on my first try, I got a NotLoggedIn when calling get_all_songs. I also got a security alert to my recovery account notifying me of a login from an unrecognized device.
  • on my second try with the same args, I got an InvalidDeviceId since the device didn't show as registered.
  • after disabling that check, I was able to get a stream url. Interestingly, the device still doesn't show in my registered list afterwards. The same id can still be used on the original account.

Trying a fake id generated here results in the same login notification, but I'm not able to get a stream url.

So, my guess is that they're verifying that the android device has been connected to GSF, but not necessarily that it's registered to the account. That means we could potentially hardcode ids into gmusicapi, though that doesn't seem like a good idea.

Unless someone comes up with something better, my proposal is:

  • remove from_mac_address support
  • add docs that streaming requires having previously installed the app
  • provide a command line way to list registered device ids (probably just a python -m gmusicapi...)
@simon-weber

This comment has been minimized.

Owner

simon-weber commented Jun 3, 2018

Oh, and checking how ios devices are handled is a good idea if someone gets a chance. They're not as tightly linked to Google as android devices are, so they might be a potential way to fix from_mac_address.

@ndg63276

This comment has been minimized.

ndg63276 commented Jun 5, 2018

For me, it doesn't work even when using any of the ids from the list of Mobileclient.get_registered_devices().

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 5, 2018

Do you have an android ID in that list?

@ndg63276

This comment has been minimized.

ndg63276 commented Jun 5, 2018

Both ANDROID and DESKTOP_APP in the list. Android ones are 16 digit hex, and Desktop ones are 64 chars. My current phone is on the list, and it's not its MAC address, maybe it is the Google Service Framework ID but I have no way to check that.

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 5, 2018

@ndg63276 what did you try and what does didn't work mean. You omitted the 0x off the front right and you have a subscription to google play, not the free version?

@ndg63276

This comment has been minimized.

ndg63276 commented Jun 5, 2018

I only have the free version, up until a couple of weeks ago that worked fine, using

api=Mobileclient()
api.login(email, password, "0123456789abcdef")

Now that logs in, but any call to api.get_stream_url(song_id) returns a 403. I get song_id from my uploaded songs, eg song_id = api.get_all_songs()[0]['id']

I have also tried:

for device in api.get_registered_devices():
    device_id = device['id']
    api.login(email, password, device_id)
    try:
        url=api.get_stream_url(song_id)
        print url
    except:
        api.logout()

This started happening when I got a new phone, so I thought maybe I'd hit a limit on number of devices (I had 9), but even after deauthorising an old device through the website, the problem remains.

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 6, 2018

What's the description of the 403? They return 403 for a few different reasons. Could it be that you're trying them too quickly and you're getting 403 forbiddened by rate limits and not by invalid device?

@ndg63276

This comment has been minimized.

ndg63276 commented Jun 6, 2018

Traceback (most recent call last):
File "", line 1, in
File "", line 2, in get_stream_url
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/utils/utils.py", line 293, in wrapper
return function(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/clients/mobileclient.py", line 372, in get_stream_url
return self._make_call(mobileclient.GetStreamUrl, song_id, device_id, quality)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/clients/shared.py", line 84, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/protocol/shared.py", line 226, in perform
raise CallFailure(err_msg, call_name)
gmusicapi.exceptions.CallFailure: GetStreamUrl: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&songid=dada4a40-1e5a-3710-8109-8120fddf38e6&pt=e&slt=1528311299883&tier=fr&sig=9CUUKbYP1-GgiccDhu-qkS7l8CU&hl=en_US&dv=0&net=mob
(requests kwargs: {u'url': u'https://mclients.googleapis.com/music/mplay', u'headers': {u'X-Device-ID': '1311768467294899695', u'Authorization': u''}, u'allow_redirects': False, u'params': {u'opt': u'hi', u'songid': u'dada4a40-1e5a-3710-8109-8120fddf38e6', u'pt': u'e', u'slt': '1528311299883', u'tier': u'fr', u'sig': '9CUUKbYP1-GgiccDhu-qkS7l8CU', u'hl': u'en_US', u'dv': 0, u'net': u'mob'}, u'method': u'GET'})

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented Jun 6, 2018

Both ANDROID and DESKTOP_APP in the list. Android ones are 16 digit hex, and Desktop ones are 64 chars. My current phone is on the list, and it's not its MAC address, maybe it is the Google Service Framework ID but I have no way to check that.

The device ID was never the MAC address of the Android Device. You can check the GSF ID in the Settings of your phone. You can also download apps to display that and other information (e.g. the Android device ID). Just search for 'device id' on the Play Store.

The device ID in your failed request posted has a length of 19. Valid Android device and GSF IDs should be 16.

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 7, 2018

Mine is also 16 characters.

@ndg63276

This comment has been minimized.

ndg63276 commented Jun 7, 2018

The device ID in your failed request posted has a length of 19. Valid Android device and GSF IDs should be 16.

Mine is length 16, but it is converted to a base 10 int by the method _ensure_device_id()

>>> api.android_id
'1234567890abcdef'
>>> api._ensure_device_id()
'1311768467294899695'

But, using my Google service framework id as my device id does work! I made a stupid mistake, in my comment 2 days ago, I looped through the registered devices and tried using the id from each of them, but without accounting for the fact that they are reported with '0x' prefixes, which I failed to strip. D'oh!

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 7, 2018

@fergyfresh

This comment has been minimized.

fergyfresh commented Jun 11, 2018

I tried this on my computer and using Mobileclient.FROM_MAC_ADDRESS and the device id created by the Mac address worked with get_stream_url. I haven't tried it back on my PAAS code, but I am confused again. So </3

@roadriverrail

This comment has been minimized.

roadriverrail commented Jul 20, 2018

Joining the thread because I just ran into this issue myself while trying to pick back up my Play Music skill for Mycroft. I can confirm that I can use the FROM_MAC_ADDRESS to login, get my registered device list, and then hard-code an id out of there to make things work again. It'd be awesome to have an alternate solution, of course, since it feels like I'd be inviting the wrath of Google to go around spoofing my phone's ID on my Mycroft skill.

@budowski

This comment has been minimized.

budowski commented Jul 25, 2018

What worked for me was using @ndg63276's answer, just stripping the 0x in the beginning:

mobile_client = Mobileclient()
mobile_client.login('mymail@gmail.com', 'mypass', mobile_client.FROM_MAC_ADDRESS)

device = mobile_client.get_registered_devices()[0]
device_id = device['id'][2:]
mobile_client = Mobileclient()
mobile_client.login('mymail@gmail.com', 'mypass', device_id)

# Continue as normal
@fergyfresh

This comment has been minimized.

fergyfresh commented Jul 25, 2018

@budowski do you know if it works with every ID type or just the ANDROID_ID type? I have a couple devices registered to my account so I was just checking if maybe I should also check for the device ID type that works for me.

For example, the iPhone's device ID doesn't have a 0x in front of it, as was the case with a few others when I looked at them a few weeks ago.

@roadriverrail

This comment has been minimized.

roadriverrail commented Jul 25, 2018

@budowski I agree that this works personally and for now, but I am really leery of distributing code to others saying "This works only if you have an Android device already." and "You're technically abusing your Android device's ID and Google might take action against you." I can certainly keep doing things that way for personal development, but it feels like an unreasonable requirement and risk to give any users of my code.

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented Jul 25, 2018

@roadriverrail By using gmusicapi at all, you're using private API calls that Google might already take action against you for using. None of this is supposed to be used by 3rd parties. And there are actually some things possible with this that would REALLY whip Google into a frenzy if they were abused.

That being said, I certainly wouldn't distribute one of my device IDs in code. Google now requires a valid Android (probably iOS, too) device ID for streaming. There is nothing to be done about it. If your application doesn't allow for users to provide their own Android device ID in some kind of configuration or hard-code it pre-install, then you're out of luck.

@fergyfresh We don't know. Earlier in this thread, Simon asked if someone could test with an iOS device ID. No responses were given.

@budowski

This comment has been minimized.

budowski commented Jul 25, 2018

I use Androids only.
But I have an iPhone laying around and can test this - how do I detect the ID for that device? Do I just need to install Play Music for it to work?

@fergyfresh

This comment has been minimized.

fergyfresh commented Jul 25, 2018

yeah, install play music and auth to that device with your account.

then you call the same function get_registered_devices() on your computer or whatever and then that new device (iPhone) should be in that list as type iPhone or something. I don't know as I don't have an iPhone.

@budowski

This comment has been minimized.

budowski commented Jul 26, 2018

Tested this with an iPhone (IOS 10.3.3): The device ID is indeed different, needs a different mechanism for building the ID to be legal:

mobile_client = Mobileclient()
mobile_client.login('mymail@gmail.com', 'mypass', mobile_client.FROM_MAC_ADDRESS)

device = mobile_client.get_registered_devices()[0]
if device['type'] == 'ANDROID':
    device_id = device['id'][2:]
elif device['type'] == 'IOS':
    device_id = 'ios' + device['id'][4:]

mobile_client = Mobileclient()
mobile_client.login('mymail@gmail.com', 'mypass', device_id)

However, even though calling an API function like get_all_user_playlist_contents and add_store_tracks works for both iOS and Android clients, calling get_stream_url with the iOS client, returns a 403 error (while it works for the Android client, with the same params).

@thebigmunch

This comment has been minimized.

Contributor

thebigmunch commented Jul 26, 2018

@budowski I'm not quite sure how you came up with 'ios' + device['id'][4:]. The returned ID previously was of the form: ios:01234567-0123-0123-0123-0123456789AB. If the returned ID is still in that form, then you wouldn't need to process it (and you missed the colon with what you did). If it's no longer in that form, then I'd be interested in seeing what form it takes. And if that doesn't work, you can try it without the ios: prefix.

PS Almost none of the calls ever required any kind of device ID. It was just the streaming calls and the podcast listing calls. That's why it doesn't matter for most calls.

@fergyfresh

This comment has been minimized.

fergyfresh commented Jul 26, 2018

So my snapchat got pwned somehow and posted a stream url of showtunes to every one of my snapchat contacts. Not sure if its a contaminated url as the url was deleted. I changed all my passwords and made everything 2 factor auth, but I was just checking to see if anyone else got pwned to see if this was related to our ANDROID_ID hack.

@budowski

This comment has been minimized.

budowski commented Jul 26, 2018

@thebigmunch - I came up with that format, since when I tried to login using the iPhone ID I received (as is, from device['id']), I got the following error:

gmusicapi.exceptions.InvalidDeviceId: Invalid device_id ios:1234ABCD-AAAA-BBBB-CCCC-1234567890AB. Your valid device IDs are:
* 123456789ABCDEF0
* ios1234ABCD-AAAA-BBBB-CCCC-1234567890AB

Where the first ID is the Android, and the second one is the iPhone device.

It seems like that's the logic the gmusicapi library uses to validate the device ID while doing a login (@simon-weber):

def _validate_device_id(self, device_id, is_mac=False):
"""Ensure that a given device_id belongs to the user supplying it."""
if is_mac: # Always allow logins with MAC address.
return device_id
devices = [
d['id'][2:] if d['id'].startswith('0x') else d['id'].replace(':', '')
for d in self.get_registered_devices()
]
if device_id in devices:
return device_id
else:
self.logout()
raise InvalidDeviceId('Invalid device_id %s.' % device_id, devices)

@simon-weber

This comment has been minimized.

Owner

simon-weber commented Sep 8, 2018

Heads up that the ios:... format ids are now accepted as of #610.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment