Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authentication failing #14

Closed
Lockszmith-GH opened this issue Jul 14, 2020 · 19 comments
Closed

Authentication failing #14

Lockszmith-GH opened this issue Jul 14, 2020 · 19 comments

Comments

@Lockszmith-GH
Copy link

Has something changed in the API?

This is what I get right now, started a few days ago:

# skybellpy -u 'user@name.email' -p 'secret_password' --devices --debug
2020-07-14 12:54:05 INFO (MainThread) [skybellpy] Updating all devices...
2020-07-14 12:54:05 DEBUG (MainThread) [skybellpy] HTTP post https://cloud.myskybell.com/api/v3/login/ Request with headers: {'user-agent': 'SkyBell/3.4.1 (iPhone9,2; iOS 11.0; loc=en_US; lang=en-US) com.skybell.doorbell/1', 'content-type': 'application/json', 'accepts': '*/*', 'x-skybell-app-id': '300b510e-d6c3-4927-90f7-b61ba541e056', 'x-skybell-client-id': 'fe2e99c9-6c4f-45d8-9bde-acf312748b2b'}
2020-07-14 12:54:06 DEBUG (MainThread) [skybellpy] <Response [403]> {"message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-west-2:********8768:4wv55lip28/v3/POST/login/ with an explicit deny"}
2020-07-14 12:54:06 ERROR (MainThread) [skybellcl] Login failed: Request failed: Retry failed

pulled the code just now to test, so it's the latest version of the code.

@adamoutler
Copy link

I'm seeing the same issue. I think the API updated. There's a version I confirmed as working here https://github.com/thoukydides/homebridge-skybell/wiki/Protocol-HTTPS

@thoukydides
Copy link

thoukydides commented Jul 17, 2020

Manually trying the request issued by skybellpy:

curl
 -X POST
 -d '{ "username": "USER@DOMAIN", "password": "PASSWORD" }'
 -H 'user-agent: SkyBell/3.4.1 (iPhone9,2; iOS 11.0; loc=en_US; lang=en-US) com.skybell.doorbell/1'
 -H 'content-type: application/json'
 -H 'accepts: */*'
 -H 'x-skybell-app-id: b3b0215d-b5b1-4ce8-8373-338380a88e1d'
 -H 'x-skybell-client-id: f16f9fbd-c1ed-4bff-aeca-24a03ff1959d'
 https://cloud.myskybell.com/api/v3/login/

results in the error reported by @Lockszmith above:

{
    "message": "User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-west-2:********8768:4wv55lip28/production/POST/v3/login/ with an explicit deny"
}

Removing the user-agent header from the request:

curl
 -X POST
 -d '{ "username": "USER@DOMAIN", "password": "PASSWORD" }'
 -H 'content-type: application/json'
 -H 'accepts: */*'
 -H 'x-skybell-app-id: b3b0215d-b5b1-4ce8-8373-338380a88e1d'
 -H 'x-skybell-client-id: f16f9fbd-c1ed-4bff-aeca-24a03ff1959d'
 https://cloud.myskybell.com/api/v3/login/

works correctly:

{
    "firstName": "Alexander",
    "lastName": "Thoukydides",
    "resourceId": "RESOURCEID",
    "createdAt": "2016-10-08T07:47:55.352Z",
    "updatedAt": "2016-10-08T07:47:55.352Z",
    "id": "ID",
    "userLinks": [
        {
            "user": "USER",
            "type": "oauth2",
            "partnerName": "nest",
            "accessToken": "ACCESSTOKEN",
            "ttl": null,
            "createdAt": "2016-10-08T07:51:09.158Z",
            "updatedAt": "2019-05-12T16:39:03.692Z",
            "meta": [
                {
                    "deviceId": "DEVICEID",
                    "structureId": "STRUCTUREID",
                    "deviceType": "co_smoke_alarm"
                },
                {
                    "deviceId": "DEVICEID",
                    "structureId": "STRUCTUREID",
                    "deviceType": "co_smoke_alarm"
                }
            ],
            "id": "ID"
        }
    ],
    "access_token": "ACCESSTOKEN"
}

Specifying a different user-agent also works. Hence, it looks like that particular user-agent string has been blocked...

@KishCom
Copy link

KishCom commented Jul 17, 2020

Hello! I'm a senior engineer at Skybell.

We did indeed have to block the particular user-agent used by this library because the way this library calls our API puts undue strain on our auth servers. We're literally in the middle of an API overhaul and I expect by very early 2021 we'll have a real developer program that allows you to register as a dev and use something like WebSockets, Webhooks, or perhaps even MQTT (we're still working out the exact details).

I can see there's no easy way for you guys to deal with this right now, so all I ask is that if you make changes to this library so it works again, that you limit your auth requests to one per minute. Unfortunately due to the way our old/current API is setup we will have to block (not even rate limit I'm afraid) clients that ignore this request. We did this original block on the user-agent level as opposed to the IP address level however now that the cat's out of the bag, we will have to start start banning actual IPs that hammer our API.

Again: a better developer friendly API is under active development. I'm sure you'll be hearing more from me in the coming weeks.

@adamoutler
Copy link

Hello! I'm a senior engineer at Skybell.

We did indeed have to block the particular user-agent used by this library because the way this library calls our API puts undue strain on our auth servers. We're literally in the middle of an API overhaul and I expect by very early 2021 we'll have a real developer program that allows you to register as a dev and use something like WebSockets, Webhooks, or perhaps even MQTT (we're still working out the exact details).

I can see there's no easy way for you guys to deal with this right now, so all I ask is that if you make changes to this library so it works again, that you limit your auth requests to one per minute. Unfortunately due to the way our old/current API is setup we will have to block (not even rate limit I'm afraid) clients that ignore this request. We did this original block on the user-agent level as opposed to the IP address level however now that the cat's out of the bag, we will have to start start banning actual IPs that hammer our API.

Again: a better developer friendly API is under active development. I'm sure you'll be hearing more from me in the coming weeks.

This IP block thing worries me. I am a small-time home user, and I have not receive a new IP from my service provider in 3 years. However, I tried changing my user-agent and restarted my home-assistant today, only to be blocked again. Is it possible that my actions could cause my IP to be blocked? Is there a time-limit on it?

@KishCom
Copy link

KishCom commented Jul 17, 2020

@adamoutler Nope, don't worry. You should be fine.

Everything is manual right now except the blocking of the user-agent string used by this library specifically -- no rate-limit or automatic IP-blocking -- we want to focus developer time on bringing up the new API. We also haven't blocked any specific IPs. We only had to do the user-agent block because lots of people using this library were making authorization calls several times per second (eg: not using our API right at all!) and clogging up the authentication service. We thought it might have been Homebridge, but it appears to be another python project that uses this library (or perhaps just hundreds of bad developers not re-using Bearer tokens 🤷‍♂️).

Don't be worried about being IP blocked unless you're re-authorizing with the Skybell API more frequently than once per minute (once authorized your Bearer token should be reused until rejected/expired, then and only then: re-auth).

@MisterWil
Copy link
Owner

MisterWil commented Jul 17, 2020

You mentioned in the linked homebridge-skybell issue that this library logins again after every request, but this is mostly incorrect. The first instantiation of the skybell class will perform a login, but a follow up login request is only done once if a different API method to update state fails. If the follow-up request fails a second time then the library will throw an exception to the top level class. See the code here to confirm this retry functionality.

So this library is functioning in exactly the same way that the homebridge-skybell issue you referenced describes their functionality.

This library was written primarily for use with Home-Assistant, and it is implemented in Home-Assistant to follow this same flow. Non-authenticating polling requests within Home-Assistant are made based on the scan interval as controlled by Home-Assistant core, and is defaulted to, I believe, once every 30 seconds. If a request to update fails, the request tries to login again, and then an exception is thrown. Another request is made 30 seconds later, a login attempt is made, if that fails it throws the exception, repeat until login is successful.

  1. Startup login
  2. Initial device API requests to get data
  3. Scan interval = 30 seconds by default triggers another round of device API requests to get any differences in state
  4. If any of those API requests fail a login is attempted
    • If login fails, an exception is thrown and execution on this scan ends
    • If login succeeds but the device API request fails again, an exception is thrown and execution on this scan ends
  5. GOTO 3

Now, if a user is making use of this library's command-line based feature, which I only implemented for testing locally, then yes, every command line request would perform a login. I'm not sure how many people are using it this way, and if they are then they shouldn't be.

If the library is utilized as it is meant to be - as a library in another project - then, again, logins are attempted only once at startup and only after failed refresh API calls, and if either fail no further attempts are made until "refresh" is called again.

So this library isn't by itself designed to pummel auth servers. Home-Assistant by default would not do a login request any more than once every 30 seconds. Users making use of this library could be using it in a way that pummels your auth servers, but I have no control of that.

The best I can think of to do is remove the command line functionality and require a User Agent be sent in. Would this be adequate?

Edit: I suppose I should say: I wrote this library to NOT login after every request and only do so when the token expires. However, it is entirely possible a bug exists that is preventing that from happening correctly. As it is designed though, each instance of the library will capture the bearer token from the login response and utilize it for follow-up requests until it expires, a logout is called, or the instance is destroyed.

@KishCom
Copy link

KishCom commented Jul 17, 2020

Hi @MisterWil ! First of all, a sincere thank you for actually building this library!

a user is making use of this library's command-line based feature, which I only implemented for testing locally

This is almost certainly the case -- either that or copy pasted what you've got here.

Users making use of this library could be using it in a way that pummels your auth servers, but I have no control of that.

Oh absolutely. It is my personal opinion that the onus is with Skybell to provide proper rate-limiting. I know it's coming with the new API.

So this library isn't by itself designed to pummel auth servers

Another sincere thank you. ☺️

The best I can think of to do is remove the command line functionality and require a User Agent be sent in. Would this be adequate?

This would be fantastic if you're willing to put the time in to do it, but by no means is it something Skybell would require or ask for.

As a hobbyist developer myself I know how frustrating it can be when companies push breaking changes downstream without notice. I starred this repo a few weeks ago, noticed someone asking about losing auth today, and knew I could provide some help. I'll definitely reach out with some documentation as we get into a beta stage with the new API.

@MisterWil
Copy link
Owner

This would be fantastic if you're willing to put the time in to do it, but by no means is it something Skybell would require or ask for.

As far as I am concerned, libraries like mine making use of undocumented "private" API's should never expect to be notified of breaking changes, should be prepared to be blocked at any time if its causing a headache for companies, and need to do their best effort to not cause headaches. I appreciate your being so willing to come forward and talk about it openly!

This weekend I'll go ahead and make the user agent changes (something like "SkyBellPy/{version}"), look into perhaps throttling the login attempt internally a bit more by tracking when a last login attempt was made, and make sure to have Home Assistant updated to use the new version with its own unique user agent identifier (something like "SkyBellPy/{version}/HomeAssistant") to help narrow down future issues.

@adamoutler
Copy link

What if a user service home-assistant restart twice?

@MisterWil
Copy link
Owner

What if a user service home-assistant restart twice?

I just looked closer at the code, and Home-Assistant is caching the bearer token and client ID's to a pickle file and restored when restarted. This means that a login attempt should only ever be made once the first time the skybell component is added, and any time the bearer token expires.

So, in your example, one login request will be made, assuming the first restart was the first time the skybell component was added.

@adamoutler
Copy link

adamoutler commented Jul 18, 2020

But the first time someone

rm ~/.homeassistant/skybell.pickle; service home-assistant restart; rm ~/.homeassistant/skybell.pickle; service home-assistant restart;

Then the user agent gets banned, correct? Maybe a per-instance user-agent, based on a random number or property of the system is in order?

@pironic
Copy link

pironic commented Jul 24, 2020

I recently purchased a skybell hd specifically because of the integration in home-assistant... I came looking for issues when i failed to set up for the first time. Am i correct in assuming that I wont be able to integrate until this new API is built in 2021? :(

@KishCom
Copy link

KishCom commented Jul 24, 2020

Am i correct in assuming that I wont be able to integrate until this new API is built in 2021?

Incorrect. Things are fine, this library is working well and will continue to. It will get even better come 2021. 👍

@pironic
Copy link

pironic commented Jul 24, 2020

Thanks so much! Wonderful to hear. I will investigate my issue elsewhere and assume this portion is not the cause. ♥️

@robyouyou
Copy link

I like pironic am still having issues using this library. I've commented out the user agent in the init.py and I still get errors (per the attached).
Screenshot from 2020-07-24 12-03-49

MisterWil added a commit that referenced this issue Jul 26, 2020
@MisterWil
Copy link
Owner

Alright. Above commit and 0.6.0 version release modifies the User-Agent to have the format: skybellpy/{version} ({identifier}) such that {version} will be the current version (0.6.0 in this case), and identifier will be a specific identifier based on the calling code. If the calling code does not specify an identifier, then default will be used as the identifier (and SkyBell could block it specifically if its causing an issue).

I left in the command line functionality because it does cache the bearer token between calls, so only one login request will be made between command line commands being executed. Login will not be called again until the bearer token is expired OR the pickle file is deleted. Command line requests will have skybellcl as the identifier part of the above User-Agent.

Home-Assistant PR was created for the reported issue to bump the version to 0.6.0 and also pass in the Home-Assistant version as the identifier: HomeAssistant/{version}.

I tested the changes locally and the login requests appeared to be going through again. I saw some people above were having issues of it not working if the User-Agent is removed, but it wouldn't surprise me if SkyBell is blocking requests without a User-Agent as then they couldn't selectively filter out requests.

Hopefully these changes are acceptable, and if there are any versions or code using this library causing issues then those specific User-Agents can be blocked in the future.

@MisterWil
Copy link
Owner

@adamoutler I just wanted to ping you here and let you know of issue #15 which might explain what you were seeing as being "banned". I think what's actually happening is after a successful login attempt the bearer token that is returned is not immediately usable. It was pretty consistent that after a successful login, the API request immediately after would fail every time. However, if I tried the same request again with the same previously failed bearer token after 3-7 seconds it would work just fine.

As such, I added an artificial 5 second delay after a successful login attempt. Suddenly, it works the first time, every time, without fail.

@Lockszmith-GH
Copy link
Author

Lockszmith-GH commented Jul 27, 2020

Thanks! according to issue #337851@home-assistant/core, the fix has been pushed to the next release of HA, so I'll be able to test this fully. But this looks like a very good fix.
Thank you so much for the quick response and designing what looks like a good solution.

@Lockszmith-GH
Copy link
Author

"I love it when a plan comes together!"

Works perfectly for a couple days now since the upgrade of HA.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants