RuntimeError: Tried 5 times to log in; all failed due to xfinity rejecting logins from automation engines #30
Comments
|
Thanks, I applied your changes and it's working for me as well. |
|
thanks @apippin ! that patch was a lifesaver! worked for me but i had to manually change the browser to firefox-headless from phantomjs in lines 84/85. I had to do this with chrome too when i had it working with chrome. I also downloaded geckodriver.exe into my script folder
|
|
I am fairly new to Docker and deployed container from https://hub.docker.com/r/billimek/comcastusage-for-influxdb/ Just as you, once I analyzed docker logs, I got exception due to login rejection 5 times by Comcast despite using correct username, password and image screenshot pulled from exceptions error output showed it was on correct login page. |
|
It seems to fail for me even with the changes mentioned by @apippin. It seems to fail to login 5 times for headless Firefox now too. It seems Comcast made changes that break automation bots from polling pages after login. Docker images need complete rebuild, they have a lot of dependencies on old upstream repositories set years ago such as phantomjs which is now deprecated. |
|
@tbabula Unfortunately, I haven't had Xfinity service in over 2 years. @billimek was tentatively going to take over this project, as without xfinity service, I have no way of testing it. There's not really a whole lot I can say aside from the fact that if Jeff is no longer actively using this, it may once again be in need of a new maintainer. Apologies. |
|
Thanks for response.
As you mentioned in the blog, the screen-scrapper may break anytime due to site redesign or element ID changes. I just noticed when recently I tried to login to Comcast via app, that they locked my account for password reset even though I used my correct username/password in the config file used by script for scrapping.
… On Aug 14, 2020, at 11:28 AM, Jason Antman ***@***.***> wrote:
@tbabula <https://github.com/tbabula> Unfortunately, I haven't had Xfinity service in over 2 years. @billimek <https://github.com/billimek> was tentatively going to take over this project, as without xfinity service, I have no way of testing it. There's not really a whole lot I can say aside from the fact that if Jeff is no longer actively using this, it may once again be in need of a new maintainer. Apologies.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM22SQEOL4VEC7PV7NJFMA3SAV65PANCNFSM4OWGEVMA>.
|
I am still having success with the patches suggested earlier. The one other change I realized I made was to edit line 85 to browser_name='firefox-headless' from browser_name='phantomjs' |
Yeah I had reflected that change in the line as well. But also just before patching was running in docker the original script to see it workflows in order to see how to patch rebuilt docker which seems Comcast's website IDS/IPS responded by locking out my login credentials for password reset due to phantomjs automation login. And of course there are couple more things to change than just the script. There are different ways to install gecokdriver for example. For example, using pip3 install webdriver install method did not work for me. So downloaded the gecokdriver directly and deployed to /usr/local/bin. Then if you want to deploy on the docker, using billimek/comcastUsage-for-influxDB repository, you have to think how to rebuilt his source image which involves installing some extra prerequisite dependencies that if its not handled correctly, will break install even if xfinity_usage.py script is correctly patched. |
|
I created an updated Docker image that uses firefox-headless as its browser: https://hub.docker.com/r/tomroush/comcastusage-for-influxdb. I've been running it in Kubernetes for a couple days. @apippin are you planning to make a pull request with your changes? |
If you had to change line 85, then the there must be an assignment missing when you created the instance of the class. That line is just providing a default and wherever you actually create the instance should provide an assignment for the browser you want to use. For instance, the example on the front page of this project has this line:
you need that browser_name= assignement in there. I tried setting up something myself recently that wasn't working and I couldn't figure out why until I stumbled across this project, and this issue. After pulling xfinity-usage, and making the changes suggested by apippin above, it's been running fine for the last week. I'm not using Docker though. |
I'm glad it is working for you. I'm surprised the command line -b option (like -b firefox-headless) didn't work for you to change the browser? I do agree that changing the default to be firefox-headless does make sense as phantomjs doesn't work at all. |
Great suggestion. I just pushed up a pull request that encapsulates my changes (#32). |
* Added firefox-headless as new browser type & default Cleaned up -b argument help message and choices Made new firefox-headless browser the new default browser (since it works) Added options to firefox-headless to avoid bot checks when getting usage from xfinity's site * Updated CHANGES and version file for these changes * Wrapped long lines to be under 80 chars for pep8 * Added missing whitespace for pep8 compliance Co-authored-by: Alan J. Pippin <alan@pippins.net>
yeah, i didnt try it in the command line, just trial and error until i got the to hardcoded the browser name to firefox headless and it worked! |
|
I want to start by saying thanks to all that have worked on this. In order to make this project work, I believe it needs to be fully included in a docker container for all dependencies. I have fully working solution, so I hope it benefits others like me who are victims of Xfinity/Comcast. Please feel free to open an issue even if it's working! |
|
I have it working without without docker using the geckodriver on ubuntu 20.04 without issues for the last4 months. |
|
@mbatesco Could you please share your solution? |
Basically, I installed the latest release of the geckodriver, selinium and the patch above and use firefox-headless as the browser. Other than the patch above the key was making sure that the firefox browser and geckodriver installed were the save release. Note, I have no experience with HASS on TrueNAS so not sure how much this helps. I have been thinking about the docker but need to look at how it pushes data out, I have a postgresql database and a google sheet where I store the data over time and report from them daily and monthly with graphs. |
|
Hi, I had this script working for several months without any issues, but yesterday morning (Apr 26, 9AM EST) it started throwing a "The Xfinity ID or password you entered was incorrect. Please try again". At 6AM it worked... I also received an email saying I needed to reset my password as they "detected a potential issue with my profile"... I know the login and passwords are correct and if I use the Firefox browser in that same machine "manually", it logs in without a problem... Is this also happening to any of you? |
|
No issues for me, but I'm using a derivative product, in a docker container. (see prev post for link) |
|
Looks like xfinity changed something again. Just started failing for me again and forced a password change. Mike |
|
Seems to be working fine for me. My last update was about 20 minute ago and doesn't appear to have any issues. Normally when I've had an issue one of two things fixes it:
You didn't enable two factor authentication or anything like that when you updated your password? |
|
I have 2 factor authentication turned off since I activated the script many months ago, and still is. I had it running every 3 hours... |
|
Still working for me given my changes I made in a previous post.
…On Wed, Apr 28, 2021, 11:47 AM PRDGS ***@***.***> wrote:
I have 2 factor authentication turned off since I activated the script
many months ago, and still is. I had it running every 3 hours...
As it also happened to @mbatesco <https://github.com/mbatesco> , I wonder
if this is some update they are rolling out regionally ?!? I'm in Florida...
I tried running it normally (as non-headless), it opens the browser, but
the result is the same...
I can only imagine they're doing some additional validations on
automation... The issue is figuring out what and how to work around it...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#30 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEBFBV5QAIT6627JKO4RXNLTLA32RANCNFSM4OWGEVMA>
.
|
|
@chuccck The latest version of the script already uses firefox-headless by default, so I cannot see any difference... |
|
Not sure what to tell you, I had to make that change for it to run on my
windows 10 machine
…On Wed, Apr 28, 2021, 12:32 PM PRDGS ***@***.***> wrote:
@chuccck <https://github.com/chuccck> The latest version of the script
already uses firefox-headless by default, so I cannot see any difference...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#30 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEBFBVZNLO2LVGFESQRUKADTLBBEJANCNFSM4OWGEVMA>
.
|
|
Just a couple of additional data points:
I can't see any other explanation other than they're detecting the automation/selenium... |
|
I don't use this particular script, but have a similar script that I wrote myself. I've started experiencing issues signing in, similar to the last time xfinity added measures to prevent automation. Timing of this is around April 27th or so as well. So I suspect that PRDGS is right that they are detecting the automation. I haven't had time to dig in deeper to understand how they might be detecting or what alternatives exist. |
|
Great - I'll be watching for that. Thanks again xfinity. It's annoying in the first place that I need to monitor this so that I don't go over the limit. What should we be looking for to diagnose the issue? |
|
I temporarily solved the problem by having a python script running some AppleScript (which controls safari, in my mac, which then publishes to my regular InfluxDB/Grafana VM).
A good advantage of using AppleScript is that there are no changes to the browser, therefore it should be very difficult for xfinity to ever detect and block it. For me it's a decent alternative until someone figures how to get selenium working again. If nothing happens in the next few weeks, I'll just create a mac VM as a longer term workaround... |
|
Sooooo any solution to this repo? I'm a man who knows how to install via HACS and not much else. Thanks for all who have worked on this so far. Would love a working version... |
|
Still haven't seen it show up in my area. It temporarily stopped working on the 12th, but it turned out that xfinity was putting up a window after login to ask me about 2FA. After logging manually and hitting 'skip', it now is working again. |
|
It looks like this fix was merged into version 3.0.4. However in my testing on Mac OS X it still does not work. :-( Is it working for others? |
|
I have this release (3.0.4) and I am running into this problem as of 19 days ago. I have checked 2FA off and logged in manually and for some reason can't get it working again. If I get time I may look at the docker version but will need to do something to it to feed the data to Postgresql DB that I have been using. Very frustrating not being able to figure out what Comcast did this time. |
Bug Report
It appears that the xfinity login page has started rejecting authentication attempts by automation engines like selenium or python based session requests. Even though I am using a valid username and password, and can login from the same computer and browser into my xfinity account, xfinity returns an invalid username and password error when attempting to do the same thing via this script.
Version
xfinity-usage 3.0.1 https://github.com/jantman/xfinity-usage
Installation Method
pip install xfinity-usage
Supporting Software Versions
Python 3.7.5
Virtualenv 15.1.0
selenium==3.141.0
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
Actual Output
Expected Output
I expect the script to run and return either the text output of the usage or json file of the usage from xfinity.
Testing Assistance
I've tested that selenium is being blocked by xfinity, the root of the issue, but launching each supported browser via the -b option supported by this script in selenium and typing in the same username and password and verifying that xfinity rejects them as invalid.
The good news is that I've also tested a fix to the script for firefox (and added a firefox-headless mode) that sets the following preferences that enables xfinity to allow the login to work through selenium using firefox for firefox-headless:
Here's a patchset, that if applied, fixes the issue and adds support for a new browser option, firefox-headless:
With the patch applied above, the script is able to login to xfinity and get the data usage properly. Note I've only patched firefox and firefox-headless. I tried to fix chrome and chrome-headless, but wasn't able to figure out the magic to get it to work like I do for firefox above.
The text was updated successfully, but these errors were encountered: