-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to automate login to gmail via TagUI - Google now unblock? #660
Comments
Attached a snippit on the login codes below :
|
Hi Lionel, thanks for raising this! Yes I can replicate the issue. After getting the message I click for more info and see the following screenshot (to attach files, just drag and drop into this comment textbox, but some company network maybe has firewall that blocks uploading to GitHub this way) - It looks like Chrome has an update that explicitly prevents automated browser activities, see point 3 above on 'Use automation testing frameworks' being one of the scenarios where logins can be blocked. If Google explicitly wants to block, there may not be a good solution for this. Copying @siowyisheng to tap on your ideas and experience on how to tackle this. In the meantime, a viable alternative is if your automation can be done using visual automation (see here for details https://github.com/kelaberetiv/TagUI#visual-automation), then you can use the computer vision, OCR, keyboard and mouse automation capabilities of TagUI to automate the same workflow on your real Chrome browser with your real user login. |
Adding on, looks like many apps and users are impacted and many are unhappy with the change - |
Hm I don't know of any good way to get around this, even if we want to (considering that they are intentionally blocking automated web browser use). |
We might have to follow how other test automation tools like Selenium and especially Puppeteer are going to address this. Problem with Chrome is it doesn't allow downloading of old releases. I googled and saw that a workaround is downloading older versions of Chromium. But that will almost certainly requires modification of TagUI config to work - in tagui\src\tagui and tagui.cmd. Would not want to go via this path as apps are still being tested for Chrome, not Chromium. So using Chromium will not exactly replicate the same webpage rendering and experience as a normal user would expect, and that will add another layer of complexity (and thus potential point of failure for automations). |
I'll follow-up on this one |
Some findings -
https://github.com/puppeteer/puppeteer/blob/8b49dc62a62282543ead43541316e23d3450ff3c/lib/Launcher.js
From above, it could be now that Puppeteer switches to using the pipe mode (using the file-system) versus port mode (using memory) to do the communication with Chrome, it leaves Chrome flexible to block out apps that uses port mode to communicate and flag as automated browser. Of course, this is a hypothesis, which can be verified if we do some POC to connect to Chrome with pipe instead of port. I don't know how to do that yet though. But putting this info here in case we want to review switching to pipe mode if that's the best practice Puppeteer is using. And if the hypothesis is true, it might be the only way to continue logging in automatically to Google accounts and services when users are using TagUI. And we might have to review the technical implementation how to achieve this, and any catch on existing features to switch. |
If sending mail via Gmail is the issue, would the use of external CLI mailers be a solution for some cases? I've tested the use of SwithMail ( https://sourceforge.net/projects/swithmail/ )with Gmail/G-Suite accounts via their SMTP interface. It works. Has anyone tried Nodemailer ( https://nodemailer.com/about/ ), a node.js mailer module ? |
An update that chrome-remote-interface maintainer responded and he said that using the pipe mode to connect also has this login issue - cyrus-and/chrome-remote-interface#381 (comment) Thanks @mbsryu yes that should work, though where possible we would prefer to go via UI, so that the automation actions can be seen happening, and also to focus on UI automation instead of through API (though that's possible with api step and API section under Developers Reference) or in cases of CLI, using run step to invoke the target app. |
Seems like Selenium is affected, from one of the users
Another user from above thread
No other leads for now, pending more developments in other tools or user inputs. |
Another Google user Have enough info to verify that Selenium is impacted, they may not have issues raised yet maybe because no one has raised it yet or they have some other channel like Slack or mailing list which we didn't find.
|
Someone suggested overriding the user agent setting of the browser, can try this next.
|
Modifying useragent fails, below still gets insecured browser message and login is blocked -
Chrome DevTools API reference - |
Hi @LIONEL743 it could turn out that Google may continue to block automated login to Gmail and has no workaround if going by accessing the web elements of the webpage directly. Have you tried using the visual automation method in my first reply above to see if that works for your use case? |
unassigning for now pending more developments. ideal case is Google nudges and re-enable such use cases, less ideal but still good is a workaround for web-based automation found from other tools. worst case, use visual automation since Google wants to start throwing its weight around to decide what should and should not be allowed by users, reminds me of Apple with its recent new releases making things tougher both for users and developers. CC @siowyisheng |
Hi Ken, Yes I did. Unfortunately, because the chrome window is opened using visual automation, that also means it wont be possible to retrieve contents using xpath (which is part of the automation script which i created to pull information from certain sites, before logging into gmail). Working around this, I had to split my script into two separate parts, one to retrieve the information and store them somewhere, while the other opens a web-browser using visual automation to login and send the information over. In the meantime, I have also created a script to send information via an outlook account (as a last resort) should the need arises in the future. In the mean time, I am waiting to see how things goes for now. |
Hi Lionel, thanks for sharing your update! I'm hoping that Chrome team do some form of roll-back for this bad change, or some workaround to continue doing automation using web identifiers. For the splitting into 2 parts, it should still be doable in the same file, just that the scrapping part you use XPath and so on to automate on the TagUI-invoked Chrome browser sessions, and the second part you using visual automation to invoke Chrome and gmail. Doing in a single browser would be hard with visual automation for scrapping, even with using keyboard step and clipboard() helper function. As it's not that precise and powerful in selecting web elements like using XPath. |
An update that the Google support staff has locked the support question from further posting - |
Closing issue for now as there's nothing to be done, but keeping a lookout for more datapoint from other users and other apps, to see if something can be done. In the meantime, using visual automation to invoke a normal Chrome session and automating the actions on Gmail that way should work, as the automated login and interactions are transparent to Gmail that way. |
Hi @LIONEL743 and @siowyisheng want to update that today I tried logging in to Gmail using TagUI automation and it seems to work for me! Maybe they roll-back the unnecessary security measure after hearing feedback from the community. Not sure it will work for you. I just did a simple workflow and it is able to login to Gmail -
|
The question arises ... what exactly is gmail/google detecting ? And if they "detect" --remote-debugging what else are they detecting. |
@kensoh Indeed! I retried my script and it works now, strangely |
Oh great! At least from my tests, it looks like they somewhat did a partial rollback. That at least for some accounts (my GSuite Gmail) it works but for some accounts (eg my personal Gmail) it fails. |
@sfantu yes I believe it is by detecting the remote-debugging switch. Detecting that is enough to sniff an automated Chrome session. Due to the architecture of Chrome, it is from what I know the only way to open a backdoor to control Chrome. Without this switch, there is no WebSocket connection for TagUI to control and read data from Chrome. For accounts that does not work, the alternative is purely use visual automation methods (UI image snapshots, keyboard combinations and keys, (x,y) coordinates of identifiers) to do the Gmail automation. This way Google can't detect because everything should appear like a normal user to Google web server. There are advanced detection methods like tracking mouse movements to see if it contains the 'noise' inherent in human users but hopefully they don't go that far. |
Hey everyone, I just tried something out that worked for me after several hours of trial and error. Adding PS - I was doing an automation using protractor so you might want to find out how to add this to your config with whatever automation tool you are using. |
Hi @p-funky Yinka, I appreciate that you share this info here! I've tried and unfortunately the workaround does not work in this case, maybe because TagUI always will open a websocket connection to Chrome with the What's interesting is why Protractor works but not TagUI. Hi @siowyisheng, can you have a look at this? Protractor is a testing library by Angular built on WebDriver. |
Hi @kensoh thanks for your response. After further tests yesterday, I realized what I tried was not why it worked i.e. I removed the args and it was still working. Weird thing is that the email was previously been denied access after sign in. Not sure what exactly changed anymore to be honest. Still investigating it and trying out theories. Two things I know I did and I want to explore:
Perhaps one or more of a combination of the above affected my email and gave me access. |
Hi Yinka, thanks for sharing back these details! Can you tell me more about the Stack Overflow Google login method you tried? |
@kensoh sure. Coincidentally, I have had someone else try that out as well and after about 24 hours, he can now use his email to login directly without going through stackoverflow.
After doing this consistently for like a whole day(24 hours), try automating your login directly to gmail directly... that's what we both have done. |
Thank you very much for sharing @p-funky Yinka! I confirm that steps 1-5 works! Fyi @siowyisheng |
@kensoh |
@jaisonclarck manually respond to the captcha request. It should stop after a bit. |
Thanx, I will try that ....but what if there is negative possibility ??? |
@jaisonclarck you saw my reponse here yeah? That's the only steady solution I have found thus far. The whole idea is that after about 24 hours of using the Google login strategy via Stack Overflow, you should be able to login directly without having to go through Stack Overflow anymore. Of course, if you (and hopefully you do) get there, then that's a steady solution. |
@p-funky |
Hi Ken,
Recently there seems to be a change in chrome's behavior where the chrome instance created by TagUI cannot be used to login to Gmail. This seems to have started early this week. I have tried both normal and incognito mode but it seems like both encountered the same issue. The error returned by Gmail after keying in the username and password goes like "This browser or app may not be secure." (couldn't attach the screenshot via GitHub, seems to get an error on this)
Tried on different PCs and different accounts, the result were the same as well. However, using just chrome without initiating from TagUI on these PCs to login gmail works fine. Do you experience the same issue too?
Thanks for your help!
Regards,
Lionel
The text was updated successfully, but these errors were encountered: