Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have Tesco changed their login #80

Open
jpf198 opened this issue May 14, 2020 · 52 comments
Open

Have Tesco changed their login #80

jpf198 opened this issue May 14, 2020 · 52 comments

Comments

@jpf198
Copy link

jpf198 commented May 14, 2020

I am getting errors on Tesco login, been working flawlessly up till now?

1589493023812 - Tesco - Thu, 14 May 2020 21:50:23 GMT Logging in with new user session Navigation Timeout Exceeded: 30000ms exceeded

@git-creat
Copy link

Yeah Tesco has stopped working for me for the last day or so also, getting a similar error message.

@andytson
Copy link
Collaborator

odd, it's still working for me. I don't have it running all day every day tho, skipping nights

@CharlesButcher
Copy link

Still working for me.

I don't know if this is relevant, but the Tesco website was performing very oddly last time we placed an order. For instance, selected items were being added to the basket in twos and it was hard to knock this down to one. My wife swears that at the best of times stuff disappears from the basket, but this was much worse than usual.

@scottri007
Copy link

scottri007 commented May 15, 2020

Tesco is still working for me and the most reliable one. Morrison and and Asda nearly always timed out for me, yet when i checked manually the response times were fine.

@jpf198
Copy link
Author

jpf198 commented May 15, 2020

Thanks guys for confirming, must be my end, I'll have a play later with different useragents see if that does anything.

@paulmaunders
Copy link
Owner

It's stopped working for me as well...

$ node ./src/index.js cron
1589544360296 - Tesco - Fri, 15 May 2020 12:06:00 GMT
Logging in with new user session
unexpected error:
{ TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (/home/paulmaunders/delivery-slot-bot/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21)
  -- ASYNC --
    at Frame.<anonymous> (/home/paulmaunders/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (/home/paulmaunders/delivery-slot-bot/node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (/home/paulmaunders/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:112:23)
    at goto (/home/paulmaunders/delivery-slot-bot/src/puppeteer-utils.js:62:43)
    at TescoStore.login (/home/paulmaunders/delivery-slot-bot/src/stores/tesco.js:58:11)
    at TescoStore.start (/home/paulmaunders/delivery-slot-bot/src/stores/tesco.js:89:18)
    at TescoStore.checkDeliveries (/home/paulmaunders/delivery-slot-bot/src/stores/tesco.js:104:16)
    at runStore (/home/paulmaunders/delivery-slot-bot/src/index.js:33:56)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'TimeoutError' }```

@paulmaunders
Copy link
Owner

I get a similar but slightly different error when running on my MacBook:

unexpected error:
TimeoutError: Navigation timeout of 30000 ms exceeded
    at /Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/LifecycleWatcher.js:101:25
    at async FrameManager.navigateFrame (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/FrameManager.js:92:21)
    at async Frame.goto (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/FrameManager.js:376:16)
    at async Page.goto (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/Page.js:623:16)
    at async goto (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/puppeteer-utils.js:62:32)
    at async TescoStore.login (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:58:5)
    at async TescoStore.start (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:89:7)
    at async TescoStore.checkDeliveries (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:104:5)
    at async runStore (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/index.js:33:44)
    at async run (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/index.js:57:5)
  -- ASYNC --
    at Frame.<anonymous> (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:82:19)
    at Page.goto (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/Page.js:623:53)
    at Page.<anonymous> (/Users/paulmaunders/Documents/Code/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:83:27)
    at goto (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/puppeteer-utils.js:62:43)
    at TescoStore.login (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:58:11)
    at TescoStore.start (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:89:18)
    at TescoStore.checkDeliveries (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/stores/tesco.js:104:16)
    at runStore (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/index.js:33:56)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async run (/Users/paulmaunders/Documents/Code/delivery-slot-bot/src/index.js:57:5) {
  name: 'TimeoutError'
}```

@jpf198
Copy link
Author

jpf198 commented May 15, 2020

Yep set a manual useragent in config.ini and it works fine again. They obviously have some sort of tracking of agents that aren't "normal"... and debian with chromium isn't well used is it?!....

Used this from issue #56 👍

useragent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"

@jpf198 jpf198 closed this as completed May 15, 2020
@jpf198 jpf198 reopened this May 15, 2020
@jpf198
Copy link
Author

jpf198 commented May 15, 2020

Paul (sorry I closed the issue on you!) have you tried changing useragent?... looks like you just have a more verbose error level, but you have a login time out too...

@saqmaster
Copy link

I've tried a different user agent and confirmed it's being used, but unfortunately am still blocked. Assume they have bot detection, or some basic WAF rules, that have now blacklisted the IP I am using.

I can wget tesco.com but not the login page.. so it looks that the block is occuring at "https://secure.tesco.com/account/en-GB/login". I don't have loads of machines (or time at the moment), to try and circumvent the block. Feels like it's being done at IP level at the moment for me as changing user agent isn't affecting it. Of course you can block on other factors too.

@git-creat
Copy link

I managed to get it working again changing user agent.

@saqmaster
Copy link

I managed to get it working again changing user agent.

Same client connecting IP?

@git-creat
Copy link

I managed to get it working again changing user agent.

Same client connecting IP?

I always use a VPN, change IP address almost every day.

But today on the same IP address, this wasn't working, then started working after changing user agent.

@saqmaster
Copy link

saqmaster commented May 15, 2020

I can't even wget the homepage now:

Resolving www.tesco.com (www.tesco.com)... 104.123.91.150
Connecting to www.tesco.com (www.tesco.com)|104.123.91.150|:443... connected.
HTTP request sent, awaiting response...

edit: managed to test from a different virtual machine on a different IP, same, blocked and times out after 30 seconds. Using:

  try {
    const page = await browser.newPage();
    await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');
    await page.setViewport({ width: 1366, height: 768 });

@git-creat
Copy link

I guess it's a good idea to only use the bot sparingly, they might notice and put blocks in place if people have it running all the time.

@saqmaster
Copy link

I guess it's a good idea to only use the bot sparingly, they might notice and put blocks in place if people have it running all the time.

Yes, that's possible, however I span up a VM in GCP and tried to wget the homepage and that refused to work too, completely vanilla. So maybe their WAF team are being super aggressive. Maybe someone is abusing this script.

@paulmaunders
Copy link
Owner

paulmaunders commented May 15, 2020

I think maybe something odd is going on with the config parsing. In my config I had this Useragent set:

useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36';

But logging it to the console when the script runs seems so show it being truncated after the semi-colon:

yarn run v1.22.4
$ node ./src/index.js
Useragent: "Mozilla/5.0 (Macintosh

Changing the useragent also worked for me, so I've had a go at implementing a random user agent string - this might be a good generic solution: #82

@saqmaster
Copy link

saqmaster commented May 15, 2020

I think maybe something odd is going on with the config parsing. In my config I had this Useragent set:

useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36';

But logging it to the console when the script runs seems so show it being truncated after the semi-colon:

yarn run v1.22.4
$ node ./src/index.js
Useragent: "Mozilla/5.0 (Macintosh

Changing the useragent also worked for me, so I've had a go at implementing a random user agent string - this might be a good generic solution: #82

To be honest Paul I am running an older version of the code, as I had to fudge some of the sandbox stuff and the newer version took care of those arguments in a different way.

For example in src/puppeteer-utils.js:

function getBrowser() {
  if (process.env.PUPPETEER_BROWSER_WS_ENDPOINT) {
    return puppeteer.connect({
      browserWSEndpoint: process.env.PUPPETEER_BROWSER_WS_ENDPOINT,
    });
  }
  return puppeteer.launch({args:['--no-sandbox']});
}

and in src/index.js:

try {
    const page = await browser.newPage();
    await page.setUserAgent('Mozilla/5.0 CK={} (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko');

        await page.goto('https://google.com');
        const userAgent = await page.evaluate(() => navigator.userAgent );
        console.log(userAgent);

    await page.setViewport({ width: 1367, height: 800 });

Output confirms the right user agent is being sent, as it's retrieved from visiting google.com as an example before hitting tesco:

[x@a delivery-slot-bot]$ node delivery-slots.js
1589558812340 Fri, 15 May 2020 16:06:52 GMT
Mozilla/5.0 CK={} (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Logging in with new user session
Navigation Timeout Exceeded: 30000ms exceeded

@jpf198
Copy link
Author

jpf198 commented May 15, 2020

I guess it's a good idea to only use the bot sparingly, they might notice and put blocks in place if people have it running all the time.

Yes, that's possible, however I span up a VM in GCP and tried to wget the homepage and that refused to work too, completely vanilla. So maybe their WAF team are being super aggressive. Maybe someone is abusing this script.

If I was their admins I would block all ips that are GCP, AWS or Alibaba... as that will not be real users... so I put a bitnami image on my machine at home.

Also with VPN's get uk routing as they would probably block all known non UK IPs.

@paulmaunders
Copy link
Owner

Changing the user agent worked for me but my server is not on GCP or AWS.

As @jpf198 suggests they may have blocked the large cloud platforms... @saqmaster have you tried running the script from a machine on your home network?

@paulmaunders
Copy link
Owner

I think I spoke to soon, I just check the script on my server and it's getting errors

<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;groceries&#46;tesco&#46;com&#47;groceries&#47;en&#45;GB&#47;slots&#47;delivery" on this server.<P>
Reference&#32;&#35;18&#46;28367a5c&#46;1589570760&#46;1836fd90
</BODY>
</HTML>

It's still working from my local machine though.

@rgaufman
Copy link

rgaufman commented May 15, 2020

It's not working for me :(

$ node delivery-slots.js
1589584301949 - Tesco - Fri, 15 May 2020 23:11:41 GMT
Logging in with new user session
unexpected error:
Error: net::ERR_TIMED_OUT at https://secure.tesco.com/account/en-GB/login?from=https%3A%2F%2Fwww.tesco.com%2Fgroceries%2Fen-GB%2Fslots%2Fdelivery
    at navigate (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/FrameManager.js:120:37)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async FrameManager.navigateFrame (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/FrameManager.js:94:17)
    at async Frame.goto (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/FrameManager.js:406:12)
    at async Page.goto (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/Page.js:674:12)
    at async goto (/Users/hackeron/Development/delivery-slot-bot/src/puppeteer-utils.js:62:32)
    at async TescoStore.login (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:58:5)
    at async TescoStore.start (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:89:7)
    at async TescoStore.checkDeliveries (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:104:5)
    at async runStore (/Users/hackeron/Development/delivery-slot-bot/src/index.js:33:44)
    at async run (/Users/hackeron/Development/delivery-slot-bot/src/index.js:57:5)
  -- ASYNC --
    at Frame.<anonymous> (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (/Users/hackeron/Development/delivery-slot-bot/node_modules/puppeteer/lib/helper.js:112:23)
    at goto (/Users/hackeron/Development/delivery-slot-bot/src/puppeteer-utils.js:62:43)
    at TescoStore.login (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:58:11)
    at TescoStore.start (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:89:18)
    at TescoStore.checkDeliveries (/Users/hackeron/Development/delivery-slot-bot/src/stores/tesco.js:104:16)
    at runStore (/Users/hackeron/Development/delivery-slot-bot/src/index.js:33:56)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async run (/Users/hackeron/Development/delivery-slot-bot/src/index.js:57:5)

But I am able to login from the browser. Maybe they are checking the user agent??

This helps, thank you!

useragent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"

@paulmaunders
Copy link
Owner

@rgaufman Are you running it from a data centre or from your home network?

Could you try this branch which sets a random user agent to see if it helps? https://github.com/paulmaunders/delivery-slot-bot/tree/random-usergent

@rgaufman
Copy link

From home, setting a useragent works :)

@paulmaunders
Copy link
Owner

It looks like Tesco have blocked major cloud hosting platforms. In the short term, running it from your home network seems to work, in the longer term we could consider integrating Tor.

@andytson
Copy link
Collaborator

works ok on the cloud hosting platform I'm using, wont mention tho, but it's also major

@aaronwardle
Copy link

@rgaufman Are you running it from a data centre or from your home network?

Could you try this branch which sets a random user agent to see if it helps? https://github.com/paulmaunders/delivery-slot-bot/tree/random-usergent

Hi I have just tried this branch and it looks like it still gets an error although my my setup I get this error: TypeError: Right-hand side of 'instanceof' is not an object
at runStore (/home/tescobot/delivery-slot-bot/src/index.js:49:13)

Thanks

@IanB999
Copy link

IanB999 commented May 20, 2020

seems like this is being blocked with AWS hosts

working from home PC .

@rgaufman
Copy link

Seems like this stopped working? I'm getting this:

$ node delivery-slots.js
1590065349829 - Tesco - Thu, 21 May 2020 12:49:09 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini

Is it still working for you guys?

@git-creat
Copy link

Seems like this stopped working? I'm getting this:

$ node delivery-slots.js
1590065349829 - Tesco - Thu, 21 May 2020 12:49:09 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini

Is it still working for you guys?

I got that error a couple of days ago, but it went away when I input a new random user agent in config.ini

Mine is still working with a new user agent set. Think if you don't change it every so often it gets blocked.

@rgaufman
Copy link

I tried changing the user agent, but it started to fail again, was working fine last week. Any ideas? <

$ node delivery-slots.js
1593155900037 - Tesco - Fri, 26 Jun 2020 07:18:20 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini

@jpf198
Copy link
Author

jpf198 commented Jun 26, 2020 via email

@rgaufman
Copy link

Yes, 100%, I copy-pasted into the browser from the config file and it logs in successfully. Is it working for you?<

error: Auth failed. Please check details are correct in config.ini
1593185700300 - Tesco - Fri, 26 Jun 2020 15:35:00 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
1593186000183 - Tesco - Fri, 26 Jun 2020 15:40:00 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
1593186300282 - Tesco - Fri, 26 Jun 2020 15:45:00 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini
1593186600264 - Tesco - Fri, 26 Jun 2020 15:50:00 GMT
Logging in with new user session
error: Auth failed. Please check details are correct in config.ini

@jpf198
Copy link
Author

jpf198 commented Jun 26, 2020 via email

@rgaufman
Copy link

Ah, weird! - When I try to login from my home EE connection, it doesn't work, but when I tether through my phone on an EE LTE connection it works! - I'm using a Macbook Pro at home in both cases.

@jpf198
Copy link
Author

jpf198 commented Jun 26, 2020 via email

@git-creat
Copy link

git-creat commented Jun 26, 2020

My bot is still working, but I haven't had to use it for a few weeks as in my area tesco have gone back to releasing slots 3 weeks ahead at midnight, so I log on at midnight on a Tuesday and get a slot three weeks ahead.

I think they stopped releasing them at midnight for a few months, but started doing it again a few weeks ago. I think they only release a small percentage of the slots at midnight and then release some during the day at random times.

You can only change your tesco order until 11:45pm the day before your delivery, one delivery driver told me that their algorithm then calculates the weight of all the existing orders for the next day, if the van is still under it's weight limit they release another slot just after midnight for the same day.

@rgaufman
Copy link

Hi guys, is this working for you at the moment? - last time I managed to get it working by using 4G tethering to my phone, as they blocked my home BT connection. But this trick isn't working anymore :(

@scottri007
Copy link

scottri007 commented Sep 24, 2020 via email

@jpf198
Copy link
Author

jpf198 commented Sep 29, 2020

I have not used for a while too, but turned back on the VM today.

Had aplay with various useragent strings and connectivty and looks like they may have changed the logon process?

Some agents get a time out and other just get a failed. But can log on from a Chromium with the same useragent set as a custom; so either they have picked up a way of detecting the bot and blocking or they have changed the logon process.

I have only dabbled in Puppet code, so wont attempt to fix, but might try and debug further.

@andytson
Copy link
Collaborator

andytson commented Sep 29, 2020

odd, I've had it running from the start and it still works for me, just is too noisy now

@jpf198
Copy link
Author

jpf198 commented Sep 29, 2020 via email

@jpf198
Copy link
Author

jpf198 commented Sep 29, 2020

Arrrrhhh looks like they have tweaked the reCaptcha tollerances.....

@andytson
Copy link
Collaborator

actually, I just checked my logs and it's just intermittently working every few days, tho last was 22nd Sep

@CharlesButcher
Copy link

I'm getting authentication failures now too, whereas previously I'd survived quite well when others were reporting problems. I've only done a couple of quick tests, since – probably like many other people – I've found it easy to get slots via the standard route.

useragent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36'

@jpf198
Copy link
Author

jpf198 commented Oct 1, 2020 via email

@rgaufman
Copy link

rgaufman commented Nov 9, 2020

Any update on this? - still not able to use with tesco :(

@jpf198
Copy link
Author

jpf198 commented Nov 9, 2020 via email

@dand11
Copy link
Contributor

dand11 commented Nov 14, 2020

I'm having the same issue. I've captured a screenshot of the error. It's not a captcha!
screenshot

@ScottM00
Copy link

Any update on Tesco logging in error?

@rgaufman
Copy link

Any update please? - I miss this app :)

@rgaufman
Copy link

rgaufman commented Feb 1, 2021

Happy New Year!! Any update?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests