Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FoursquareException: you are using automation tools to browse the website #88

Closed
jacebrowning opened this issue Oct 27, 2018 · 9 comments
Closed

FoursquareException: you are using automation tools to browse the website #88

jacebrowning opened this issue Oct 27, 2018 · 9 comments

Comments

@jacebrowning
Copy link
Contributor

Starting yesterday, I can no longer retrieve Foursquare data with this library.

client.users.checkins() is raising a FoursquareException, which appears to contain HTML instead of a JSON response:

Invalid response: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Access to this page has been denied.</title> <link href="https://fonts.googleapis.com/css?family=Open+Sans:300" rel="stylesheet"> <style> html, body { margin: 0; padding: 0; font-family: 'Open Sans', sans-serif; color: #000; } a { color: #c5c5c5; text-decoration: none; } .container { align-items: center; display: flex; flex: 1; justify-content: space-between; flex-direction: column; height: 100%; } .container > div { width: 100%; display: flex; justify-content: center; } .container > div > div { display: flex; width: 80%; } .customer-logo-wrapper { padding-top: 2rem; flex-grow: 0; background-color: #fff; visibility: hidden; } .customer-logo { border-bottom: 1px solid #000; } .customer-logo > img { padding-bottom: 1rem; max-height: 50px; max-width: 100%; } .page-title-wrapper { flex-grow: 2; } .page-title { flex-direction: column-reverse; } .content-wrapper { flex-grow: 5; } .content { flex-direction: column; } .page-footer-wrapper { align-items: center; flex-grow: 0.2; background-color: #000; color: #c5c5c5; font-size: 70%; } @media (min-width: 768px) { html, body { height: 100%; } } </style> <!-- Custom CSS -->  </head> <body> <section class="container"> <div class="customer-logo-wrapper"> <div class="customer-logo"> <img src="" alt="Logo"/> </div> </div> <div class="page-title-wrapper"> <div class="page-title"> <h1>Please verify you are a human</h1> </div> </div> <div class="content-wrapper"> <div class="content"> <div id="px-captcha"> </div> <p> Access to this page has been denied because we believe you are using automation tools to browse the website. </p> <p> This may happen as a result of the following: </p> <ul> <li> Javascript is disabled or blocked by an extension (ad blockers for example) </li> <li> Your browser does not support cookies </li> </ul> <p> Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading. </p> <p> Reference ID: #4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa </p> </div> </div> <div class="page-footer-wrapper"> <div class="page-footer"> <p> Powered by <a href="https://www.perimeterx.com/whywasiblocked">PerimeterX</a> , Inc. </p> </div> </div> </section> <!-- Px --> <script> window._pxAppId = 'PX0UqK4c76'; window._pxJsClientSrc = '/0UqK4c76/init.js'; window._pxFirstPartyEnabled = true; window._pxVid = ''; window._pxUuid = '4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa'; window._pxHostUrl = '/0UqK4c76/xhr'; </script> <script src="/0UqK4c76/captcha/captcha.js?a=c&u=4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa&v=&m=0"></script> <!-- Custom Script -->  </body> </html>

with the relevant part of the message being:

Access to this page has been denied because we believe you are using automation tools to browse the website.
This may happen as a result of the following:

  • Javascript is disabled or blocked by an extension (ad blockers for example)
  • Your browser does not support cookies

Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading.

Is there anything this library can do to mitigate this?
@mLewisLogic
Copy link
Owner

mLewisLogic commented Oct 28, 2018 •
edited
Loading

I'm guessing that Foursquare either changed an API endpoint or they made a targeted decision to restrict access rights of your keys due to your usage patterns (TOS violation).

If it's the later, that's between you and Foursquare.

If it's the former, could you check that the developer "explore" interface still works for this endpoint?

@diegmonti
Copy link

I have experienced a similar problem with the Foursquare APIs by using the requests library directly.
I was finally able to solve the problem by setting the User-Agent to the one of a well-known browser.

@ghost
Copy link

ghost commented Oct 30, 2018 •
edited by ghost
Loading

I am getting this too all of a sudden. What's strange is the version I have in PHP is working fine, just the Python returns this long error response.

@diegmonti How did you set the user-agent?

Edit: Got it working using
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}

@diegmonti
Copy link

I set the user-agent like the latest version of Firefox on Ubuntu, but I think that any widespread browser will work.

@mLewisLogic
Copy link
Owner

Yikes. Seems like a server issue on Foursquare's side of things... Likely an overzealous spam detection heuristic or ML model.

I'm not currently comfortable spoofing a legit browser as default functionality. Feels shady. At the end of the day, Foursquare is in charge of what sort of API traffic they want to receive.

Hopefully @diegmonti's suggestion should get folks through until Foursquare fixes things on their end. Thanks for the help @diegmonti!

I'll keep this issue open for others to find until it gets resolved by Foursquare.

@mLewisLogic
Copy link
Owner

Anybody know anybody at @foursquare? My contacts there have long since departed.

Also, what's the repro? Just this endpoint or others?
Any counterfactual examples where this is still working for other people?

@diegmonti
Copy link

I have checked again and the filter based on the user-agent seems now gone.

@jacebrowning
Copy link
Contributor Author

I am able to authenticate in my app again as well.

@cglacet
Copy link

cglacet commented Oct 28, 2021

Does anyone knows how to solve this issue? I'm getting this error even though I'm only using the API. The foursquare dashboard doesn't show any error. Note that I don't even scrape data, I just forward user requests injecting our client id/secret. Strangely if I copy paste my server generated URL in my browser or postman it does work.

Using a user agent doesn't make any difference in my case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mLewisLogic @jacebrowning @diegmonti @cglacet