API calls blocked #44

rboarman · 2022-01-13T17:10:29Z

As of yesterday, my Camply API calls to recreation.gov are blocked.

Here's the error I am getting back from Camply:

Error: 403
Request blocked.

<TITLE>ERROR: The request could not be satisfied</TITLE>

403 ERROR

The request could not be satisfied.

Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront)
Request ID: -duVM4wmJCI8E0sAi7ZfMguXA7_p3nVi6JRuT64Cuxp7FfeYd-lg1Q==

I've tried changing ip addresses with no luck.
I've tried changing to my own API key with no luck.

Any suggestions?

Is there a way to specify a different User-Agent header?

Thanks!!

Rick

juftin · 2022-01-13T17:46:19Z

Thanks for reporting @rboarman. I'll take a look into this - we're currently rotating between a few User-Agent headers:

camply/camply/config/api_config.py

Lines 18 to 31 in 62d1a44

    
           USER_AGENTS: List[dict] = [ 
        
               {"User-Agent": ("Mozilla/5.0 (X11; Linux x86_64; rv:10.0) " 
        
                               "Gecko/20100101 Firefox/10.0")}, 
        
               {"User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " 
        
                               "(KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36")}, 
        
               {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 " 
        
                               "(KHTML, like Gecko) Version/14.0.3 Safari/605.1.15")}, 
        
               {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 " 
        
                               "(KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36")}, 
        
               {"User-Agent": ("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) " 
        
                               "Gecko/20100101 Firefox/87.0")}, 
        
               {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " 
        
                               "(KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36")} 
        
           ]

Hopefully this is something we'll be able to get around.

juftin · 2022-01-13T18:01:54Z

Ahh, I have more info! This is a problem with the RIDB Database API.

❯ export LOG_LEVEL=debug
❯ camply campsites \
    --rec-area 2725 \
    --start-date 2022-07-12 \
    --end-date 2022-07-13 \
    --continuous \
    --notifications pushover \
    --notify-first-try
2022-01-13 10:47:50,836 [  CAMPLY]: camply, the campsite finder ⛺️
2022-01-13 10:47:50,842 [    INFO]: 1 booking nights selected for search, ranging from 2022-07-12 to 2022-07-12
2022-01-13 10:47:50,844 [    INFO]: Retrieving Facility Information for Recreation Area ID: `2725`.
2022-01-13 10:47:50,852 [   DEBUG]: Starting new HTTPS connection (1): ridb.recreation.gov:443
2022-01-13 10:47:50,908 [   DEBUG]: https://ridb.recreation.gov:443 "GET /api/v1/recareas/2725/facilities?full=true&offset=0 HTTP/1.1" 403 919
2022-01-13 10:47:50,910 [    INFO]: {'accept': 'application/json', 'apikey': 'XXXXXXXXXXXXXXXXXXX'}
2022-01-13 10:47:50,910 [   ERROR]: Receiving bad data from Recreation.gov API: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: uWqHopzPrg1JtSkQPWOfThlGmwJIVPqmmAfRNN9oZMniNvv9v-rxrg==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

camply uses two APIs to talk to recreation.gov:

https://ridb.recreation.gov
- This is the metadata Database / REST API for Recreation.gov - we get campsite, campground, and recreation area metadata from this API
- camply uses an API Key embedded into the software to get this data. You can override this with your RIDB_API_KEY environment variable
https://recreation.gov/api/camps/availability/campground/
- This is the undocumented Campsite Availability API that camply uses
- There is no authorization on this API - this is where we randomly select a User-Agent header to make requests

The strangest thing about this issue is the RIDB Database API Landing Page is displaying the following warning:

Due to high volume of API calls, cache time for RIBD’s camping-availability endpoint will increase from 15 seconds to 1 minute moving forward. Please reach out to the Contact us center for any issues in regards to API disconnecton or downtime.

My understanding though is that the camping-availability endpoint doesn't live on the RIDB 🤷‍♂️. I'll keep digging and see what I can find and follow up here

rboarman · 2022-01-13T18:09:08Z

Thank you. My calls were hitting every five minutes which should be fine right?

One thing to consider, is to allow us to pass in an array of user agents via the .camply file. That might help.

Thank you for the assistance!!

juftin · 2022-01-13T19:12:36Z

@rboarman you were absolutely right. The RIDB API is kicking back the User-Agent passed from the Python Requests Package - they haven't documented this anywhere but that looks to be the problem. It's probably just a matter of time until it invalidates the attached RIDB API Key as well -- I'll look into having it choose a random API key from a few options soon.

This PR (#45) adds a ton of User-Agents between Chrome, FireFox, and Safari and implements the RIDB API to randomly choose one for each API call.

juftin · 2022-01-13T19:32:23Z

I closed this out - tests are passing on my end and the fix has ben released to PyPi and Docker. A simple pip install --upgrade camply should fix everything on your end (once this workflow has finished). Please let me know or re-open this issue if you have any problems. Thanks for reporting @rboarman

rboarman · 2022-01-13T19:35:05Z

Thank you! I'll send you a little gift. Love your library!

juftin · 2022-01-13T19:38:33Z

Thanks - so happy you like it. I've tried to build camply to be respectful of the APIs and services it uses - this includes rate limiting and exponential backoff. However this is the first I've seen of recreation.gov pushing back and trying to crack down on tools like camply hitting their API - I wouldn't be surprised to see more bugs like this coming out in the future.

juftin self-assigned this Jan 13, 2022

juftin added the bug Something isn't working label Jan 13, 2022

juftin linked a pull request Jan 13, 2022 that will close this issue

BugFix: RIDB API Randomized User-Agent #45

Merged

juftin closed this as completed in #45 Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API calls blocked #44

API calls blocked #44

rboarman commented Jan 13, 2022

juftin commented Jan 13, 2022

juftin commented Jan 13, 2022

rboarman commented Jan 13, 2022

juftin commented Jan 13, 2022

juftin commented Jan 13, 2022 •

edited

rboarman commented Jan 13, 2022

juftin commented Jan 13, 2022

API calls blocked #44

API calls blocked #44

Comments

rboarman commented Jan 13, 2022

403 ERROR

The request could not be satisfied.

juftin commented Jan 13, 2022

juftin commented Jan 13, 2022

rboarman commented Jan 13, 2022

juftin commented Jan 13, 2022

juftin commented Jan 13, 2022 • edited

rboarman commented Jan 13, 2022

juftin commented Jan 13, 2022

juftin commented Jan 13, 2022 •

edited