TooManyRedirects: Exceeded 30 redirects.

[DLHTurboDieselLtd]

import requests
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS

gateway = ApiGateway("https://spare.avspart.com/", access_key_id = 'my key', access_key_secret = 'my secret key')
gateway.start()

session = requests.Session()
session.mount("https://spare.avspart.com/", gateway)

response = session.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")
print(response.status_code)

gateway.shutdown()

When I try and send a request to the page I get this error. Not sure what's happening?

Starting API gateways in 10 regions.
Using 10 endpoints with name 'https://spare.avspart.com/ - IP Rotate API' (10 new).
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    response = session.get("https://spare.avspart.com/catalog/case/64848/4534337/677993/")
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/mnt/c/Users/hewin/Downloads/Asyncio/.venv/lib/python3.8/site-packages/requests/sessions.py", line 166, in resolve_redirects
    raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

[Ge0rg3]

Hi, do you get this error with only this site, or all sites when using the library?

[DLHTurboDieselLtd]

I can run the gateway_2 example in the readme file OK, but every time I try to make a request to any other sites I seem to get this error.

[Ge0rg3]

If this is the case, then this seems site-specific rather than a library bug, I'm afraid. They may be blocking requests from the AWS IP range, or perhaps you are missing some required headers?

[DLHTurboDieselLtd]

import requests
from bs4 import BeautifulSoup
from requests_ip_rotator import ApiGateway, EXTRA_REGIONS

if __name__ == "__main__":
    # Create gateway object and initialise in AWS
    gateway = ApiGateway("https://spare.avspart.com", regions=EXTRA_REGIONS, access_key_id = 'my key', access_key_secret = 'my key')
    gateway.start(force=True)

    # Execute Google search query from random IP
    # https://spare.avspart.com/catalog/case/64848/4534337/677993/
    session = requests.Session()
    session.mount("https://spare.avspart.com", gateway)
    response = session.get("https://spare.avspart.com")
    print(response)

If I try response = session.get("https://spare.avspart.com") with just the www.site.com pages I get a response of 200. It's just when I try and access the "https://site.com/index.html" pages I get the error.

I did try allow_redirects=False and it gives <Response [301]>.

[DLHTurboDieselLtd]

If this is the case, then this seems site-specific rather than a library bug, I'm afraid. They may be blocking requests from the AWS IP range, or perhaps you are missing some required headers?

OK, Thanks. Can you recommend how I might solve the headers issue?

[Ge0rg3]

Take a look at what headers your browser is sending via the network tab, and check that they match those sent via the library.