Nodriver proxy with authentication

[strukiii]

Hi all, I'm putting this here since there is no nodriver discussions section. I'm trying to get the browser setup with an authenticated proxy but chrome does not support credentials via browser args. I found this, where the answer from Arkadiy Bolotov is interesting. There are the same cdp methods as in his post available in Network though they are deprecated for those in Fetch. Here is my current attempt I put together this afternoon, which does intercept the requests however no authRequired events are fired as the documentation suggests.

class Scraper:
    main_tab: tab

    def __init__(self):
        loop().run_until_complete(self.run())

    async def run(self):
        browser = await start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = browser.main_tab
        await self.main_tab.send(cdp.fetch.enable(handle_auth_requests=True))
        self.main_tab.add_handler(cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(cdp.fetch.AuthRequired, self.auth_challenge_handler)
        page = await browser.get('https://www.whatismyip.com/')
        await asyncio.sleep(6000)


    async def auth_challenge_handler(self, event: cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        await self.main_tab.send(cdp.fetch.continue_with_auth(
            request_id=event.request_id,
            auth_challenge_response=cdp.fetch.AuthChallengeResponse(
                response="ProvideCredentials",
                username=USERNAME,
                password=PASSWORD
            )
        ))


    async def req_paused(self, event: cdp.fetch.RequestPaused):
        await self.main_tab.send(cdp.fetch.continue_request(request_id=event.request_id))


if __name__ == '__main__':
    Scraper()

[WindBora]

Hi

Here 2 problems arises:

1. You add handlers after enabling domain (internal implementation overrides enabling domain) - that's the reason why handlers do not receive an event
2. When you will finally receive an event, your response will block all processes

Solutions:

1. Add handlers before enabling domain. Only after adding them do enable domain
2. For such blocking requests there is a must to just use fire-and-forget method e.g. do not await for sending response on req_paused or auth_required. Just wrap the call into asyncio.create_task

Example of working code based of of yours:

class Scraper:
    main_tab: uc.Tab

    def __init__(self):
        uc.loop().run_until_complete(self.run())

    async def run(self):
        browser = await uc.start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = await browser.get("draft:,")
        self.main_tab.add_handler(uc.cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(
            uc.cdp.fetch.AuthRequired, self.auth_challenge_handler
        )
        await self.main_tab.send(uc.cdp.fetch.enable(handle_auth_requests=True))
        page = await browser.get("https://www.whatismyip.com/")
        await asyncio.sleep(6000)

    async def auth_challenge_handler(self, event: uc.cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_with_auth(
                    request_id=event.request_id,
                    auth_challenge_response=uc.cdp.fetch.AuthChallengeResponse(
                        response="ProvideCredentials",
                        username=USERNAME,
                        password=PASSWORD,
                    ),
                )
            )
        )

    async def req_paused(self, event: uc.cdp.fetch.RequestPaused):
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_request(request_id=event.request_id)
            )
        )


if __name__ == "__main__":
    Scraper()

[mushifali]

import asyncio
import nodriver as uc

These should be enough.

[tidal10]

Do you have a way to run multiple proxy sessions on a threaded environment with different proxies?

[epicwhale]

This works for me with non-ssl domains/pages, but fails when it's an https:// url I am trying to access. I get the following (screenshot below)...

Wondering is there some flag or extra args you needed to set somewhere to workaround this by any chance? @mushifali @strukiii @WindBora

[mushifali]

@epicwhale It is also working for https:// domains for me. Can you please share your code?

[epicwhale]

@mushifali looks like it was an issue with the proxy I was using, thanks for confirming! A better proxy seemed to do it for me too.

[SSujitX]

import nodriver as uc
from nodriver.cdp import fetch

PROXYHOST = "aaaaaaaaaaaaaa.ika.na.pyproxy.io"
PROXYPORT = "2000"
PROXYUSERNAME = "aaa"
PROXYPASSWORD = "bbbb"
PROXY = f"http://{PROXYHOST}:{PROXYPORT}"

async def setup_proxy(username, password, tab):
    async def auth_challenge_handler(event: fetch.AuthRequired):
        # Respond to the authentication challenge
        await tab.send(
            fetch.continue_with_auth(
                request_id=event.request_id,
                auth_challenge_response=fetch.AuthChallengeResponse(
                    response="ProvideCredentials",
                    username=username,
                    password=password,
                ),
            )
        )

    async def req_paused(event: fetch.RequestPaused):
        # Continue with the request
        await tab.send(fetch.continue_request(request_id=event.request_id))

    # Add handlers for fetch events
    tab.add_handler(
        fetch.RequestPaused, lambda event: asyncio.create_task(req_paused(event))
    )
    tab.add_handler(
        fetch.AuthRequired,
        lambda event: asyncio.create_task(auth_challenge_handler(event)),
    )

    # Enable fetch domain with auth requests handling
    await tab.send(fetch.enable(handle_auth_requests=True))


    # -------- PROXY -------- #
    browser = await uc.start(browser_args=[f"--proxy-server={PROXY}"])
    main_tab = await browser.get("draft:,")
    await setup_proxy(PROXYUSERNAME, PROXYPASSWORD, main_tab)
    ipPage = await browser.get("https://www.myexternalip.com/raw")
    await asyncio.sleep(2)
    ip = await ipPage.evaluate("document.body.textContent.trim()")
    print(f">= IP Address: {ip}")
    # -------- PROXY -------- #

[aoddspotato]

This is awesome

[kashif-ghafoor]

Thanks Brother. Saved a lot of my time.

[LuisGMM]

Hi all, I have been trying to use a proxy with nodriver for a couple of days now, without success. I tried the suggested solutions but I keep getting this error in the chrome tab: ERR_NO_SUPPORTED_PROXIES

Image:

In my case, in am trying to use smartproxy residential proxies. A snippet of working code with requests would be this:

import requests
url = 'https://ip.smartproxy.com/json'
username = 'some_user'
password = 'some_password'
proxy = f"http://{username}:{password}@es.smartproxy.com:10000"
result = requests.get(url, proxies = {
    'http': proxy,
    'https': proxy
})
print(result.text)

A working example with undetected_chromedriver & selenium_wire would be this:

import seleniumwire.undetected_chromedriver as uc

# Proxy server details
PROXYUSERNAME = 'some_user'
PROXYPASSWORD = 'some_password'
PROXYPORT = '10000'

# define your proxy credentials
proxy_username = PROXYUSERNAME
proxy_password = PROXYPASSWORD
proxy_host = 'es.smartproxy.com'
proxy_port = PROXYPORT

# form the proxy address
proxy_address = f'http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}'

# add the proxy address to proxy options
proxy_options = {
    'proxy': {
        'http': proxy_address,
        'https': proxy_address,
    }
}

if __name__ == '__main__':
    # set Chrome options
    options = uc.ChromeOptions()

    # run Chrome in headless mode
    options.headless = True
    options.add_argument('--ignore-certificate-errors')

    # create a Chrome instance with the proxy options
    driver = uc.Chrome(
        seleniumwire_options=proxy_options,
        options=options,
        use_subprocess=False,
    )

    # visit the test URL to check your proxy IP
    driver.get('https://httpbin.io/ip')
    ip_address = driver.find_element(By.TAG_NAME, 'body').text
    print(ip_address)

    # close the browser
    driver.quit()

However, when using these credentials with nodriver I get the previous error. Any ideas anyone?

Ps: @strukiii @WindBora @epicwhale @SSujitX

[SSujitX]

Hi all, I have been trying to use a proxy with nodriver for a couple of days now, without success. I tried the suggested solutions but I keep getting this error in the chrome tab: ERR_NO_SUPPORTED_PROXIES

Image:

In my case, in am trying to use smartproxy residential proxies. A snippet of working code with requests would be this:

import requests
url = 'https://ip.smartproxy.com/json'
username = 'some_user'
password = 'some_password'
proxy = f"http://{username}:{password}@es.smartproxy.com:10000"
result = requests.get(url, proxies = {
    'http': proxy,
    'https': proxy
})
print(result.text)

A working example with undetected_chromedriver & selenium_wire would be this:

import seleniumwire.undetected_chromedriver as uc

# Proxy server details
PROXYUSERNAME = 'some_user'
PROXYPASSWORD = 'some_password'
PROXYPORT = '10000'

# define your proxy credentials
proxy_username = PROXYUSERNAME
proxy_password = PROXYPASSWORD
proxy_host = 'es.smartproxy.com'
proxy_port = PROXYPORT

# form the proxy address
proxy_address = f'http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}'

# add the proxy address to proxy options
proxy_options = {
    'proxy': {
        'http': proxy_address,
        'https': proxy_address,
    }
}

if __name__ == '__main__':
    # set Chrome options
    options = uc.ChromeOptions()

    # run Chrome in headless mode
    options.headless = True
    options.add_argument('--ignore-certificate-errors')

    # create a Chrome instance with the proxy options
    driver = uc.Chrome(
        seleniumwire_options=proxy_options,
        options=options,
        use_subprocess=False,
    )

    # visit the test URL to check your proxy IP
    driver.get('https://httpbin.io/ip')
    ip_address = driver.find_element(By.TAG_NAME, 'body').text
    print(ip_address)

    # close the browser
    driver.quit()

However, when using these credentials with nodriver I get the previous error. Any ideas anyone?

Ps: @strukiii @WindBora @epicwhale @SSujitX

First, tell me which library you want to use: nodriver, seleniumwire, seleniumbase, or requests?

[LuisGMM]

Sure, maybe I explained myself wrong, sorry about that :)
I would love to use nodriver with a proxy, as described above by other colleagues, however I cannot manage to make it work as I could using requests or undetected_chromedriver with seleniumwire

[ultrafunkamsterdam]

await start(browser_args=["--proxy-server=socks5://127.0.0.1:9150"])

[LuisGMM]

What does that mean @ultrafunkamsterdam ?