uc.Remote for use in Docker container

[ajtadeo]

I'm trying to use Undetected Chrome driver within a Docker container, but there seems to be no Remote module to do so. This is the code I'm trying to use, as seen in this tutorial (https://www.browserstack.com/guide/run-selenium-tests-in-docker).

options = uc.ChromeOptions()
options.add_argument('--headless=new')
options.add_argument("--disable-gpu")
options.add_argument('--blink-settings=imagesEnabled=false')

driver = uc.Remote(
    command_executor=f'http://{os.environ.get("SELENIUM_SERVICE_NAME")}:4444/wd/hub', 
    options=options, 
)

[matez0]

Did you managed to pass the cloudflare on https://nowsecure.nl/ as in the README example? - Me not.

But first, you have to build a patched remote selenium driver:

# Dockerfile
FROM selenoid/chrome:117.0
USER root
RUN apt-get update || true
RUN apt-get install -y --no-install-recommends pip
RUN pip install undetected-chromedriver
COPY patch_driver.py .
RUN python3 patch_driver.py

and

# patch_driver.py
from undetected_chromedriver import Patcher

Patcher(executable_path='/usr/bin/chromedriver').patch() or quit(1)

Then you can make a workaround:

import undetected_chromedriver as uc


class RemoteService:
    SELENOID_HOST = 'your-selenoid-host'
    service_url = f'http://{SELENOID_HOST}:4444/wd/hub'  # If you use the driver container directly, remove `/wd/hub`.
    path = '/usr/bin/true'  # some existing fake path

    def start(self):
        pass

    def stop(self):
        pass


class UdChrome(uc.Chrome):
    def __init__(self):
        options = uc.ChromeOptions()
        # Set your options here...
        options._session = self
        
        super(uc.Chrome, self).__init__(options=options, service=RemoteService(), keep_alive=True)
        
        # Copy here what else you want from the uc.Chrome constructor...
        self._delay = 3
        self.options = options


# Test whether it passes the cloudflare check:
driver = UdChrome()
driver.get('https://nowsecure.nl')
import time; time.sleep(5)
driver.save_screenshot('nowsecure.png')

Note: the code of undetected_chromedriver is a bit chaotic.

[rchvalbo]

@ajtadeo any luck with this solution? Was hoping to see if you had any luck before I took a stab at it. If you haven't tried yet I'll probably try this out soon.

[matez0]

There is an issue with inheriting from uc.Chrome. It is better to inherit from normal selenium Chrome and picking / calling some methods from uc.Chrome class.

What I found useful so far is only the patching. One can make a multi-stage build for the patching as well patching the binary in a python:alpine image and copying back the patched file into the selenoid/chrome image to avoid running the apt and installing extra packages that are not used runtime.

[matez0]

What I would like to know that could someone reproduce to pass the cloudflare on https://nowsecure.nl/ as in the README example?

[hnnoye04]

@matez0 #1577 prob try with chrome 116?

[matez0]

Using this docker file:

# Dockerfile
ARG CHROME_VERSION

FROM selenoid/chrome:${CHROME_VERSION:-latest} as selenoid_chrome


FROM python:3.10-alpine as patcher

RUN pip install undetected-chromedriver

COPY --from=selenoid_chrome /usr/bin/chromedriver /usr/bin/chromedriver

COPY patch_driver.py .

RUN python3 patch_driver.py


FROM selenoid_chrome

USER root

COPY --from=patcher /usr/bin/chromedriver /usr/bin/chromedriver

ENV DRIVER_ARGS="--headless=new --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36' --verbose"

RUN AAA=$(echo '#!/bin/bash\necho $DRIVER_ARGS' ; cat /entrypoint.sh ; ) && echo "$AAA" > /entrypoint.sh

USER selenium

where the user agent option from #1577 and the headless option are explicitly added to the driver arguments

and with the patch driver script from this comment

I built a chrome driver 116 image:

docker build -t ud-chrome-116 --build-arg=CHROME_VERSION="116.0" .

Whether the DRIVER_ARGS is passed to the driver can also be checked:

echo "tail /entrypoint.sh" | docker run --rm -i -p 127.0.0.1:4444:4444 --entrypoint=/bin/bash ud-chrome-116

The following test script extends the undetectable chrome driver to be able to use a remote chrome driver:

# test_uc.py
import undetected_chromedriver as uc


class RemoteService:
    SELENOID_HOST = '127.0.0.1'
    service_url = f'http://{SELENOID_HOST}:4444'
    path = '/usr/bin/true'

    def start(self):
        pass

    def stop(self):
        pass


class Uchrome(uc.Chrome):
    def __init__(self, options):
        options._session = self
        super(uc.Chrome, self).__init__(options=options, service=RemoteService(), keep_alive=True)
        self._delay = 3
        self.options = options


options = uc.ChromeOptions()
options.headless=True

ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'
options.add_argument(f'--user-agent={ua}')
options.add_argument("--headless=new")

driver = Uchrome(options)
driver._configure_headless()
driver.get('https://nowsecure.nl')
import time; time.sleep(5)
driver.save_screenshot('nowsecure.png')

Instead of calling the original constructor, I try to set everything to configure headless and also set the user agent option from #1577.

Starting the container:

docker run --rm -p 127.0.0.1:4444:4444 ud-chrome-116

And running the test script, the captured screenshot shows that the cloudflare is not passed.

@hnnoye04, you can execute these steps and see what happens.

[hnnoye04]

@matez0 can your one pass iphey/pixelscan? If not maybe it no using the ucdriver

[matez0]

With these additional options to get bigger screenshot:

options.add_argument("--window-size=1024,1620")
options.add_argument("--start-maximized")

my result for that site:

You can see that the site detected chrome 117 because it was given in the user agent option, but I used chrome driver 116.
If I changed the chrome version in the user agent option, then the site display the changed chrome version.

[hnnoye04]

[matez0]

I could not reproduce so far what undetected chromedriver states in the readme.

@hnnoye04 Could you provide a docker file to reproduce the stated feature for passing cloudflare on https://nowsecure.nl?

[hnnoye04]

[hnnoye04]

Without undetected driver you will get block by most bot detection website with this driver you can hide your fingerprint and auto frame work

[ajtadeo]

@matez0 do you mind posting the code snippet/instructions for how you implemented the solution? glad to see a workaround was found!

[hnnoye04]

@ajtadeo build the docker then take a look at seleniumbase patch don’t want to get by cf soon

[matez0]

@ajtadeo, @hnnoye04
The workaround is not only relying on the undetected-chromedriver, but uses some code from seleniumbase/SeleniumBase@be477fa. The SeleniumBase has MIT license and it would be good to know how much one can reuse from their solution which here is not so much.

The open or get method of the undetected_chromedriver.Chrome class could be implemented like seleniumbase.core.browser_launcher.uc_open function.

I experimented with the driver options suggested by @hnnoye04 and only these options were really necessary to pass the cloudflare:

options.add_argument('--disable-blink-features=AutomationControlled')
with open('ext.crx', 'wb') as f:
    f.write(requests.get('https://nopecha.com/f/ext.crx').content)
options.add_extension('ext.crx')

these options were not necessary:

options.set_capability("selenoid:options", {"enableVNC": True})
options.add_argument("--start-maximized")
options.add_argument('--disable-features=TranslateUI')
options.add_argument('--disable-translate')
options.add_argument('--lang=en-US')
options.add_argument('--disable-web-security')
options.add_argument("--no-sandbox");
options.add_argument("--disable-popup-blocking")
options.add_argument("--disable-notifications")
options.add_argument('--disable-dev-shm-usage')
options.add_experimental_option("excludeSwitches", ['enable-automation'])
options.add_experimental_option('useAutomationExtension', False)
ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36'
options.add_argument(f'--user-agent={ua}')
options.add_argument("--headless=new")

I used the chrome driver version 116 and even removed the setting of DRIVER_ARGS from the docker file mentioned here.

We do not need the undetected_chromedriver.Chrome class neither.
Only the patched selenoid image from here.

Note: To pass the cloudflare, the nopecha chrome extension was essential. Without this extension and the special window opening based on the SeleniumBase, using the patched chrome driver and the undetected_chromedriver.Chrome class cannot pass the cloudflare as stated in the README.

Finally, the reduced script:

import time

import requests
from selenium.webdriver.chrome.remote_connection import ChromeRemoteConnection
from selenium.webdriver.chromium.options import ChromiumOptions
from selenium.webdriver.remote.webdriver import WebDriver

SELENOID_HOST = 'localhost'
executor = ChromeRemoteConnection(remote_server_addr=f'http://{SELENOID_HOST}:4444')

options = ChromiumOptions()

options.add_argument('--disable-blink-features=AutomationControlled')

with open('ext.crx', 'wb') as ext_file:
    ext_file.write(requests.get('https://nopecha.com/f/ext.crx').content)
options.add_extension('ext.crx')

driver = WebDriver(command_executor=executor, options=options)

url = 'https://nowsecure.nl'
driver.execute_script(f'window.open("{url}","_blank");')
time.sleep(3)
driver.execute_script("window.close();")
driver.switch_to.window(driver.window_handles[-1])
time.sleep(8)
driver.save_screenshot('nowsecure.png')

Of course, you can use Uchrome class from here or similar instead of WebDriver, if you would like to use something from the undetected_chromedriver.Chrome class as well.

[hnnoye04]

@ajtadeo if you wish to use the extension follow this guide https://developers.nopecha.com/guides/extension/
and test on any on these https://nopecha.com/demo

[hnnoye04]

@matez0 Just like thissss i just copy the code from seleniumbase/SeleniumBase@be477fa xddd

[Tongcheng]

Doesn't work for me when using the above window open/close tactic with nopecha, does it still work for you?

[hnnoye04]

try use nodriver or seleniumbase directly this repo no update again more.This version of ud driver detect already

[hnnoye04]

from seleniumbase import SB

def open_the_turnstile_page(sb):
    sb.driver.uc_open_with_reconnect(
        "https://nowsecure.nl/", reconnect_time=2.5,
    )

def click_turnstile_and_verify(sb):
    sb.driver.uc_switch_to_frame("iframe")
    sb.driver.uc_click("span.mark")
    sb.assert_element("img#captcha-success", timeout=3.33)

with SB(uc=True, test=True) as sb:
    open_the_turnstile_page(sb)
    try:
        click_turnstile_and_verify(sb)
    except Exception:
        open_the_turnstile_page(sb)
        click_turnstile_and_verify(sb)
    sb.set_messenger_theme(location="top_left")
    sb.post_message("SeleniumBase wasn't detected", duration=3)

cause uc_open_with_reconnect work with seleniumbase driver

[Tongcheng]

Appreciate the help and directions @hnnoye04 ! Let me try both!

[big-camel]

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": """
      var newProto = navigator.__proto__;
      delete newProto.webdriver;
      newProto.permissions={query:function(parameters){ return parameters.name === 'notifications' ? Promise.resolve({ state: Notification.permission }) : originalQuery(parameters) }};
      newProto.language='en';newProto.languages=['zh-CN', 'zh', 'en'];
      navigator.__proto__ = newProto;
    """
  })

Some applications may use JavaScript to detect the webdriver property

[Tongcheng]

Thank you @big-camel ! Will try it out!