Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message: session not created: Chrome failed to start: exited normally.(session not created: DevToolsActivePort file doesn't exist)(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) #2247

Closed
STARKILLER-1 opened this issue Nov 7, 2023 · 20 comments
Labels
incorrect usage You may need to change what you're doing question Someone is looking for answers UC Mode Undetected Chromedriver Mode (--uc)

Comments

@STARKILLER-1
Copy link

STARKILLER-1 commented Nov 7, 2023

I'm sorry for may be being stupid, but..

Bug only on ubuntu 22.04. On Windows 11 no issues!

I have been struggling with this problem for several days:

THIS CODE WORKS ON UBUNTU 22.04:

from selenium import webdriver
from selenium_stealth import stealth
import seleniumbase as sb
def initialize_driver():
    try:
        options = webdriver.ChromeOptions()
        options.add_argument("start-maximized")
        options.add_argument("--headless") 
        driver = sb.Driver(browser='chrome', headless=True, uc=True)
        wait = webdriver.Chrome.implicitly_wait(driver,500.00)
        stealth(driver,
                languages=["en-EN", "en"],
                vendor="Google Inc.",
                platform="Win64",
                webgl_vendor="Intel Inc.",
                renderer="Intel Iris OpenGL Engine",
                fix_hairline=True,
                wait=wait
                )
        time.sleep(3)
        return driver
    except:
        return None

THIS CODE ALWAYS FAILS ON UBUNTU 22.04:

def initialize_driver(account_number, cookies_file, url, proxy, attempt=1):
    if attempt > 5: 
        return None
    try:
        driver = sb.Driver(browser='chrome', headless=True, uc=True, proxy=proxy)
        wait = webdriver.Chrome.implicitly_wait(driver,500.00)
        stealth(driver,
                languages=["en-EN", "en"],
                vendor="Google Inc.",
                platform="Win64",
                webgl_vendor="Intel Inc.",
                renderer="Intel Iris OpenGL Engine",
                fix_hairline=True,
                wait=wait
                )
        driver.get(url)
        time.sleep(3)
        with open(cookies_file, 'r') as f:
            cookies = json.load(f)
            current_domain = driver.current_url  
            for cookie in cookies:
                if 'sameSite' in cookie and cookie['sameSite'] not in ['Strict', 'Lax', 'None']:
                    del cookie['sameSite']
                if 'domain' in cookie:
                    if current_domain.find(cookie['domain']) != -1:  # Check if the cookie domain is in the current domain
                        driver.add_cookie(cookie)
        driver.refresh()
        time.sleep(3)
        try:
            wait = WebDriverWait(driver, 2)
            wait.until(EC.presence_of_element_located((By.XPATH, "//*[contains(normalize-space(), 'Enter or registrate')]")))
            driver.quit()
            return initialize_driver(account_number, cookies_file, url, proxy, attempt + 1)
        except Exception as e:
            print(e)
        return driver
    except Exception as e:
        print(e)
        driver.quit()
        return None

I was debugging alot. On Windows 11 no issues!
Only this code can go through Cloudflare on a target site

Thank you!

@mdmintz mdmintz added question Someone is looking for answers incorrect usage You may need to change what you're doing labels Nov 7, 2023
@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

Greetings. Let's go through your code and give you some solutions to work with...

You've got multiple recursive calls happening with initialize_driver(), which is certainly going to cause various issues. (See the code below, where I highlighted it in two places.)

Screenshot 2023-11-07 at 12 39 19 PM

The officially-supported way of doing multithreading in UC Mode is by using pytest (via pytest-dist). Since it looks like you're setting multiple simultaneous proxies, you'll need to use --multi-proxy / multi_proxy=True with that. (See #1832)

Depending on your Ubuntu configuration, you may need to use --xvfb / xvfb=True instead of using headless mode. Headless mode is the default on Ubuntu, but there is a --headed / headed=True option to override.

You may be overcomplicating your selenium-stealth setup. You can integrate it directly into the custom setUp() method of your tests like this: (A BaseCase format lets you use proper multithreading with pytest / pytest-xdist.)

from seleniumbase import BaseCase
from selenium_stealth import stealth

class BaseTestCase(BaseCase):
    def setUp(self):
        super().setUp()
        stealth(self.driver,
            languages=["en-US", "en"],
            vendor="Google Inc.",
            platform="Win32",
            webgl_vendor="Intel Inc.",
            renderer="Intel Iris OpenGL Engine",
            fix_hairline=True,
        )

Then have your test classes inherit BaseTestCase instead of BaseCase. (See SeleniumBase/help_docs/syntax_formats.md#sb_sf_02)

SeleniumBase methods have automatic-waiting. You should never be using the external implicitly_wait, WebDriverWait, or EC.presence_of_element_located anywhere in your code. Use the built-in methods instead. For the raw driver formats, see examples such as SeleniumBase/examples/raw_login_driver.py, SeleniumBase/examples/raw_driver_manager.py, and SeleniumBase/examples/offline_examples/test_extended_driver.py. For the pytest formats, see any example test that starts with test_ or ends with _test in the SeleniumBase/examples folder.

Also note that you have unused code in your examples. You declared options with options = webdriver.ChromeOptions(), but never used it. You shouldn't be using that at all since SeleniumBase options are passed via pytest command-line args, or via method args, eg. driver = Driver(uc=True).

Finally, selenium-stealth is an external repo with its own issues. SeleniumBase won't magically fix those issues if you encounter them while using both frameworks together. To determine between the two, try examples with just SeleniumBase (but not selenium-stealth), and see if those issues still happen.

@mdmintz mdmintz closed this as completed Nov 7, 2023
@mdmintz mdmintz added the UC Mode Undetected Chromedriver Mode (--uc) label Nov 7, 2023
@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 7, 2023

@mdmintz I found the issue point
driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)# not working on ubuntu 22.04
driver = sb.Driver(browser='chrome', headless=False, uc=True)# working on ubuntu 22.04
the question was about the proxy with auth (definitely working) but the raw driver cannot connect to them.
Somehow "proxy =" argument is not working on ubuntu (selenium-wire works with it. But it's not so good as SeleniumBase =)
Do you know easy way to connect SeleniumBase raw Driver to proxy?

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

Are your proxy strings in a valid format?
host:port or user:pass@host:port ?

Are you using multithreading / multiple proxies? (Then use pytest and multi_proxy=True)

from parameterized import parameterized
from seleniumbase import BaseCase
BaseCase.main(__name__, __file__, "-n3")

class ProxyTests(BaseCase):
    @parameterized.expand(
        [
            ["user1:pass1@host1:port1"],
            ["user2:pass2@host2:port2"],
            ["user3:pass3@host3:port3"],
        ]
    )
    def test_multiple_proxies(self, proxy_string):
        self.get_new_driver(
            undetectable=True, proxy=proxy_string, multi_proxy=True
        )
        self.driver.get("https://browserleaks.com/webrtc")
        self.sleep(30)

Or if not using multithreading and not using multiple proxies at the same time, here's a way to do it using the Driver():

driver = Driver(uc=True, proxy="user:pass@host:port")

@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 7, 2023

@mdmintz Nope, just a single driver at a time
My proxies are strings Example_proxy="XXXX:XXXX@XX.XX.XX.XXX:8000"
I'm confused..
Never had issues with those proxies.

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

Do you have a stack trace? It'll make it easier to debug.

@STARKILLER-1
Copy link
Author

Ok, give me a minute, i will make a sample script and test it

@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 7, 2023

XXXXXXXXXXXXXXX/test# python3 main.py
Proxy: XXXXXX:XXXXX@XX.XX.XX.237:8000
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 3342, in get_local_driver
    driver = undetected.Chrome(
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/undetected/__init__.py", line 305, in __init__
    super().__init__(options=options, service=service_)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 205, in __init__
    self.start_session(capabilities)
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/undetected/__init__.py", line 430, in start_session
    super().start_session(capabilities)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 289, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 344, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
from chrome not reachable
Stacktrace:
#0 0x5598162d2933 <unknown>
#1 0x559815fac546 <unknown>
#2 0x559815f9821a <unknown>
#3 0x559815fe372c <unknown>
#4 0x559815fdad59 <unknown>
#5 0x55981601e77a <unknown>
#6 0x5598160153d3 <unknown>
#7 0x559815fe7e64 <unknown>
#8 0x559815fe8c4e <unknown>
#9 0x559816298558 <unknown>
#10 0x55981629c4a0 <unknown>
#11 0x5598162a697c <unknown>
#12 0x55981629d0b8 <unknown>
#13 0x559816268cdf <unknown>
#14 0x5598162c1048 <unknown>
#15 0x5598162c1219 <unknown>
#16 0x5598162d1ac3 <unknown>
#17 0x7f31683c3ac3 <unknown>


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/champcommerce/test/main.py", line 38, in initialize_driver
    driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/plugins/driver_manager.py", line 453, in Driver
    driver = browser_launcher.get_driver(
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 1524, in get_driver
    return get_local_driver(
  File "/usr/local/lib/python3.10/dist-packages/seleniumbase/core/browser_launcher.py", line 3702, in get_local_driver
    driver = webdriver.Chrome(service=service)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py", line 56, in __init__
    super().__init__(
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 205, in __init__
    self.start_session(capabilities)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 289, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 344, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
#0 0x55a6bd2b8933 <unknown>
#1 0x55a6bcf926f7 <unknown>
#2 0x55a6bcfc5a55 <unknown>
#3 0x55a6bcfc1bbc <unknown>
#4 0x55a6bd00477a <unknown>
#5 0x55a6bcffb3d3 <unknown>
#6 0x55a6bcfcde64 <unknown>
#7 0x55a6bcfcec4e <unknown>
#8 0x55a6bd27e558 <unknown>
#9 0x55a6bd2824a0 <unknown>
#10 0x55a6bd28c97c <unknown>
#11 0x55a6bd2830b8 <unknown>
#12 0x55a6bd24ecdf <unknown>
#13 0x55a6bd2a7048 <unknown>
#14 0x55a6bd2a7219 <unknown>
#15 0x55a6bd2b7ac3 <unknown>
#16 0x7f5ccfb19ac3 <unknown>

@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 7, 2023

def initialize_driver(account_number, cookies_file, url, proxy, attempt=1):
    if attempt > 5: 
        return None
    try:
        print(f"Proxy: {proxy}")
        driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)
        driver.get(url)
        time.sleep(3)
        with open(cookies_file, 'r') as f:
            cookies = json.load(f)
            current_domain = driver.current_url  # get the current domain
            for cookie in cookies:
                if 'sameSite' in cookie and cookie['sameSite'] not in ['Strict', 'Lax', 'None']:
                    del cookie['sameSite']
                if 'domain' in cookie:
                    if current_domain.find(cookie['domain']) != -1:  
                        driver.add_cookie(cookie)
        driver.refresh()
        time.sleep(3)
        try:
            wait = WebDriverWait(driver, 2)
            wait.until(EC.presence_of_element_located((By.XPATH, "//*[contains(normalize-space(), 'Enter or login in')]")))
            if driver:
                driver.quit()
            return initialize_driver(account_number, cookies_file, url, proxy, attempt + 1)
        except Exception as e:
            send_telegram_notification(e)
        return driver
    except Exception as e:
        send_telegram_notification(e)
        if driver:
            driver.quit()
        return None

Same simple raw driver

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

I think you missed a few important parts from my original answer:

Depending on your Ubuntu configuration, you may need to use --xvfb / xvfb=True instead of using headless mode. Headless mode is the default on Ubuntu, but there is a --headed / headed=True option to override.

So, don't use headless=False. Try xvfb=True instead. And if that doesn't work, headed=True.

And your script still has a recursive call in it, which could lead to multiple drivers, which may crash because you weren't using pytest to spin up your multiple browsers.

And remember that SeleniumBase methods have automatic-waiting. You should never be using the external implicitly_wait, WebDriverWait, or EC.presence_of_element_located anywhere in your code. Use the built-in methods instead.

@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 7, 2023

I have made some tests with print. Turning on proxy kills further script running.It freezes for a while and then gives an error =(
I will make all your recommendations, after script goes further then driver = sb.Driver(browser='chrome', headless=False, uc=True, proxy=proxy)
And i'm sorry, I copied the code incorrectly headless =True (I have no gui)
Thank you!
I can give you my proxy for test, if you want

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

Chrome headless mode doesn't support extensions. If you have to run on a headless machine, use headed=True or xvfb=True. (By default, it was probably using headless2=True, which is Chrome's newer headless mode.)

@STARKILLER-1
Copy link
Author

I understand, but without proxy headless works. Sorry if I misunderstood

@STARKILLER-1
Copy link
Author

OMG. It worked! driver = sb.Driver(browser='chrome', headless2=True, uc=True, proxy=proxy)

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

Yes, because proxy settings are set via a Chrome extension that gets generated at runtime. If you don't use proxy with auth, then you don't have a Chrome extension, and therefore regular headless mode works fine. (Hence the reason for Chrome's newer headless mode, which is set with headless2.)

@mdmintz
Copy link
Member

mdmintz commented Nov 7, 2023

There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266

@STARKILLER-1
Copy link
Author

@mdmintz Thank you so much! I hope you won't see me with my stupid questions soon=). I will rewrite all my code from selenium-wire to your framework, I like your Tracebacks they are so detailed and helpful

@STARKILLER-1
Copy link
Author

There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266

Yes i used it once but in original selenium(some elements were not displayed on screenshots via old headless mode). Gosh...

@STARKILLER-1
Copy link
Author

STARKILLER-1 commented Nov 8, 2023

There's a popular StackOverflow post about it: https://stackoverflow.com/a/73840130/7058266

driver = sb.Driver(browser='chrome', headless2=True, uc=True, proxy=proxy)
is unstable and works 1/10 times =(
UPD: 1/100
headed=True- not working
xvfb=True - not working
headless2=True- not working

@mdmintz
Copy link
Member

mdmintz commented Nov 8, 2023

After a quick refresher, I recalled that the Driver() format doesn't have the Xvfb launcher for Linux like the other SeleniumBase formats. You'll need to add that in separately: (https://github.com/mdmintz/sbVirtualDisplay)

from sbvirtualdisplay import Display
from seleniumbase import Driver

display = Display(visible=0, size=(1440, 1880))
display.start()

driver = Driver()
try:
    driver.open("seleniumbase.github.io/demo_page")
    driver.highlight("h2")
    driver.type("#myTextInput", "Automation")
    driver.click("#checkBox1")
    driver.highlight("img", loops=6)
finally:
    driver.quit()

display.stop()

@STARKILLER-1
Copy link
Author

Thank you. This is a stable solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
incorrect usage You may need to change what you're doing question Someone is looking for answers UC Mode Undetected Chromedriver Mode (--uc)
Projects
None yet
Development

No branches or pull requests

2 participants