Multi-threaded use of UC is impossible

[ben-elbert]

Hello, Im using UC in a multi-threaded manner.
Ever since this update, (and im quoting):

3.4.0
Big update! be careful as it -potentially- could break your code.

rewritten the anti-detection mechanism instead of removing and renaming variables, we just keep them, but prevent them from being injected in the first place. This will keep us safe from detection at least for the near future.

rewritten the file naming, to prevent ending up with 1000 of {randomstring}_chromedriver.exe 's instead it is just called undetected_chromedriver.exe

I get this error (And for a logical reason):
OSError: [Errno 26] Text file busy: ‘/root/.local/share/undetected_chromedriver/undetected_chromedriver’

Thanks in advance!

[AlexPaiva]

Not sure if this is a new error (to this version of UC) or not but in my multi threaded previous scripts I had to resort to multi-processing AND multi-threading and a very specific pooling method for it to work properly. Not sure if it's the solution or not but theres my two cents.

[krypterro]

We are running hundreds of instances on a single Ubuntu server concurrently, it works fine. However, you can't use the machine's install of Chrome. Build a portable version, and have each Python app instance use their own. We keep different directories for each one, just in case Chrome caches something, but it works.

[ColeMaverick]

Thank you! May I ask right now, do you use your original solution (build portable chrome) or do you use @ultrafunkamsterdam's pip solution? because i tried @ultrafunkamsterdam's pip solution, it didn't work...

[krypterro]

Thank you! May I ask right now, do you use your original solution (build portable chrome) or do you use @ultrafunkamsterdam's pip solution? because i tried @ultrafunkamsterdam's pip solution, it didn't work...

Actually I built a Launcher that deploys my bots in separate directories, so I have a master bot, then the Launcher runs, and clones the master into bot-slot-1, bot-slot-2, and etc. I then use psutil to monitor how many Python venv are up in the bot slots, and periodically kill the stuck ones, and only launch new instances in slots that have no active bot.

The fix I found that works, is to add import secrets to patcher.py and then:

'''
#prefix = "undetected"
prefix = secrets.token_hex(8)
'''

And I didn't monkey patch, I just cloned undetected-chrimedriver to a local folder in my app so it works in my code with the same namespace.

That being said, I look forward to trying out the solution discussed here. Although my way works, it would be better to have our app running on the main branch of a community supported library than using my patch.

[ColeMaverick]

Understood, thank you! I was able to make it work without using undetected chromedriver and just uses normal webdriver:))

[ndmd23]

@krypterro : Your post just ended a day-long search. Thank you for sharing!

[dapmel]

This is the correct answer.

I had a lot of trouble trying to orchestrate multiple instances with Airflow, using the chrome installed in the OS causes cryptic exceptions such as selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome....

I also found that only one portable executor can be enough for multiple runs and different scripts without any conflict.

The code looks like this:

    options = uc.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument("--disable-extensions")
    options.add_argument("--disable-application-cache")
    options.add_argument("--disable-gpu")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-setuid-sandbox")
    options.add_argument("--disable-dev-shm-usage")

    return uc.Chrome(
        options=options,
        use_subprocess=False,
        browser_executable_path=f"{str(Path.home())}/GCPM/data/chrome",
        user_multi_procs=True,
    )

[labodj]

@ultrafunkamsterdam I think that we should have a way to run multiple instances by design, like before 3.4, without the need to set custom options manually.

The fact is that it can happen to run completely unrelated scripts that use undetected chromedriver under the hood at the same time.

I just retested the code updated to 57a2bd3 and it seems to be working again using chrome 110.

You can try it like so

python -m pip uninstall undetected-chromedriver
python -m pip install git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance
python -m pip install --upgrade selenium

[ultrafunkamsterdam]

i just pushed a commit to #fix-multiple-instance branch
actually, it is quite an update, especially in the cleanup mechanism, and also skipping download when compatible driver is found (handy when running 100 instances, yikes)

python -m pip uninstall undetected-chromedriver -y
python -m pip install git+https://github.com/ultrafunkamsterdam/undetected-chromedriver@fix-multiple-instance

Oh, and please check if it is actually installed. pip doesn't correctly report when the installation or uninstallation has failed because some file was still in use. and then you end up unknowingly still using old version)

[shelton-xiaoteng-ma]

Sure, we assume it is important enough to not delete it

# undetected_chromedriver/patcher.py
def __del__(self):
    if self._custom_exe_path:
        # if the driver binary is specified by user
        # we assume it is important enough to not delete it
        return

also have a look here: https://blog.maxiaoteng.xyz/python/python_crawler/browser_automation.html#undetected-chromedriver

[ColeMaverick]

Thank you. But may I ask if driver.quit() is not executed, what would happen to that instance? Would not closing it burns Memory resource...?

[ultrafunkamsterdam]

Afaik multiple threads and even processes are perfectly capable of running the same executable. So just 1 driver should be sufficient. Until the first ends and removes it. Will tonight spend time to come up with a suitable solution

[ultrafunkamsterdam]

Sure, we assume it is important enough to not delete it

# undetected_chromedriver/patcher.py
def __del__(self):
    if self._custom_exe_path:
        # if the driver binary is specified by user
        # we assume it is important enough to not delete it
        return

also have a look here: https://blog.maxiaoteng.xyz/python/python_crawler/browser_automation.html#undetected-chromedriver

This is correct. Removal of the driver only happens when not specified yourself . It increases detection though since you won't get a fresh executable with randomized values this way

[shelton-xiaoteng-ma]

This is correct. Removal of the driver only happens when not specified yourself . It increases detection though since you won't get a fresh executable with randomized values this way

a fresh executable with randomized values ?, for chromedriver, patch_exe() do this, then use this patched file driver Chrome:

# undetected_chromedriver/patcher.py
new_target_bytes = (
    b'{console.log("undetected chromedriver 1337!")}'.ljust(
        len(target_bytes), b" "
    )
)
new_content = content.replace(target_bytes, new_target_bytes)

Patch content seems to be almost constant, Sorry I didn't find any other place to set random value for chromedriver.

[ultrafunkamsterdam]

i just pushed 3.4.7 - which adds more compatibility for it .
one big REQUIREMENT: you should have at least 1 undetected_chromedriver(.exe) in your appdata/roaming/undetected_chromedriver , or whereever it usually unpacks the driver to.

tested it using this lousy script

import logging
import time
import multiprocessing as mp
from queue import Empty , Full


import undetected_chromedriver as uc

logging.basicConfig( level = 10 )
logging.getLogger( 'parso' ).setLevel( 50 )


class Worker( mp.Process ):
    
    
    def __init__( self , name = None , queue = mp.Queue() , stop_ev = mp.Event() , daemon = True ):
        self.queue = queue
        self.driver = None
        self.stop_ev = stop_ev
        if name is None:
            self.__class__.n += 1
            name = f'worker-{self.__class__.n}'
        super().__init__( name = name , daemon = daemon )
    
    
    def run( self ):
        self.driver = uc.Chrome(user_multi_procs = True)
        while True:
            try:
                job = self.queue.get_nowait()
            except Empty:
                time.sleep( .1 )
                continue
            if not job or not job.get( 'action' ):
                break
            try:
                self.driver.get( job[ 'url' ] )
                print( 'loaded ' , job[ 'url' ] , 'completely' )
            except Exception as e:
                print( e )
            
            print( self.name , 'loading ' , job.get( 'url' ) )
    
    
    def start( self ) -> None:
        super().start()


USELESS_SITES = [
    
    "https://www.blogger.com" ,
    "https://www.google.com" ,
    "https://youtube.com" ,
    "https://microsoft.com" ,
    "https://linkedin.com" ,
    "https://apple.com" ,
    "https://support.google.com" ,
    "https://play.google.com" ,
    "https://youtu.be" ,
    "https://wordpress.org" ,
    "https://cloudflare.com" ,
    "https://docs.google.com" ,
    "https://maps.google.com" ,
    "https://whatsapp.com" ,
    "https://en.wikipedia.org" ,
    "https://googleusercontent.com" ,
    "https://bp.blogspot.com" ,
    "https://sites.google.com" ,
    "https://plus.google.com" ,
    "https://mozilla.org" ,
    "https://adobe.com" ,
    "https://accounts.google.com" ,
    "https://europa.eu" ,
    "https://drive.google.com" ,
    "https://uol.com.br" ,
    "https://t.me" ,
    "https://github.com" ,
    "https://vimeo.com" ,
    "https://istockphoto.com" ,
    "https://facebook.com" ,
    "https://vk.com" ,
    "https://policies.google.com" ,
    "https://amazon.com" ,
    "https://shopify.com" ,
    "https://mail.google.com" ,
    "https://opera.com" ,
    "https://google.de" ,
    "https://search.google.com" ,
    "https://globo.com" ,
    "https://live.com" ,
    "https://gstatic.com" ,
    "https://theguardian.com" ,
    "https://tiktok.com" ,
    "https://nytimes.com" ,
    "https://dropbox.com" ,
    "https://msn.com" ,
    "https://files.wordpress.com" ,
    "https://ok.ru" ,
    "https://myspace.com" ,
    "https://slideshare.net" ,
    "https://gravatar.com" ,
    "https://wikimedia.org" ,
    "https://jimdofree.com" ,
    "https://bbc.co.uk" ,
    "https://creativecommons.org" ,
    "https://wa.me" ,
    "https://news.google.com" ,
    "https://nih.gov" ,
    "https://forbes.com" ,
    "https://paypal.com" ,
    "https://brandbucket.com" ,
    "https://www.weebly.com" ,
    "https://google.co.jp" ,
    "https://medium.com" ,
    "https://google.com.br" ,
    "https://imdb.com" ,
    "https://google.es" ,
    "https://line.me" ,
    "https://who.int" ,
    "https://www.yahoo.com" ,
    "https://developers.google.com" ,
    "https://dailymotion.com" ,
    "https://fr.wikipedia.org" ,
    "https://tools.google.com" ,
    "https://es.wikipedia.org" ,
    "https://draft.blogger.com" ,
    "https://pt.wikipedia.org" ,
    "https://cnn.com" ,
    "https://feedburner.com" ,
    "https://enable-javascript.com" ,
    "https://buydomains.com" ,
    "https://goo.gl" ,
    "https://hatena.ne.jp" ,
    "https://mail.ru" ,
    "https://planalto.gov.br" ,
    "https://thesun.co.uk" ,
    "https://samsung.com" ,
    "https://google.co.uk" ,
    "https://huffpost.com" ,
    "https://afternic.com" ,
    "https://pinterest.com" ,
    "https://abril.com.br" ,
    "https://telegraph.co.uk" ,
    "https://nasa.gov" ,
    "https://amazon.co.jp" ,
    "https://twitter.com" ,
    "https://indiatimes.com" ,
    "https://youronlinechoices.com" ,
    "https://netvibes.com" ,
    "https://aboutads.info" ,
    "https://independent.co.uk" ,
    "https://bing.com" ,
    "https://pixabay.com" ,
    "https://adssettings.google.com" ,
    "https://myaccount.google.com" ,
    "https://discord.com" ,
    "https://booking.com" ,
    "https://hugedomains.com" ,
    "https://telegram.me" ,
    "https://reuters.com" ,
    "https://bloomberg.com" ,
    "https://mediafire.com" ,
    "https://time.com" ,
    "https://bit.ly" ,
    "https://cdc.gov" ,
    "https://wikia.com" ,
    "https://google.fr" ,
    "https://amazon.de" ,
    "https://webmd.com" ,
    "https://domainmarket.com" ,
    "https://photos.google.com" ,
    "https://it.wikipedia.org" ,
    "https://aliexpress.com" ,
    "https://estadao.com.br" ,
    "https://news.yahoo.com" ,
    "https://washingtonpost.com" ,
    "https://tinyurl.com" ,
    "https://foxnews.com" ,
    "https://terra.com.br" ,
    "https://change.org" ,
    "https://amazon.co.uk" ,
    "https://google.pl" ,
    "https://soundcloud.com" ,
    "https://amazon.es" ,
    "https://picasaweb.google.com" ,
    "https://w3.org" ,
    "https://huffingtonpost.com" ,
    "https://4shared.com" ,
    "https://cnet.com" ,
    "https://www.gov.uk" ,
    "https://nature.com" ,
    "https://wp.com" ,
    "https://get.google.com" ,
    "https://office.com" ,
    "https://cbsnews.com" ,
    "https://businessinsider.com" ,
    "https://ft.com" ,
    "https://un.org" ,
    "https://ig.com.br" ,
    "https://de.wikipedia.org" ,
    "https://networkadvertising.org" ,
    "https://ytimg.com" ,
    "https://marketingplatform.google.com" ,
    "https://issuu.com" ,
    "https://mirror.co.uk" ,
    "https://abc.es" ,
    "https://shutterstock.com" ,
    "https://cpanel.net" ,
    "https://plesk.com" ,
    "https://wsj.com" ,
    "https://dailymail.co.uk" ,
    "https://ru.wikipedia.org" ,
    "https://namecheap.com" ,
    "https://forms.gle" ,
    "https://instagram.com" ,
    "https://imageshack.us" ,
    "https://list-manage.com" ,
    "https://usatoday.com" ,
    "https://yadi.sk" ,
    "https://scribd.com" ,
    "https://t.co" ,
    "https://dan.com" ,
    "https://elpais.com" ,
    "https://storage.googleapis.com" ,
    "https://researchgate.net" ,
    "https://google.it" ,
    "https://bbc.com" ,
    "https://archive.org" ,
    "https://fandom.com" ,
    "https://fb.com" ,
    "https://php.net" ,
    "https://sciencedirect.com" ,
    "https://disqus.com" ,
    "https://lemonde.fr" ,
    "https://ja.wikipedia.org" ,
    "https://netflix.com" ,
    "https://express.co.uk" ,
    "https://engadget.com" ,
    "https://mit.edu" ,
    "https://canada.ca" ,
    "https://wired.com" ,
    "https://radiofrance.fr" ,
    "https://thenai.org" ,
    "https://aol.com" ,
    "https://imageshack.com" ,
    "https://themeforest.net" ,
    "https://picasa.google.com" ,
    "https://zdf.de" ,
    "https://lefigaro.fr" ,
    "https://usnews.com" ,
    "https://mega.nz" ,
    "https://fb.me" ,
    "https://loc.gov" ,
    "https://ftc.gov" ,
    "https://arxiv.org" ,
    "https://deezer.com" ,
    "https://offset.com" ,
    "https://biglobe.ne.jp" ,
    "https://discord.gg" ,
    "https://clickbank.net" ,
    "https://search.yahoo.com" ,
    "https://amazon.fr" ,
    "https://yelp.com" ,
    "https://dreamstime.com" ,
    "https://abc.net.au" ,
    "https://cointernet.com.co" ,
    "https://walmart.com" ,
    "https://ssl-images-amazon.com" ,
    "https://bandcamp.com" ,
    "https://www.wix.com" ,
    "https://pl.wikipedia.org" ,
    "https://m.wikipedia.org" ,
    "https://outlook.com" ,
    "https://gnu.org" ,
    "https://yandex.com" ,
    "https://skype.com" ,
    "https://vkontakte.ru" ,
    "https://francetvinfo.fr" ,
    "https://tmz.com" ,
    "https://ria.ru" ,
    "https://oup.com" ,
    "https://guardian.co.uk" ,
    "https://pbs.org" ,
    "https://000webhost.com" ,
    "https://welt.de" ,
    "https://ietf.org" ,
    "https://disney.com" ,
    "https://eventbrite.com" ,
    "https://nginx.org" ,
    "https://hp.com" ,
    "https://secureserver.net" ,
    "https://wikihow.com" ,
    "https://stanford.edu" ,
    "https://liveinternet.ru" ,
    "https://smh.com.au" ,
    "https://rambler.ru" ,
    "https://photos1.blogger.com" ,
    "https://twitch.tv" ,
    "https://google.ru" ,
    "https://abcnews.go.com" ,
    "https://home.pl" ,
    "https://academia.edu" ,
    "https://naver.com" ,
    "https://sakura.ne.jp" ,
    "https://addthis.com" ,
    "https://android.com" ,
    "https://nbcnews.com" ,
    "https://zoom.us" ,
    "https://ebay.com" ,
    "https://surveymonkey.com" ,
    "https://webnode.page" ,
    "https://techcrunch.com" ,
    "https://buzzfeed.com" ,
    "https://britannica.com" ,
    "https://xing.com" ,
    "https://yahoo.co.jp" ,
    "https://ovhcloud.com" ,
    "https://doubleclick.net" ,
    "https://ipv4.google.com" ,
    "https://berkeley.edu" ,
    "https://statista.com" ,
    "https://newyorker.com" ,
    "https://wiktionary.org" ,
    "https://wiley.com" ,
    "https://espn.com" ,
    "https://npr.org" ,
    "https://kickstarter.com" ,
    "https://gmail.com" ,
    "https://thetimes.co.uk" ,
    "https://amzn.to" ,
    "https://unesco.org" ,
    "https://calendar.google.com" ,
    "https://nydailynews.com" ,
    "https://alicdn.com" ,
    "https://nginx.com" ,
    "https://20minutos.es" ,
    "https://cbc.ca" ,
    "https://spotify.com" ,
    "https://google.nl" ,
    "https://sky.com" ,
    "https://ibm.com" ,
    "https://psychologytoday.com" ,
    "https://plos.org" ,
    "https://amazonaws.com" ,
    "https://xbox.com" ,
    "https://icann.org" ,
    "https://rakuten.co.jp" ,
    "https://ziddu.com" ,
    "https://sedo.com" ,
    "https://amazon.it" ,
    "https://akamaihd.net" ,
    "https://urbandictionary.com" ,
    "https://insider.com" ,
    "https://newsweek.com" ,
    "https://bloglovin.com" ,
    "https://sputniknews.com" ,
    "https://whitehouse.gov" ,
    "https://www.gov.br" ,
    "https://www.wikipedia.org" ,
    "https://economist.com" ,
    "https://books.google.com" ,
    "https://sfgate.com" ,
    "https://akamaized.net" ,
    "https://tripadvisor.com" ,
    "https://leparisien.fr" ,
    "https://apache.org" ,
    "https://oracle.com" ,
    "https://id.wikipedia.org" ,
    "https://theatlantic.com" ,
    "https://privacyshield.gov" ,
    "https://ggpht.com" ,
    "https://ca.gov" ,
    "https://goodreads.com" ,
    "https://huawei.com" ,
    "https://windows.net" ,
    "https://detik.com" ,
    "https://hindustantimes.com" ,
    "https://target.com" ,
    "https://translate.google.com" ,
    "https://sciencedaily.com" ,
    "https://yandex.ru" ,
    "https://dw.com" ,
    "https://pexels.com" ,
    "https://latimes.com" ,
    "https://photobucket.com" ,
    "https://www.canalblog.com" ,
    "https://steampowered.com" ,
    "https://addtoany.com" ,
    "https://code.google.com" ,
    "https://pinterest.fr" ,
    "https://qq.com" ,
    "https://instructables.com" ,
    "https://ikea.com" ,
    "https://taringa.net" ,
    "https://ouest-france.fr" ,
    "https://nationalgeographic.com" ,
    "https://ea.com" ,
    "https://spiegel.de" ,
    "https://zendesk.com" ,
    "https://rtve.es" ,
    "https://cambridge.org" ,
    "https://www.livejournal.com" ,
    "https://springer.com" ,
    "https://groups.google.com" ,
    "https://nikkei.com" ,
    "https://hubspot.com" ,
    "https://unsplash.com" ,
    "https://prezi.com" ,
    "https://ovh.com" ,
    "https://cnil.fr" ,
    "https://rt.com" ,
    "https://quora.com" ,
    "https://gizmodo.com" ,
    "https://metro.co.uk" ,
    "https://weibo.com" ,
    "https://cnbc.com" ,
    "https://harvard.edu" ,
    "https://godaddy.com" ,
    "https://timeweb.ru" ,
    "https://nypost.com" ,
    "https://rapidshare.com" ,
    "https://googleblog.com" ,
    "https://mozilla.com" ,
    "https://cornell.edu" ,
    "https://gofundme.com" ,
    "https://cpanel.com" ,
    "https://sendspace.com" ,
    "https://e-monsite.com" ,
    "https://alexa.com" ,
    "https://finance.yahoo.com" ,
    "https://playstation.com" ,
    "https://mashable.com" ,
    "https://variety.com" ,
    "https://elmundo.es" ,
    "https://gooyaabitemplates.com" ,
    "https://vice.com" ,
    "https://redbull.com" ,
    "https://doi.org" ,
    "https://sapo.pt" ,
    "https://freepik.com" ,
    "https://lavanguardia.com" ,
    "https://zippyshare.com" ,
    "https://bp2.blogger.com" ,
    "https://clarin.com" ,
    "https://ted.com" ,
    "https://www.over-blog.com" ,
    "https://google.co.in" ,
    "https://google.ca" ,
    "https://theverge.com" ,
    "https://reg.ru" ,
    "https://netlify.app" ,
    "https://about.com" ,
    "https://linktr.ee" ,
    "https://alibaba.com" ,
    "https://video.google.com" ,
    "https://salesforce.com" ,
    "https://intel.com" ,
    "https://ads.google.com" ,
    "https://cbslocal.com" ,
    "https://corriere.it" ,
    "https://marketwatch.com" ,
    "https://weather.com" ,
    "https://chaturbate.com" ,
    "https://jhu.edu" ,
    "https://focus.de" ,
    "https://airbnb.com" ,
    "https://mysql.com" ,
    "https://excite.co.jp" ,
    "https://over-blog-kiwi.com" ,
    "https://google.com.tw" ,
    "https://g.co" ,
    "https://oecd.org" ,
    "https://stuff.co.nz" ,
    "https://bustle.com" ,
    "https://softonic.com" ,
    "https://messenger.com" ,
    "https://orange.fr" ,
    "https://reverbnation.com" ,
    "https://google.co.th" ,
    "https://kakao.com" ,
    "https://usgs.gov" ,
    "https://public-api.wordpress.com" ,
    "https://waze.com" ,
    "https://noaa.gov" ,
    "https://amazon.in" ,
    "https://pnas.org" ,
    "https://goo.ne.jp" ,
    "https://amazon.ca" ,
    "https://archives.gov" ,
    "https://teamviewer.com" ,
    "https://rottentomatoes.com" ,
    "https://debian.org" ,
    "https://prnewswire.com" ,
    "https://thefreedictionary.com" ,
    "https://fifa.com" ,
    "https://giphy.com" ,
    "https://scholastic.com" ,
    "https://narod.ru" ,
    "https://java.com" ,
    "https://ovh.co.uk" ,
    "https://theconversation.com" ,
    "https://mailchimp.com" ,
    "https://feedproxy.google.com" ,
    "https://howstuffworks.com" ,
    "https://bestfreecams.club" ,
    "https://admin.ch" ,
    "https://washington.edu" ,
    "https://espn.go.com" ,
    "https://com.com" ,
    "https://pcmag.com" ,
    "https://ubuntu.com" ,
    "https://deloitte.com" ,
    "https://parallels.com" ,
    "https://usc.edu" ,
    "https://jstor.org" ,
    "https://history.com" ,
    "https://scientificamerican.com" ,
    "https://psu.edu" ,
    "https://gfycat.com" ,
    "https://patreon.com" ,
    "https://fortune.com" ,
    "https://pewresearch.org" ,
    "https://bigcommerce.com" ,
    "https://fastcompany.com" ,
    "https://justgiving.com" ,
    "https://daaz.com" ,
    "https://usda.gov" ,
    "https://chicagotribune.com" ,
    "https://umich.edu" ,
    "https://rollingstone.com" ,
    "https://inc.com" ,
    "https://merriam-webster.com" ,
    "https://lexpress.fr" ,
    "https://techradar.com" ,
    "https://asus.com" ,
    "https://ibb.co" ,
    "https://uber.com" ,
    "https://channel4.com" ,
    "https://woocommerce.com" ,
    "https://my.yahoo.com" ,
    "https://lin.ee" ,
    "https://bp0.blogger.com" ,
    "https://nl.wikipedia.org" ,
    "https://main.jp" ,
    "https://canva.com" ,
    "https://esa.int" ,
    "https://unam.mx" ,
    "https://theglobeandmail.com" ,
    "https://xrea.com" ,
    "https://dailystar.co.uk"
    ]

if __name__ == '__main__':

    workers = [ Worker( name = f'worker-{n}' ) for n in range( 10 ) ]
    sites = USELESS_SITES
    
    for w in workers:
        w.start()
    from itertools import cycle
    
    
    sites = cycle( sites )
    print( 'ALL RUNNING' )
    
    for w in workers:
        [ w.queue.put_nowait( { 'action': 'get' , 'url': next( sites ) } ) for _ in range( 100 ) ]
    
    for w in workers:
        print( w.queue.qsize() )
    
    for w in workers:
        w.join()
    print( 'all stopped' )

[ColeMaverick]

Hi, by saying "you should have at least 1 undetected_chromedriver(.exe) in your appdata/roaming/undetected_chromedrive", does this means i need a undetected_chromedriver(.exe) in my .local/share/undetected_chromedriver/undetected_chromedriver path? Also, now that you pushed, we no longer need to specify the driver_executable_path=driver_path when we initialize the driver anymore right? Do we have to add in user_multi_procs = True though? Thank you!!

[ultrafunkamsterdam]

Exactly. A bit cumbersome but it is either this or N downloads and unpacking and crashing. (Where N is your number of processes)

[ultrafunkamsterdam]

user_multi_procs=True also ensures the driver won't be deleted

[ultrafunkamsterdam]

You should not do multiple driver=uc.Chrome() since then you will lose references. However you can make a list of drivers for example.
Sure driver will quit but wont be deleted. Which is normally the case. So you have to keep an eye on version changes andsoforth

[ColeMaverick]

Thank you! But I meant to say that I do multiple driver=uc.Chrome() as responses to multiple requests from multiple users at the same time to a page on my website (Process A). Your new undetected chromedriver version 3.4.7 and your explanation of "driver will quit but won't be deleted" means that Process A won't have a build-up memory use from multiple drivers being created and then not deleted right? I assume memory will only be build up when multiple drivers is running, and memory will goes down once they are quit?

[ultrafunkamsterdam]

Usage is per instance . When you close 1 less memory is used. The same driver still in use for your other threads. If you have a code sample i might better understand

[ultrafunkamsterdam]

Besides. Browser used a 100-fold of the memory so wouldn't be concerned.

[ColeMaverick]

Great thank you! I think I understand now. Thank you for creating such an awesome web driver!