Proxies

[Pingo-v0]

Looks great ! (seems to work fine while many other solutions had failed -thanks)

Would there be a way to set the proxies when creating the driver instance ?

[kaliiiiiiiiii]

Thank you!

Adding proxies should be possible as following:

from selenium_profiles.driver import driver as mydriver
from selenium_profiles.profiles import profiles
from selenium.webdriver.common.by import By  # locate elements

mydriver = mydriver()
profile = profiles.Windows() # or .Android


PROXY = "23.23.23.23:3128" # IP:PORT or HOST:PORT
PROXY_METHOD = "socks5://" # optional, default is socks5

# add to profile
profile["options"]["browser"].update({"proxy":PROXY, "proxy_method":PROXY_METHOD})


driver = mydriver.start(profile, uc_driver=False) 

# get url
driver.get('https://browserleaks.com/client-hints')  # test client hints

input("Press ENTER to exit: ")
driver.quit()  # Execute on the End!

For other options in the profile, have a look at Example-Profile

Didn't test it yet tho, but in theory it should work like that:) Feel free to tell me if it works as intended.

[Pingo-v0]

(That was a silly question..)

Thanks for all the answers !

[Pingo-v0]

In case you see what's wrong (don't hesitate to stop my questions, which may go on for a while if you don't since I'm trying to make it work not knowing how much work is remaining)

1. self.options.options.add_extension("full_path_to_extension.zip") returned the following error :
   AttributeError: 'Options' object has no attribute 'options'
   => I assumed you meant self.options.add_extension("full_path_to_extension.zip")

2. Then a new error popped up :

driver = mydriver.start(prof, uc_driver=False, proxy=proxy)  # or .Android
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium_profiles/driver.py", line 174, in start
self.driver = webdriver.Chrome(options=self.options)  # start selenium webdriver
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
super().__init__(
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/chromium/webdriver.py", line 106, in __init__
super().__init__(
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 288, in __init__
self.start_session(capabilities, browser_profile)
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 381, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 444, in execute
self.error_handler.check_response(response)
File "/home/zephyr/Git/api_token/.venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 249, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://jhdijllimahpfmeiniakbilgljppfmlp/_generated_background_page.html
from unknown error: page could not be found: chrome-extension://jhdijllimahpfmeiniakbilgljppfmlp/_generated_background_page.html

Any idea ?

PS : this guy seems to have had the same issue
https://www.solveforum.com/forums/threads/solved-python-selenium-headless-mode-not-working-with-proxy.2108486/

[kaliiiiiiiiii]

Oh yes, exactly, sorry that was a typo.
Soo I think the extension got loaded, but then produced en error. Where do you specify that chome should headless? In the profile? Is it even headless?

If you specify it using '--headless' as an argument, you might try out using '--headless=chrome' or without headless.

Oh and btw you can ask me as much as you want. I'l just aswer when I do have time anyways. I mean you help me to develop & test

[Pingo-v0]

Here is what I did :

prof = profiles.Windows()
prof['browser']['headless']=True

(without the proxy, it does work)

[kaliiiiiiiiii]

Good, that should automatically support extensions in headless

[kaliiiiiiiiii]

Proxy-authentication

see documentation

EDIT

• temp_dir as argument added @Pingo-v0 Version >= 2.2.5.4)
• new syntax for starting driver and profile >= 2.2.6
• new syntax & dynamic proxies supported >= 2.2.7

[kaliiiiiiiiii]

@PAJeantaud Please tell me, if that works in your case.

[Pingo-v0]

Hi @kaliiiiiiiiii
I had not seen your last post, thanks a lot for the changes made ! -looks promising -fingers crossed. I'll test it tomorrow (day spent on front-end dev issues..) and will of course let you know the results (this topic being crucial to me, no chance I'll give up before having tried as hard as possible)

[oldwizard1010]

(Hi from stackoverflow)
It says, selenium-profiles doesn't support linux yet, if it's true, i'm afraid i can't use it(
But i already looked here and probably will try it soon, so anyway, thanks)

[kaliiiiiiiiii]

Oh just not officially. If you install Chrome and chromedriver in /bin, it should work I think.
Feel free to open a issue In case it doesn't work//error-message.

If you are able to run Selenium, then Selenium-profiles should also work.
What code did you use?

[Pingo-v0]

Back(end) in the game :

1. Minor point :
profiles.Windows()['browser']['headless']=True is no longer working, to be replaced by the following ?
profiles.Windows()['options']['browser']['headless']=True

2. Proxy auth doesn't work so far (I may have done somthing wrong, I'm looking more closely at it) since
driver.get("http://lumtest.com/myip.json") returns the same value at each iteration... And i've got the following error :

/home/zephyr/Git/webscrapping_test/.venv/lib/python3.8/site-packages/selenium_profiles/scripts/cdp_tools.py:19: UserWarning: disabling emulation not supported
  warnings.warn('disabling emulation not supported')

[Pingo-v0]

Sure, I'm going to dig this issue deeper,
Anyway thanks for the great reactivity (and this library which still solves a point any other I had found would not)

[kaliiiiiiiiii]

Actually, maybe there's another way using (desired-capabilities](https://chromedriver.chromium.org/capabilities#h.p_ID_52) argument for chrome. But no idea if it supports authentification.
See https://www.w3.org/TR/webdriver/#proxy

[Pingo-v0]

Hi @kaliiiiiiiiii ,
Two more questions about the library:

• does it support POST method ? driver.post(data=json.dumps({...})
• does it handle jvscrpt chllenges ?
  

[Pingo-v0]

[kaliiiiiiiiii]

Well for post methods, I see 2 options:

• using javascript (see https://stackoverflow.com/a/65309073/20443541)
• using request interception ( I can give you a sample script in case you need it)

For javascript challanges:

• tbh I never really worked with javascript, so not shure what you mean. something like reCaptcha? In that case, ir really depends what type of captcha, so you might provide me with an example//or website? I do have approaches to solve them, but none yet implemented.
• do you mean cloudfare-undetectability? Like how to pass the cloudfare waiting-room?

Note: Maybe move to a new discussion?

[kaliiiiiiiiii]

Closing as it seems to be resolved