Today bet365.com has completely blocked access.

[joaosppy]

The biggest betting site in the world blocked access to Undetected-Chromedriver today, you can enter the home page but are prevented from accessing any event page that contains odds.

[ultrafunkamsterdam]

Not our problem

[ovciina]

I agree! Because if they have found a way to block its just a matter of time before others find the same way and undetected wont work...

Anyway, @ultrafunkamsterdam do you have a tip or suggestion we could try ourselves? This happend for me already on friday but I used version 3.2.1 and it worked, I dont know if you/someone else changed something in previous versions or if bet365 blocked something from the previous versions now also.. It's kind of strange that the new release of UC didnt work on friday but the older did, then now on Monday no version works anymore..

[peppelongo96]

@ovciina i had the same issue but after a while all went back normally also with last version. I think that was a test for the patch full applied today

[puppetrx]

@peppelongo96 can you share the version of the library (which I assume is the latest) and the Chrome version? And if possible how do you import and open the driver?
Perhaps it is because I'm working with spanish web, but it keeps detecting me.

[peppelongo96]

@puppetrx no mate, I mean that I'm still detected :( sorry I said that badly

[mmbertasi]

bet365 is pretty much a bellwether; as others said it just means more will come

[peppelongo96]

@joaosppy did you find something to solve?

[joaosppy]

I didn't get anything yet

[peppetemp]

@mmbertasi check your skype

[ghost]

@peppetemp Can you message me on skype as well I'm interested in whatever you may have live:.cid.238df685e9155833

[peppetemp]

@caph99 i did

[ovciina]

@peppetemp

Did you find a solution you could share with us, would be very kind as it's blocked now again..

[strukiii]

Are you running a proxy? Headless? Does Bet365 make you login to view updated odds? You haven't given any info here.

[ovciina]

Proxy doesn't help. Headless doesn't help. You don't need to login to view odds only to play/place a bet but from their last updates the odds aren't visible when accessing with UC/Chromedriver. It's "hidden/blank page" where odds/data are supposed to be seen.

[rafaeltgt]

Hello, bet365 is blocking access today again. Did you figure it out any way to bypass this detection?

Even in regular browser if I open developer tools the website is blocking the content.

[mmbertasi]

Yea I think this is how they are blocking it as its active by default in selenium and i'm not sure if there is a way to suppress it temporarily

[mmbertasi]

I had it working but as others said this morning they are back to detecting. I think they are detecting the CDP Runtime active in selenium. Is there a way to hide/suppress this temporarily

[mmbertasi]

think undetected chromedriver is already doing that removing / changing that cdc section

[strukiii]

theres a big thread in issues I believe give it a search. A pull request was merged in because of it?

[strukiii]

Can try this out this is what I used before it got merged

def removeCDC(driver):
    cdc_props = driver.execute_script(
        'const j=[];for(const p in window){'
        'if(/^[a-z]{3}_[a-zA-Z0-9]{22}_.*/i.test(p)){'
        'j.push(p);delete window[p];}}return j;'
    )
    if len(cdc_props) > 0:
        cdc_props_js_array = '[' + \
            ','.join('"' + p + '"' for p in cdc_props) + ']'
        driver.execute_cdp_cmd(
            'Page.addScriptToEvaluateOnNewDocument',
            {
                'source': cdc_props_js_array + '.forEach(k=>delete window[k]);'
            }
        )

[mmbertasi]

tried that it's not it

[peppelongo96]

I found that site finally load if you go to the page manually in the chrome instance opened from chromedriver. Why this? i thought that CDP was intended opened for the chromedriver session overall

[peppelongo96]

@ultrafunkamsterdam I know that is not a proper problem for our tool but please try to investigate when you have time. It will be more appreciated. Thanks

[igor-stojanoski]

I had the same problem when trying to scrape odds and after hours of trying I just found a way to not be detected by bet365.com
Instead of driver.get("link_to_odds") I'm opening a new tab with some website, delay a few seconds and open another tab with the 'link_to_odds'.
Here is the simple solution:

driver = uc.Chrome()

driver.execute_script("window.open('https://www.google.com')")
time.sleep(5)
driver.execute_script("window.open('link_to_odds');")
time.sleep(5)
driver.switch_to.window(driver.window_handles[1])
#The rest of the code...

I didn't have time to modify the code, it's just a simple way I found it working and I hope it'll help others. If someone can explain why this solution is working it will be very appreciated.

[joaosppy]

Thanks mate, but still can't click on closed markets, where the odds are 'hidden' odds. Data does not load.

[peppelongo96]

Thanks mate, but still can't click on closed markets, where the odds are 'hidden' odds. Data does not load.

Just click and get url. So load a new page with that link.

[joaosppy]

close market https://ibb.co/xjt7LLh
open market https://ibb.co/NK0vC7R

[peppelongo96]

close market https://ibb.co/xjt7LLh open market https://ibb.co/NK0vC7R

yes you are right. You can't execute any action staying on the same page

[igor-stojanoski]

Thanks mate, but still can't click on closed markets, where the odds are 'hidden' odds. Data does not load.

What browser are you using?

I'm running:
Ubuntu 22.04
Google Chrome 111.0.5563.64
undetected-chromedriver 3.4.6
Python 3.10.6

[mmbertasi]

Anyone find a way to deal with this other than the extra window workaround?

[ultrafunkamsterdam]

Bet365 scraping....uhuh

[ovciina]

Haha, "undeteced chromedriver 1337" is that bet365 that is logging it to "mock" or is that coming from UC. Never seen it before but if it's bet365 then they surley are spectating this discussion and we have their attention :D

[mmbertasi]

Someone over there was put in charge of this and 1000% they are reading whatever we put here; or at least, next they are told to block it they will come here. In the past once I got around their changes i could bypass block freely for a year or so.

[bigredcardinal]

I see the same thing in console, when visiting any other site. @mmbertasi if you find any way past the block, I am very interested.

[ovciina]

@ultrafunkamsterdam Have you tried any workaround to "hide" CDC from Bet365 or other sites that can recognize its beeing used?
I dont want to ask you if you dont care about Bet365 to help us solve/fix it, I respect that but can you please at least give us a hint what we can "read up" or do something ourselves.

I asked this already 40 days ago in the first comment but you didn't respond. I don't know if you missed my comment but like I said, I dont ask you more than to "guide" me in a direction that you think can solve it and I'll do the hard work myself, or at least try.

Been using UC last 3 years on a lot of sites, one of them as you probably understand Bet365 with no issues (or some small but solved them myself), until now, this is a major one.

[mmbertasi]

It's a fun challenge for sure to anyone up to the task. These betting sites have a lot of financial incentive to prevent their data from being harvested.

[mmbertasi]

Anyone figure this out?

[bigredcardinal]

Yeah, I got something that still works. Send me a message

[ovciina]

@bigredcardinal how do i message you? Could you message me by any chance?

[Shadownin2]

Can someone share with me as well please?

Does a screen recording data capturing program not exist yet? T_T

One that could detect data and copy to excel

[igor-stojanoski]

Yeah, I got something that still works. Send me a message

Why don't you share your way here?

[icinemagr]

How is possible bet365 to detect Console Open Even if is floating?
Try to connect to bet365 with console open and you are not getting any odds and not connecting to Live.

[b-nnett]

Fairly easy, there's multiple approaches - simplest would just be monitoring the devtools opened event listener and block scripts executing.

[ovciina]

Could you give an example how you mean by doing this?
Have you tried to lets say go to 5 different urls with one session on bet365 or you just assume it is fairly easy.

I can tell you I haven't got it to work at least so please help if you can.

[icinemagr]

All the approaches they do not work if devtools are open in float mode in seperate window.
so i be happy to know how he is doing this.

[CarachinoAlessio]

[mmbertasi]

just workarounds documented above; would love a better solution

[CarachinoAlessio]

You mean the workaround related to open it in a new window?

[ovciina]

That workaround doesnt work for me when I got to another URL on Bet365.
Lets say I first go to one page, then it works then I use driver.get("next_url") then it doesnt work, it's gletting blocked.

Any solution to move around in same session without restarting for each url if you have for e.g 20 urls to check?

[b-nnett]

@mmbertasi Able to run me through your use-case just so I can see where things are going wrong? Haven't used B365 before.

[mmbertasi]

Currently open browser with selenium - open new window via javascript - then connect selenium to new window this gets me access to lines/odds but only whatever you can access without actually refreshing or performing a full form submit

[ultrafunkamsterdam]

It is not accessible by websites. Even if they overwrite console.log. it runs in a different context. And yes there is a trick to check if devtools is open and it's fairly simple so recommend to keep it closed.

[kaliiiiiiiiii]

It is not accessible by websites. Even if they overwrite console.log. it runs in a different context. And yes there is a trick to check if devtools is open and it's fairly simple so recommend to keep it closed.

Seems like Chromedriver//CDP triggers in the same way, as if devtools would be open (test with devtools-detector using a selenium-controlled browser).

• doesn't apply to new tabs
• get triggered as soon the driver switches to the tab
• likely relying on Chrome source-code //CDP stuff, and not chromedriver ?

@ultrafunkamsterdam Did you come up with any way to overwite those javascript-objects to prevent that? Possibly using a Proxy ?

[icinemagr]

i suppose all of you are working to bet365 as bettesters or ethical hackers otherwise i cannot explain the stupidity to try to find solutions in here and suggests ticks to by pass eh?????
Thats why dev of bet365 every 10 days is fixing bugs.
thanks to you all.

[b-nnett]

I can't tell if you're annoyed people are suggesting methods you're using and getting them fixed or you're against scraping.

Being annoyed about it on an open-source scraping forum is funny either way lol.

[icinemagr]

seems there is no so much scraping on bet365.So better change the name :)

[Shadownin2]

Where else can we go lol

…

________________________________ From: icinemagr ***@***.***> Sent: 12 April 2023 11:50 To: ultrafunkamsterdam/undetected-chromedriver ***@***.***> Cc: Shadownin2 ***@***.***>; Comment ***@***.***> Subject: Re: [ultrafunkamsterdam/undetected-chromedriver] Today bet365.com has completely blocked access. (Discussion #1078) i suppose all of you are working to bet365 as bettesters or ethical hackers otherwise i cannot explain the stupidity to try to find solutions in here and suggests ticks to by pass eh????? Thats why dev of bet365 every 10 days is fixing bugs. thanks to you all. — Reply to this email directly, view it on GitHub<#1078 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AW3KZHMBKHNS3IEH77EJUILXA2CILANCNFSM6AAAAAAVBZ7K7A>. You are receiving this because you commented.Message ID: ***@***.***>

[mmbertasi]

Anyone have any updates on how to deal with this besides workarounds or do we need a more fundamental update to undetected chromedriver? Is it even possible to not expose chrome dev-tools protocol to detection?

[kaliiiiiiiiii]

Of course it is possible. It's just code. Everything is possible.
Its more about how much work you want to invest. If you're lucky, you can just tamper some javascript. But that most likely's gonna be detected after some time.

The other option would probably be to recompille Chromium. But that's a lot of work:)

[ultrafunkamsterdam]

import undetected_chromedriver as uc
driver = uc.Chrome()
driver.execute_script("""setTimeout(() => window.location.href="https://www.bet365.com", 100)""");driver.service.stop(); driver.reconnect()

2023-04-25.08-26-42-08_1.mp4

[ovciina]

I just tried it out and I dont know if its just for me but the site loades as in your video but if I manually press on any sport, lets say Soccer/Football on left side menu on the site it doesn't load properly. It detected chromedriver is there.

I get a message that there are no markets in this category but I know there are.

I dont need it to click around the page but I'd like to access for e.g a sport.
Lets say Champions League for example with this URL:

https://www.bet365.com/#/AC/B1/C1/D1002/E79147586/G40/

If I access it with your code it works as "first page" but if I for example write driver.get or driver.execute_script for another url it doesnt work. Lets say for this.

https://www.bet365.com/#/AC/B1/C1/D1002/E86295012/G40/

My code sample just to show what I mean:

 driver = uc.Chrome()
    driver.execute_script("""setTimeout(() => window.location.href="https://www.bet365.com/#/AC/B1/C1/D1002/E79147586/G40/", 100)""")
    driver.service.stop()
    driver.reconnect()
    time.sleep(3)
    driver.execute_script("""setTimeout(() => window.location.href="https://www.bet365.com/#/AC/B1/C1/D1002/E86295012/G40/", 100)""")

[githubinrn]

NOTE: @kaliiiiiiiiiii already mentioned the same thing in this thread, so i'm not saying anything new.

From what I have noticed it has something to do with having Google devtools open. When you first load up the page, the webpage thinks that you do not have Google devtools opened (this can be checked by using the devtools-detector that was mentioned in this thread before, link to website: https://blog.aepkill.com/demos/devtools-detector/)

BUT, when you execute a script to go to another link, the webpage thinks that you have Google devtools opened.

And if you check the bet365 website, it seems like when you use normal selenium, Bet365 acts like you have Google devtools opened. (In both instances the club names and matchup odds are blurred out, but the side bar is still there).

So it could be something that you can fix by somehow hiding devtools/CDP in selenium browser? Even though I haven't found a way to do it

[kaliiiiiiiiii]

So it could be something that you can fix by somehow hiding devtools/CDP in selenium browser? Even though I haven't found a way to do it

@githubinrn

Yeah, thought of the same approach. devtools-detector seems to detect how long the execution of window.console.log("some_big_object") takes, so I've come up with the following script to evaluate on each new page:

patcher = {}
patcher.proxies = new Set()
patcher.window_keys = ['debug','dir', 'table', 'clear', 'profile', 'profileEnd']
patcher.console_keys = ["clear",'debug', 'error', 'info', 'log', 'warn', 'dir', 'dirxml', 'table', 'trace', 'group', 'groupCollapsed', 'groupEnd', 'count', 'countReset', 'assert', 'profile', 'profileEnd', 'time', 'timeLog', 'timeEnd', 'timeStamp', 'context', 'createTask']

patcher.console_handler = {
    get(target, prop, receiver){
        if (patcher.console_keys.includes(prop)){
            var func = Reflect.get(...arguments);
            var proxied_func = new Proxy(func, {
                               apply: function (target, thisArg, args){return}
                            });
            patcher.proxies.add(proxied_func)
            return proxied_func
           }
        return Reflect.get(target, prop, arguments);
    }
}

patcher.toStringProxy = new Proxy(Function.prototype.toString, {
	apply: function (target, thisArg, args) {
		if (patcher.proxies.has(thisArg)) {
			return 'function '+thisArg.name+'() { [native code] }';
		}
		return target.call(thisArg);
	}
});

patcher.patch_window = function(){
    patcher.window_keys.forEach(function(key, value){
            window[key] = console[key];
            patcher.proxies.add(window[key])
        });
}


patcher.patch = function(){
    patcher.patch_window()
    window.console = new Proxy(window.console, patcher.console_handler);
    Function.prototype.toString = patcher.toStringProxy;
    patcher.proxies.add(Function.prototype.toString);
    patcher.patch_window()
}

patcher.patch()

It basically replaces the console functions with an empty function, and mocks function.toString to hide that it got changed (there are still inconsistencies bc of overwriting, test with creep-js

Got it to work with devtools-detector, but bet365 didn't.

@ultrafunkamsterdam Any idea what I might be missing here? Maybe id detects that the actual value to log never gets called?

[githubinrn]

For now if anyone is concerned about speed, I suggest using multithreading.
Then you can open multiple tabs at the same time and bet365 won't detect it.

I know that this might not be the most efficient way, but most people with a decent cpu could just multithread it.
I've made a demo code that gets the team names from 8 different leagues in bet365 at the same time, and it runs in about 9 seconds.

before you just run the code, please read:
for the multithreading to work, you need to fix an issue with undetected-chromedriver, I used Tomslarkins fix that he mentioned in this thread: #1051

The multithreading technique to get return values is not my own, I used a method that can be found in an old stackoverflow discussion through this link: https://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread.

The code:

from selenium import webdriver
import undetected_chromedriver as uc
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

from threading import Thread
import time

# all the links lead to some of the top soccer leagues
links_for_bet365 = [   
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76169570/F0/G40/",  #English premier league
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76509991/F0/G40/",  # Italy serie A
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76473591/F0/G40/",
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76468858/F0/G40/",
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76469135/F0/G40/",
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76947770/F0/G40/",
    "https://www.bet365.com/#/AC/B1/C1/D1002/E76460147/F0/G40/",
    "https://www.bet365.com/#/AC/B1/C1/D1002/E83381592/F0/G40/",
]

start_time = time.time()

number_of_tabs = 8
threads = [None] * number_of_tabs  #8
results = [None] * number_of_tabs

def get_the_bet365_element(links, result, an_index) -> None:
        prefs = {"profile.managed_default_content_settings.images": 2}  #disable images on the website

        options = webdriver.ChromeOptions()
        options.add_argument("--disable-blink-features=AutomationControlled")

        driver = uc.Chrome(options=options, prefs=prefs)
        
        driver.execute_script(f"""setTimeout(() => window.location.href="{links}", 100)""")
        driver.service.stop()
        driver.reconnect()

        teams_string = "//*[@class='rcl-ParticipantFixtureDetails_TeamNames']" # you can use any XPATH element to search, I parsed all the team names on the site.
        team_names_check = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, teams_string)))  #waits until elements is present
        team_elems = driver.find_elements(By.XPATH, value=teams_string)  # get the element


        team_names_list = []

        for a_team in team_elems:  #because the team names are both in one string per matchup, I did this simple fix to add the team names separately
            a_team2 = a_team.text.split("\n")
            team_names_list.append(a_team2[0])
            team_names_list.append(a_team2[1])

    

        result[an_index] = team_names_list  # the <result> list will contain the data from every thread.


y = 0
for a_link in links_for_bet365[0:number_of_tabs]:
    threads[y] = Thread(target=get_the_bet365_element, args=(a_link, results, y))
    threads[y].start()
    y += 1

for y in range(len(threads)):
            threads[y].join()

print(results)
print(f"it took: {time.time()-start_time} seconds to run the code")

runtime for me is around 9 seconds, it might be higher for others. But hey it's selenium, ultra-fast code is not the biggest priority.

[githubinrn]

@kaliiiiiiiiii its also probable that bet365 uses more devtools detection methods than just the standard methods, which are in the public devtools-detector GitHub available for anyone.

[mmbertasi]

So is the latest accepted solution @ultrafunkamsterdam timeout method or extra window method? Or is there something more solid?

[ultrafunkamsterdam]

It might not help you just now.: that website has the strictest of strictest policies. Even in normal situation it may not even work. Of course, they need to hide from both bad actors -and- authorities. However, A solution is coming up quite soon.

[mmbertasi]

@ultrafunkamsterdam I've been using the extra window method for a while with pretty good success just looking to optimize where possible. Sidenote I appreciate everything you are doing here and use it in my work on a daily basis

[kaliiiiiiiiii]

@mmbertasi maybe have a look at kaliiiiiiiiii/Selenium-Profiles#57 (comment)

[mmbertasi]

I will check that out thank you!

[ArsendiMattia]

Is something coming? I am asking on behalf of a friend! ;)