WebDriverException: Message: unknown error: bad inspector message

[AxleMaxGit]

🐛 Bug Report

Problem is described in detail here https://stackoverflow.com/questions/56880164/selenium-parsing-unicode-problem-webdriverexception-message-unknown-error-ba?noredirect=1#comment100310339_56880164

Basically the problem is Selenium can't parse the HTML or text of the elements because the elements are invalid unicode characters.

I get the same error using any of: driver.page_source, element.text, element.get_attribute('innerHTML')

Error looks like

WebDriverException: Message: unknown error: bad inspector message: {"id":327,"result":{"result":{"type":"object","value":{"status":0,"value":"<a class="influencer-analytics-link" href="/influencers/lebelletour">

Belle Nog\uD83D\uDC83\uD83C\uDFFC\u2708\uFE0F\uD83C\uDDE7\uD83C\uDDF7\uD83C\uDDEA\uD83C\uDDF8\uD83C\uDDFA\uD83C...

"}}}}
(Session info: chrome=75.0.3770.100)
(Driver info: chromedriver=2.40.565386 (45a059dc425e08165f9a10324bd1380cc13ca363),platform=Mac OS X 10.14.0 x86_64)

The offending characters are somewhere around "\uD83C\uDDF7\uD83C"

You can see at https://ibb.co/HtkzzPM the screen shot where the unicode characters (emojis) aren't rendering. The "U" and the "?" are actually flag emojis.

To Reproduce

It's hard to tell you how to reproduce because the page is behind a paywall

[diemol]

Really hard to say where the problem is without a way to reproduce the issue. For now I can only see that the version of Chrome and ChromeDriver are out of sycn. Please update them and try again.

If the error persists, please provide a concise reproducible test case, that includes the page required to execute the scenario (a sample page or a public site).

[teledyn]

this page will reliably trigger the effect: https://www.cohensfashionoptical.com/locations/2-west-14th-st/ -- those Unicode characters are said to be "High Surrogates"

[diemol]

Would be great to actually get a piece of code that helps reproduce the issue instead of ranting comments. Selenium belongs to everyone, so we all need to help to diagnose and fix bugs, and the first step is to provide a way to reproduce it.

[xlacara]

any page with special characters .. just get element OuterHTML.. and you will get this error..

[barancev]

@xlacara Please attach an executable scenario instead of narrative descriptions.

[xlacara]

i really wish i could.. is happening on my icq when i have a lot of messages..
to recreate this situation.. i should make public my uin and password or .. create a new icq account and add 500+ contacts ... and this is really hard to do just for a scenario.. i didnt mean to make a narrative description ..but please understand the situation.. i will update if i will find out a solution.. thank you

[xlacara]

seems that weird emoji create an error when parsing the DOM.. some of my contacts use emoji as part of the username and this can cause the error

[deltagreen]

In theory, one way to overcome this, would be to read the html through a normal streamer, convert it to UTF-8, put it on disk, and then invoke Selenium but give it the location on disk instead of leaving it fetching over the net. While awaiting for a fix on this, that workaround could potentially help although I know that reading it from the disk instead has lots of potential negative effect in terms of DOM construction and interpretation since the origin server will have changed.

[AmineAgrane]

Getting this error while scraping users reviews of Android app. It depends of the app (its content) I'm trying to scrape. Sometimes I can scrape 8000+ reviews even with emojis etc, sometimes a single emoji can completely ruin the driver.page_source element. Any tips ?

[deltagreen]

I had to do this to overcome (selectively keeping characters below 20): private static String removeUnallowedCharacters(String content) { for (int charval = 0; charval <= 20; charval++) { if (charval == 0x9) continue; // tabs are allowed if (charval == 0xA) continue; // CR are allowed if (charval == 0xD) continue; // LF are allowed content = content.replace("" + (char)charval, ""); } content = content.replace("" + (char)0x1F, ""); return content; }

…

On Thu, Dec 26, 2019 at 9:22 AM Amine Agrane ***@***.***> wrote: Getting this error while scraping users reviews of Android app. It depends of the app (its content) I'm trying to scrape. Sometimes I can scrape 8000+ reviews even with emojis etc, sometimes a single emoji can completely ruin the driver.page_source element. Any tips ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7359?email_source=notifications&email_token=ACD5TURIVOJ7C6WUZF2D3NLQ2SVZ3A5CNFSM4H6FYB6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHVSELA#issuecomment-569057836>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACD5TUSDHYXYHUB4SCDJR43Q2SVZ3ANCNFSM4H6FYB6A> .

[diemol]

I will close this since we never got a way to reproduce it so we could validate if it was a bug or not.
If someone still has this potential issue, feel free to open a new issue and please provide a self-contained way to reproduce it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.