CefSharp3 crashes shortly after startup in some proxy use cases

[oxbough]

My organization is very interested in embedding CEF3 in our WPF app. I am starting investigation of CefSharp3. I used NuGet to populate my VS2010 solution with the latest CefSharp3 files and built a simple example. I had done the same for the latest CefSharp1. The CefSharp1 build works, but the CefSharp3 version crashes shortly after startup. So I downloaded the CefSharp-3.29.0-pre.0 source zip file and built the CefSharp.Wpf.Example executable from there. It also crashes shortly after startup. For this I see the rendering of the custom://cefsharp/home page for a few seconds and then it crashes. Using the VS2010 debugger on the dump file shows that there is null pointer exception in libcef.dll at 081470fd...

081470DB cmp eax,40h
081470DE jge 081470EA
081470E0 mov esi,dword ptr fs:[eax_4+0E10h]
081470E8 jmp 081470FD
081470EA mov eax,dword ptr fs:[00000F94h]
081470F0 mov ecx,dword ptr ds:[94BE1C0h]
081470F6 mov esi,dword ptr [eax+ecx_4-100h]
*** 081470FD cmp dword ptr [esi],1
08147100 je 08147155
08147102 cmp byte ptr ds:[94BE2B7h],0
08147109 jne 08147114
0814710B cmp byte ptr ds:[94BE2B6h],0
08147112 je 08147122

Registers:
EAX = 0000002F EBX = 09E21CC0 ECX = 00000000 EDX = 0BDDF7E0 ESI = 00000000 EDI = 09E21CC0 EIP = 081470FD ESP = 0BDDF6C8 EBP = 0BDDF6D4 EFL = 00010283

Why do I get this? Is there a solution available?
If I can't get a working version of CefSharp3 built by me under VS2010, I can't even start to integrate it into our project.

[oxbough]

I grabbed the correct libcef.dll debug build and symbols from the cefbuild downloads. The crash was the same but the crash data looked different, and pointed to an exception in platform-win32.cc. But I don't have that source file. I'll include more info a little later.

[perlun]

Try with this project: https://github.com/cefsharp/CefSharp.MinimalExample

It should be a well-working, simple example of how to get CefSharp3 running in your (WPF) app. Does it compile and run on your machine?

[jornh]

Hi @oxbough and welcome!

Argh! @perlun just beat me on the finish line here - still - here's the more elaborate post

Your persistence with getting to the bottom of this is appreciated. That's the kind of attitude we need around here 👍 The codebase have evolved some since 3.29 and IMHO have seen some good improvements regarding resource management etc. with a lot of the heavy lifting done by @JanEggers. We are working towards 3.31 (which will be named just 31.0.0 according to the venerable @perlun 😉).

So to save you some hassle with something that might end up being either due to:

• an error in CEF3 "29" since fixed in CEF3 "31" aka the CEF 1650 branch
• or, something that is already fixed in the CefSharp codebase in the master branch

I'd suggest you check if you still see this issue with either a CefSharp source build or through the

Compared to 3.29 the source build has changed to now pull in the upstream CEF code through NuGet since #288 got merged. This way you are able to compare apples to apples with CefSharp.Wpf.Example

There are also since today #137 (comment) the first functional (on my machine at least) NuGet packages which I'm really keen on getting feedback on A good testing vehicle for that is the https://github.com/cefsharp/CefSharp.MinimalExample repo.

[perlun]

Argh! @perlun just beat me on the finish line here - still - here's the more elaborate post

Hehe! 😉 I'm actually on the Heathrow Express at the moment, on my way back home from London. Too bad they have Internet on the trains these times, for you... 😀

[jornh]

hah! Here comes hi-speed @perlun .... BTW try looking at the change you just did with 31.0.0 ChangeLog (as per my suggestion) as someone not logged in to GitHub - not very informative!

[perlun]

That's a good point. How do we come around that?

[oxbough]

Thanks @perlun and @jornh for the suggestions. Are you saying it's possible for my to swap out CEF3 29 and swap in CEF3 31 with what I already have going? I've downloaded he MinimalExample zip file. I suppose I should go straight to the git repo. Kind of a git nubee.

[oxbough]

No problem getting the repos. But it's still failing.
Out of the box, https://github.com/cefsharp/CefSharp.MinimalExample repo has the same problem I had with my own use of CefSharp3. Nothing gets rendered in window, and it crashes seconds after it opens. NuGet actual gets the cefSharp 29 pre version. So I wasn't surprised it failed.
I grabbed the full CefSharp repo. NuGet auto downloaded CEF 3.1650.1562. Great! Compiled it all and still got exactly the same failure behaviour from CefSharp.Wpf.Example. Not great. The home.html page renders briefly and then it crashes. I'm using Windows 7 Professional, VS2010. I tried x86 first, then x64. Both give me the same result.

[jornh]

Huh, that's not intended behaviour at all! I'm on win7 + VS2010 too but with SP1.

Let's try isolating it to CEF alone or with CefSharp on top. Either build cefclient.exe from our cef-binary repo - or download it from the CEF download page.

[oxbough]

OK. I downloaded the cef-binary zip file for 3.1650.1562, opened the vs2010.sln file and built the cefclient.exe. That works. I used it's larger libcef.dll with the CefSharp.Wpf.Example.exe file I built and the wpf example still crashes like before.

[oxbough]

Oh, And while the wpf example still crashes, the winforms example does not crash, but it won't navigate. The navigation url window and the back and forward buttons exist, but the HTML rendering area remains blank.

[oxbough]

Also grabbed the cefsharp cef-binary repo and built it. Same results as the cef-binary zip file as above, as expected. cefclient works there.
Yes I have Win 7 Prof + SP1 and VS2010 + SP1

[jornh]

Yes the WinForms thing is in need of quite some love. If you apply jornh@e951c2f I hope - for consistency - you should have that chrashing too?

[jornh]

Here is a dropboxed MinimalExample x86 Release binary zip I built with VS2010 SP1 and this https://staging.nuget.org/packages/CefSharp.Wpf/3.31.0-pre1 fairly recent 3.31 staging NuGet - which runs just fine on my Lenovo T420 machine.

I guess that will also crash on your machine? But then we have ruled out your build environment as source of the problem. My understanding of the problem will then be: "cefclient.exe works in your environment, anything CefSharp3 on top of CEF3 crashes due to your hardware or general Windows configuration"..

If so, it will basically leave us back at square one - and next step is probably to continue from where you were heading with debugging the crash from the top of this issue - or what do you think? Or could you try on another machine?

Oh, one last thing: Does it get as far as it leaves something in the debug.log file? Maybe we can turn up the debug level there? I haven't tinkered too much with that yet - so I don't know exactly how...

[oxbough]

Thanks @jornh . Yes when I apply jornh/CefSharp@e951c2f the winforms example navigates to the page and crashes. I tried finding the line of code where the crash happens. The debugger told me line 5649 in api.cc, but I couldn't figure out which version of that file I needed to gab. Nothing seemed to make sense. I grabbed the version Nov 14 based on when I thought the particular version of CEF was built. And tried versions around that date. Nothing matched up with the disassembly. I will try the info in your second post.

[jornh]

OK, thanks for the update!

I haven't really seriously dug around up the CEF -> Chromium food chain before. But here is what I think (hope?) we need to follow:

1. We have pulled in the "official" 1650.1562 build - not from cefbuilds.com (nightlies) but http://www.magpcss.net/cef_downloads/
2. On the 1650 branch in CEF there is a https://code.google.com/p/chromiumembedded/source/browse/branches/1650/cef3/CHROMIUM_BUILD_COMPATIBILITY.txt which says http://src.chromium.org/svn/releases/31.0.1650.57
3. Which contains just one file http://src.chromium.org/svn/releases/31.0.1650.57/DEPS detailing what I guess is SVN revisions for the different parts of chromium used in the build

So, how/where did you dig up e.g. that api.cc file? And does it make more sense with line numbers if you base revision on my (admittedly, just a) guess on how to follow revisions?

[oxbough]

I pulled the symbol files from cefbuilds. In one instance I even replaced the libcef.dll with the debug version available. When it crashes, I collect the dmp file generated and load it into VS2010. That told me the exception happened at the specific location, ie. filename and line number. VS2010 tells me it not the matching source file, and I can see that it was about 200 lines off on the location. Right file, wrong version. It appears to be crashing in a call to an "External::new" method or nearby in the file. I think it's in the v8 javascript engine. The source for the file itself was obtainable in all revisions via google search.

[oxbough]

I ran the minimal example you provided. It crashed the same way, and the crash dump showed it happened at the same place.
0FAFE35A 57 push edi
0FAFE35B 83 F8 40 cmp eax,40h
0FAFE35E 7D 0A jge v8::External::New+1Ah (0FAFE36Ah)
0FAFE360 64 8B 34 85 10 0E 00 00 mov esi,dword ptr fs:[eax_4+0E10h]
0FAFE368 EB 13 jmp v8::External::New+2Dh (0FAFE37Dh)
0FAFE36A 64 A1 94 0F 00 00 mov eax,dword ptr fs:[00000F94h]
0FAFE370 8B 0D 74 34 4A 11 mov ecx,dword ptr [v8::internal::Isolate::isolate_key_ (114A3474h)]
0FAFE376 8B B4 88 00 FF FF FF mov esi,dword ptr [eax+ecx_4-100h]
*****>>>> 0FAFE37D 83 3E 01 cmp dword ptr [esi],1
0FAFE380 74 53 je v8::External::New+85h (0FAFE3D5h)
0FAFE382 80 BE B9 49 00 00 00 cmp byte ptr [esi+49B9h],0
0FAFE389 74 0E je v8::External::New+49h (0FAFE399h)
0FAFE38B BA 9C 71 0A 11 mov edx,offset string "v8::External::New()" (110A719Ch)
0FAFE390 E8 EB C9 FF FF call v8::ReportV8Dead (0FAFAD80h)
0FAFE395 84 C0 test al,al
0FAFE397 75 3C jne v8::External::New+85h (0FAFE3D5h)
0FAFE399 83 3E 01 cmp dword ptr [esi],1
0FAFE39C 74 37 je v8::External::New+85h (0FAFE3D5h)

[perlun]

I think the problem is that you lack the VS2012 C++ redistributable (Yes, 2012) on the machine in question. Since CefSharp is compiled with VS2012 (and will be for the near future at least), you must have VS2012 redist (either installed or as separate .dll files in your app's path). See the FAQ for more details.

[oxbough]

Thanks @perlun. You are probably correct.

[jornh]

Duh, I knew that already with VCRedist 2012 for the builds using the official CefSharp NuGets wrapping all the native/unmanaged code built on @perlun's VS2012 - so why did I not write it?

Still, how would that explain this crash with a "CefSharp from sources build"?

I grabbed the full CefSharp repo. NuGet auto downloaded CEF 3.1650.1562. Great! Compiled it all and still got exactly the same failure behaviour from CefSharp.Wpf.Example. Not great. The home.html page renders briefly and then it crashes. I'm using Windows 7 Professional, VS2010. I tried x86 first, then x64. Both give me the same result.

Hmm, here's hoping that at least installing VCRedist 2012 gets you running with MinimalExample ...

Maybe we should add runtime detection of the VCRedist version matching what CefSharp.Core.dll got compiled with? It should be the only remaining dependency in the FAQ not accounted for now that all the others should be handled by the cef.redist NuGet.

[oxbough]

I'm having trouble following what I need to do for the redist. I downloaded a VC2012 redist file (about 7 MB) and installed. I did this rather silently. Never created the directories noted in the FAQ. I found an msvcr110.dll in my system32 dir. But now what? I'm installing VS2012. Hopefully I get the files from that?

[jornh]

Download and install like you did should be all that is needed. The installer for it is quite silent...

After this you should be able to get MinimalExample running when building with VS2010

update the directory noted in the FAQ is where you copy it from (on a developer machine with VS2012) if you as a developer want to just copy the two DLLs as part of deploying your CefSharp based app to client machines. So it's the alternative to make sure VCRedist is deployed on very client.

[oxbough]

OK. Your minimalexample binary runs without crashing but displays only about:blank. I rebuilt the minimal example but that crashes. Could there be an overriding setting in my VS that changes the output even when using a pre-canned solution file?

[jornh]

Ok so it seems installing the VCRedist made a difference - at least to my binary drop even though it's not running as expected - any clues in it's debug.log file? Can you get it running on another machine?

The continued crashing with your VS2010 builds really baffles me. But as you say, it can be any little detail or setting. Again switch environment somehow - another machine - or as you suggest - try a fresh VS2012 install. It might flip something into the right position. This remote debugging really is frustrating. ... And I'm not even the one who has to look at all the crashes 😀 ...

[oxbough]

It seems it crashes when navigating. My build of cefsharp.wpf.example sat at about:blank without crashing. It let me type in google.com, and crashed a second after hitting "go". Thanks for all the suggestions. I am using a Lenovo T510. I did run a copy on a desktop yesterday, That crashed the same way. But of course, there was no VC2012 redist on that. But even the minimal example you built still crashes on my laptop.

[oxbough]

I did try the various executables on an HP desktop PC running Windows 7 Home Premium.
My first cefsharp3 simple app runs. The response to rendering a URL is slow. A few seconds delay. The cefsharp3 cefsharp.wpf.example app ran from my release build, but crashed with the debug build. I'll regenerate and try again. I may have messed up some files in the dir. Both of the PCs on which it doesn't run are work PCs with customized OS images. Could be something about that causing the issues.

[brock8503]

Hi @oxbough when you say crash do you have the debugger connecting to the project and can you get us some information about the crash? (Missing dependencies...etc)

I would suspect the debug Cefsharp.dll is failing to load the vs debug dependencies. If you want to double check go ahead and download http://technet.microsoft.com/en-us/sysinternals/bb545021.aspx
and run procmon to see if a file is missing!

[oxbough]

Thanks @brock8503 If I run in debug mode within VS, it crashes, but does not stop in the debugger. Just terminates as if not in the debugger. Perhaps I have some settings that are avoiding it ??? I will look at running procmon. Won't happen until Monday now. I also plan to create a VM on a lab PC and load a fresh OS to see if it behaves differently.

[oxbough]

Still failing. :( Created a Win7 Prof 32 bit VM, installed the VC2012 redist, applied DotNet 4.0, ran a release version of CefSharp.Wpf.Example. It crashed as usual.

[oxbough]

OK, I found out how to capture the exception in VS2010 ...

• go to Projects Properties
• click on the Debug tab and Enable Unmanaged Code Debugging

And I get the same info in debug mode as using the dump file. Line 5649 in api.cc. But still don't have the right version of api.cc

[jornh]

Created a Win7 Prof 32 bit VM, installed the VC2012 redist, applied DotNet 4.0, ran a release version of CefSharp.Wpf.Example. It crashed as usual.

That release version - did you build it yourself form the CefSharp sources?

Yes I have Win 7 Prof + SP1 and VS2010 + SP1

32 bit or 64 bit OS? Which VCRedist versions does it have if you look under "Add remove programs"?

I did try the various executables on an HP desktop PC running Windows 7 Home Premium.

32 bit or 64 bit OS? Which VCRedist x86 versions does it have?

Note: Just to clarify, whenever you use an "official" CefSharp.dll use VCRedist 2012 - when you use those you build from source with your VS2010 SP1 make sure the machine you run it on has VCRedist for VS2010 SP1. Maybe you are already fully aware but better be on the safe side

---

I just tried my dropbox .zip from above of MinimalExample on a VMWare Server 2008 R2 (yeah 64bit) which had only 2008 VCRedist to start with. It crashed on startup - as expected. I then installed http://www.microsoft.com/en-us/download/details.aspx?id=8328 and it ran as expected. I forgot to mention that in the "excitement" yesterday Reason: The obvious; I built it with VS2010 SP1 - not VS2012 as @perlun does for the official binaries.

I note one minor difference here. The crashes are different in that without the right VCRedist it never gets as far as launching any window (from what I have seen). What you experience is it crashes when it gets down to presumably allocate resources for something V8'ish ... in other words - "lets process on some JavaScript".

Also, does it get as far as it gives any clues in debug.log (in the bin/ folder of the executable)?

I could see tomorrow if I can get hold of a good ol' XP 32bit and see what happens when I throw some CefSharp3 at it.

[oxbough]

I have a 64 bit Win7 Prof. My builds have all been 32 bit. I did one build of x64, just to try that. Didn't work either. I only installed the x86 (32bit) 2012 redist - update 4 (version 11.0.61030). I did build the exe myself from the cefsharp3 sources. I'm concentrating my app starts on cefshart.wpf.example.exe built from the cefsharp repo, but my own simple app built using the simple app suggestion examples behaves the same way. I also installed the x64 2012 redist, and I downloaded x86 and x64 2010 redists and selected "repair" on start of these packages. cefsharp still failing.

Oh, the debug log has several entries like this ...
[0407/160345:INFO:dhcp_proxy_script_adapter_fetcher_win.cc(263)] Error fetching PAC URL from DHCP: 2

[oxbough]

Found the following post with what appears to be the same problem with CEF3 a year ago.
http://www.magpcss.org/ceforum/viewtopic.php?f=6&t=10570

[oxbough]

As suggested by the post, passing the command line parameter --no-proxy-server cures it from crashing. Why does this fix it? Why does it crash on some PCs and not others?
OK. It think the follow explains my last question... http://answers.oreilly.com/topic/675-how-to-configure-proxy-settings-in-windows-7/
It says that chrome uses system wide proxy settings. My work PC is configured to use an automatic configuration script. But surely the app should not crash because of this. And why does cefsharp crash, but not cefclient?
One last comment... my simple app takes a long time to navigate or render a page, such as google. Whereas cefsharp.wpf.example is quick. Any idea why?
Now that I can keep cefsharp running, I will start integrating it into my target app.

[jornh]

http://www.magpcss.org/ceforum/viewtopic.php?f=6&t=10570

hah - just found the same!

proxy settings - wow ... sound like you are onto something now 😄 ... about time we got closer to it ...

[jornh]

But surely the app should not crash because of this. And why does cefsharp crash, but not cefclient?

Right! It shouldn't.. Well AFAIK one difference is the Wpf example uses offscreen rendering. Cefclient doesn't. update not to say that it's directly related. Just to say that there are differences ...

my simple app takes a long time to navigate or render a page, such as google. Whereas cefsharp.wpf.example is quick. Any idea why?

Not really ATM - is both your simple app and cefsharp.wpf.example built from recent master source? How does it compare to my zip drop if MinimalExample (which is also ~ fairly recent master) - it should run now where we know it needs VCRedist 2010 and the --no-proxy-server

[oxbough]

My simple app is still using the downloaded cefsharp 29 library. I'd say your zip drop of minimalexample is also completing the page in a delayed manner. Some parts of the google page came up almost immediately, but the large colourful font Google name was really delayed the first time I launched it. Faster on subsequent launches. I'll move my new work over to the newer master.

[jornh]

And the CEF issue born out of above forum thread is still open - and confirmed by others with CEF 1650 ~ Chromium 31 https://code.google.com/p/chromiumembedded/issues/detail?id=927 so now we can adjust title and put a few flags on this issue

[oxbough]

CefSharp does not appear to have a way to programatically set no-proxy-server for CEF like it does for MultiThreadedMessageLoop=true.

[perlun]

CefSharp does not appear to have a way to programatically set no-proxy-server for CEF like it does for MultiThreadedMessageLoop=true.

That might be the case. Please investigate and see if you could add that as an option, and submit a pull request. We'll leave this case open in the meanwhile.

[brock8503]

@oxbough the proxy settings are required at Cef Init time so you will need to pass this parameter as args to the main program wrapping the cef initialize call.

[brock8503]

@oxbough @jornh our team actually was able to reproduce this and this workaround worked for us. http://www.magpcss.org/ceforum/viewtopic.php?f=6&t=11711

[jornh]

With #365 now merged the reason for keeping this issue open: #309 (comment) is gone. So let's close this one. The upstream issue isn't gone though. Hopefully it will some day when we update ...