CesiumJS at least partially broken in latest Chrome on Linux

[mramato]

Sancastle seems to not work locally and hard locks the tab. But sometimes it works. Same for Cesium Viewer and anything using unbuilt Cesium. By running http://localhost:8080/Apps/Sandcastle/gallery/Hello%20World.html directly, I uncovered what I think it shte problem, shader compilation. Still looking further into it CC @lilleyse

[mramato]

So the entire shader problem may be a red herring and just a surface level problem caused by a deeper issue with this release of Chrome. Sometimes it works, sometimes it hard locks. Enabling shader validation provides no useful insight because I can't get the above to happen again (but it still hard locks)

[mramato]

And sometimes it all "just works". So I have no idea what the root may be here.

[mramato]

Firefox does not seem to have an issue, but I do see these warnings in the console:

The problem here is that it stops reporting warnings after the first 32, so even if the above is unrelated, it makes it hard to get useful information out of FF debugging here. Have we ever looked into these?

[lilleyse]

CC #8360 for the firefox warnings

[mramato]

New random WebGL error:

I'm starting to think that Chrome broke something deep down.

[mramato]

Text version for easier reading.

DeveloperError: Framebuffer is not complete.  Unsupported: the combination of internal formats of the attached images violates an implementation-dependent set of restrictions.

Error
    at new DeveloperError (http://localhost:8080/Source/Core/DeveloperError.js:39:11)
    at validateFramebuffer (http://localhost:8080/Source/Renderer/Context.js:994:13)
    at bindFramebuffer (http://localhost:8080/Source/Renderer/Context.js:1028:7)
    at Context.clear (http://localhost:8080/Source/Renderer/Context.js:1088:3)
    at ClearCommand.execute (http://localhost:8080/Source/Renderer/ClearCommand.js:100:11)
    at GlobeDepth.clear (http://localhost:8080/Source/Scene/GlobeDepth.js:554:11)
    at updateAndClearFramebuffers (http://localhost:8080/Source/Scene/Scene.js:3429:21)
    at executeCommandsInViewport (http://localhost:8080/Source/Scene/Scene.js:3156:7)
    at Scene.updateAndExecuteCommands (http://localhost:8080/Source/Scene/Scene.js:2892:5)
    at render (http://localhost:8080/Source/Scene/Scene.js:3768:9)

[mramato]

So if I use Hello World with combined Cesium, it never crashes, but I do get crazy imagery artifacts like this:

[mramato]

Okay I'm convinced that this is not a WebGL issue.

• Switched to Intel: No Change
• Set up Context to use getWebGlStub: No Change

The browser appears to just "stop". Network requests are stuck in pending, the page is totally non-responsive, can't break in the debugger.

[mramato]

As a test, I updated the CesiumViewer.js sample app to be 3 lines long:

window.CESIUM_BASE_URL = "../../Source/";
import { Viewer } from "../../Source/Cesium.js";
new Viewer("cesiumContainer");

Hangs (almost) every time.

Changing it to use CesiumWidget instead, works.

window.CESIUM_BASE_URL = "../../Source/";
import { CesiumWidget } from "../../Source/Cesium.js";
new CesiumWidget("cesiumContainer");

I had this crazy idea that ES module loading was causing problems and that we were simply overwhelming the browser with too many files, but the above loads the same exact set of JS, just only uses the widget instead of the viewer. This may still yet be a problem on our end.

[mramato]

I tried enabling/disabling various Viewer features one at a time to determine if something was triggering a bug (either in our code or Chrome) but it seems completely non-deterministic. On first load, Cesium Viewer loads fine and will continue to load fine if you just refresh the tab. If you start on a brand new tab or incognito mode, it will hang almost immediately..

Having the debugger up when you know it's going to hang does nothing, hitting the break button has no effect whatsoever,

At this point I 100% believe the problem is in Chrome on Linux, but I have no idea what a reduced test case looks like because it's non-deterministic.

[mramato]

I think we're out of options from our side and given this works everywhere by Chrome/Chromium on Linux, I'm going to open a Chromium issue to hopefully get some help.

[mramato]

I checked all the way back to 1.70 and it's still broken, so definitely not a regression anywhere near our end..

[mramato]

This gets worse. The same thing that crashes constantly locally for me works when on S3, this could point to a possible race condition related to the network stack? This is one of the most baffling CesiumJS issues I've ever tracked.

[mramato]

@kring pointed out that createTaskProcessorWorker.js seems to be the last file successfully loaded right before everything hangs. I've verified that this is the case, but it seems to be a red herring. I also looked at the very next file that is always loaded after it (approximateTerrainHeights.json) and that doesn't seem to matter either.

Making things go slower seems to prevent the crash, which is why I still think this is some sort of race condition in Chrome's network stack.

[mramato]

Another thing I noticed loading Cesium Viewer is that approximateTerrainHeights.json is the first item that the initiator is the VM and not an import from a JS file (i.e. I believe this is the first XHR request as opposed to an import directive)

[mramato]

Looks like this is actually working again.

[hpinkos]

Not for me. Just upgraded to Chrome Version 89.0.4389.114 and sandcastle still hangs

[hpinkos]

I can't run the unit tests at http://localhost:8080/Specs/SpecRunner.html either

[mramato]

That's what @kring and @lilleyse are still seeing as well. I have no idea what changed on my machine that it's suddenly working again.

[mramato]

UPDATE: I rebooted and it's broken on my machine again. I have no idea what anything means anymore.

[mramato]

@ebogo1 @lilleyse We really need to to get to the bottom of this issue.

[ebogo1]

@kring pointed out that createTaskProcessorWorker.js seems to be the last file successfully loaded right before everything hangs. I've verified that this is the case, but it seems to be a red herring. I also looked at the very next file that is always loaded after it (approximateTerrainHeights.json) and that doesn't seem to matter either.

As another data point, it seems random on my machine which file is the last one to get loaded before the next ones get stuck on "pending."

[ebogo1]

Changing it to use CesiumWidget instead, works.

On my end running the 3-line Viewer app with just CesiumWidget breaks in Chrome as well (but works in FF).

[ebogo1]

After some more testing, I think my verdict on this bug is that it is intermittent and related to Chrome's handling of "many" ES modules. This is based on these steps:

1. Modify gulpfile so that Cesium.js only exports KmlDataSource.
2. Modify CesiumViewer.js to import just KmlDataSource from "../../Source/Cesium.js".
3. Comment out the entirety of KmlDataSource.js and change the file to export an empty KmlDataSource function as default.
4. Manually add in random imports back at the top of KmlDataSource.js and see if the Chrome tab becomes unresponsive. This seemingly happens randomly - sometimes it takes 2 refreshes to freeze, sometimes it takes 6 or more. I tried a bunch of combinations of imports, some from Core, from ThirdParty, Scene, etc., and found no correlation between when Chrome starts freezing and what is being imported. The same set of imports can take any number of refreshes to break the Chrome tab.

As another note, the second step isn't strictly necessary. For example - I can also change the createCesiumJs function in gulpfile.cjs to put only KmlDataSource and Cartesian3 into Cesium.js, import only Cartesian3 in CesiumViewer.js, and modify the imports in KmlDataSource.js to cause the browser to freeze (or not, if I remove enough import statements) after a few refreshes.

[ebogo1]

I have submitted a Chromium bug report here.

[mramato]

FYI, with the help of the Chrome team, the issue has been narrowed down to a low default ulimit. On machines exhibiting the problem the ulimit is defaulted to 1024. Raising the limit works around the problem for the time being.

The Chrome team is still looking into the root cause on their end, but can now reliably reproduce it. Details in Eli's link, above.

[ggetz]

Closing as recent build updates have changed how we are loading modules in the browser. The linked issue in Chromium is still marked as a regression, but there's nothing actionable on our part.