Profile plugin, trace viewer, does not work in Chrome 80+

[stephanwlee]

Chrome 80+ deprecated two APIs: document.registerElement and Element.prototype.createShadowRoot. This broke Chrome's trace viewer, Catapult, which was assuming presence of the APIs on Chrome (it did not include proper polyfills; was only meant to run in Chrome anyways!).

To overcome the issue, we have tried few things and I will try to enumerate what was tried:

• include the webcomponents polyfill
  • Catapult uses Polymer 1.x which uses document.registerElement
  • only polyfill v0.x (https://github.com/webcomponents/webcomponentsjs/tree/v0.7.24) contains the polyfill for document.registerElement
• catapult has checks for real shadow DOM as opposed to polyfilled shadow DOM
  • it uses the value here: https://github.com/Polymer/polymer/blob/7a0f0181f573b79a1ec0b42a12bee37508ca52c6/polymer-micro.html#L39
  • Polymer.Settings does feature detection by checking for certain APIs
  • webcomponentjs does not polyfill Element.prototype.createShadowRoot which is critical for the feature detection: https://github.com/Polymer/polymer/blob/7a0f0181f573b79a1ec0b42a12bee37508ca52c6/polymer-micro.html#L33
• We can manually polyfill Element.prototype.createShadowRoot by using Element.prototype.attachShadow

Element.prototype.createShadowRoot = function() {
      const name = this.tagName;
      if (name.includes('-') ||
          name == 'div' ||
          name ==  'article' ||
          name == 'aside' ||
          name == 'blockquote' ||
          name == 'body' ||
          name == 'div' ||
          name == 'footer' ||
          name == 'h1' ||
          name == 'h2' ||
          name == 'h3' ||
          name == 'h4' ||
          name == 'h5' ||
          name == 'h6' ||
          name == 'header' ||
          name == 'main' ||
          name == 'nav' ||
          name == 'p' ||
          name == 'section' ||
          name == 'span'
      ) {
        return this.attachShadow({mode: 'open'});
      }
    };

• Even with our polyfill for createShadowRoot, we still get below

Uncaught DOMException: Failed to execute 'appendChild' on 'Node': Nodes of type '#document-fragment' may not be inserted inside nodes of type '#document'.
    at DomApi.<computed> [as appendChild

At this point, we can try to identify the problem and manually patch everything to make sure trace_viewer works (barely), but I think we should try to look what it will take to:

1. turn off native shadow DOM requirement, and
2. upgrade dependency on catapult to use Polymer 2.x and polyfill v1

If above two things are done, we will be able to fix #1291.

@psybuzz I can see how to vulcanize the trace viewer but I have no idea how the dependencies are managed in Chromium/catapult. Can you please help us try above two? I think I can, too, check for correctness of (1).

[lminer]

Is there any way around this? I've tried to install an older version of chrome on ubuntu and have been unsuccessful

[psybuzz]

A Chromium bug here was filed to fix/upgrade the Catapult trace viewer. I would recommend 'starring' the chromium bug report to add attention.
https://bugs.chromium.org/p/chromium/issues/detail?id=1020620

At this time, there isn't a fix, but there are workarounds if viewing Profile data is critical:

1. Download your trace's JSON data, go to chrome://tracing (if using Chrome), and load your data there to view it.

To get your data from TensorBoard, you can craft a URL like
http://<tensorboard-url>/data/plugin/profile/data?host=&run=<your-URI-encoded-run-name>&tag=trace_viewer

for example:
http://localhost:6006/data/plugin/profile/data?host=&run=profile_demo%2Ffoo&tag=trace_viewer

2. Run a Chromium-based browser from the command line with flags. Note that these flags apply not just to TensorBoard, but to any page loaded within that Chromium session.
   --enable-blink-features=ShadowDOMV0,CustomElementsV0,HTMLImports

3. If you are running 64-bit Linux specifically, you can download a snapshot of an earlier version of Chromium that has WebComponents V0 enabled. For example,
   https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Linux_x64/710083/

Note that using an older version of Chromium for an extended period introduces security risks.

[OverLordGoldDragon]

Download your trace's JSON data

@psybuzz How is this done?

[psybuzz]

If your TensorBoard process is running and you have a Chromium-based browser (e.g. the new Edge, Brave, Opera, Chrome), you can do this specifically as follows:

• Get the [run name] (e.g. "profile_demo/foo") from the left sidepane (or if you know it). If you have any [host name] other than 'default', get that too
  
• Craft a URL in this form:
  http://<tensorboard-url>/data/plugin/profile/data?host=<host-name>&run=<run-name>&tag=trace_viewer.
  For example if you are running TensorBoard on the origin localhost:6006 with run profile_demo/foo and empty host name, you would write the URL:
  http://localhost:6006/data/plugin/profile/data?host=&run=profile_demo%2Ffoo&tag=trace_viewer
• In your browser, open a tab and navigate to the URL you crafted from the last step. The page should show you the raw JSON data, something like this:
  
• Now, you can do Ctrl+S to save the page as a .json file on your filesystem.
• Then in a new browser tab, navigate to the url about://tracing. This should open your browser's trace viewing tool. Click the "load" button in the top-left
  
• Once you select the .json file that you saved earlier, it should load the trace into the viewer, and show what would normally appear in the TensorBoard Profile plugin:
  

Please let us know if one of these steps doesn't work.

[OverLordGoldDragon]

@psybuzz Perfect, thank you.

[mathpluscode]

Got the same problem and if the Host is local, you need to add a point after local like host=local.&run= to make it work

[CalebEverett]

It looks like this was fixed in the master branch of catapult, but I couldn't figure out how to get the fix into Chrome or Firefox. I tried installing the Canary version of Chrome to no avail.

Also tried downloading both the .trace and .tracetable files and loading them locally with about://tracing. The .tracetable file loaded but nothing showed up and the .trace file produced the following error on load:

While importing:
Error: Couldn't create an importer for the provided eventData.
    at Import.createImporter_ (chrome://tracing/tracing.js:2019:2071)
    at chrome://tracing/tracing.js:2014:167
    at Task.run (chrome://tracing/tracing.js:3247:95)
    at runAnother (chrome://tracing/tracing.js:3250:371)
    at runTask (chrome://tracing/tracing.js:3015:57)
    at processIdleWork (chrome://tracing/tracing.js:3020:116)
    at window.requestIdleCallback.timeout (chrome://tracing/tracing.js:3013:81)

UPDATE - downloaded correct file
Downloaded the file with the .trace.json.gz extension and was able to load it in about://tracing.

[fpmchu]

Hi,

I was able to get the workaround working. But I'm curious - could there be another issue? When looking at the Chrome network trace, it's loading from a URL of the form /data/plugin/profile/hosts?run=...&tag=trace_viewer and not /data/plugins/profile/data?host=&run=...&tag=trace_viewer. On my network trace I see the /hosts? link return an empty JSON object.

[Rubix982]

@psybuzz I'm following the tutorial on TensorFlow Profiler: Profile model performance and I decided to test it on colab.research.google.com . Screenshot is attached,

I'm unsure how to exactly copy or get that long string for Runs - the GUI allow / let me. How else can I get that string? Or, how do I get it to be like profile_demo/foo like you have?

[psybuzz]

Hi @Rubix982 , thanks for the screenshot. I'm unable to reproduce a blank screen on Colab, the tutorial seems to work for me.

Could you please share:

• Your browser version (e.g. Chrome 81.0.4044.113)
• Open the DevTools console (cmd+opt+J / ctrl+shift+J) and share any error messages you see?

[Rubix982]

Hi, @psybuzz ! Thanks for your time.
Browser: Chromium
Browser Version: Chromium 81.0.4044.92 Arch Linux

I used Ctrl + Shift + J, and yes, there were warnings and errors. An overview of them is given below. A screenshot is also attached below to give a look at the original errors (how I actually see them vs copy pasted here in the debug console tab),

tf-tensorboard.html.js:22294 paper-header-panel is deprecated. Please use app-layout instead!
ready @ tf-tensorboard.html.js:22294

tf-tensorboard.html.js:22317 paper-toolbar is deprecated. Please use app-layout instead!
ready @ tf-tensorboard.html.js:22317

outputframe.html?vrz=colab-20200420-085603-RC00_307386026:1 Failed to load resource: the server responded with a status of 500 (Not found)

DevTools failed to load SourceMap: Could not load content for https://colab.research.google.com/v2/common/webcomponentsjs/v2/webcomponents-lite.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

DevTools failed to load SourceMap: Could not load content for https://colab.research.google.com/v2/external/js/min-maps/vs/loader.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

DevTools failed to load SourceMap: Could not load content for https://colab.research.google.com/v2/common/webanimationsjs/web-animations-next-lite.min.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

DevTools failed to load SourceMap: Could not load content for https://colab.research.google.com/v2/external/js/min-maps/vs/editor/editor.main.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

DevTools failed to load SourceMap: Could not load content for https://colab.research.google.com/v2/external/js/min-maps/vs/base/worker/workerMain.js.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:1101 GET https://colab.research.google.com/tun/m/gpu-t4-s-2qztlh6b912ot/_proxy/6006/data/plugin/profile/trace_viewer%23;ea=20200421-184252%2Ftrain%2F2020_04_21_18_43_06;tag=trace_viewer%23;host=04ec0d756d0b?authuser=0 404

Failed to load resource: the server responded with a status of 500 ()

external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:1101 GET https://colab.research.google.com/tun/m/gpu-t4-s-2qztlh6b912ot/_proxy/6006/outputframe.html?vrz=colab-20200420-085603-RC00_307386026&authuser=0 404
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:1101
Ga @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:17
Ea.next_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:15
Ia.next @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:18
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
La @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
F @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
k.fetch @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:1101
N5.fetch @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:4117
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2637
Ga @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:17
Ea.next_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:15
Ia.next @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:18
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
La @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
F @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
AX.fetchResource_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2637
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2635
Ga @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:17
Ea.next_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:15
Ia.next @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:18
(anonymous) @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
La @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
F @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:26
k.handleServiceWorkerFetch_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2635
k.handleServiceWorkerMessage_ @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2634
m.onmessage @ external_polymer_binary.js?vrz=colab-20200420-085603-RC00_307386026:2630

outputframe.html?vrz=colab-20200420-085603-RC00_307386026:1 Failed to load resource: the server responded with a status of 500 ()

trace_viewer_index.js:5270 Uncaught Error: Assertion failed
    at assert (trace_viewer_index.js:5270)
    at assertIsNodeWrapper (trace_viewer_index.js:6763)
    at GeneratedWrapper.insertBefore (trace_viewer_index.js:6977)
    at GeneratedWrapper.appendChild (trace_viewer_index.js:6974)
    at HTMLHeadElement.constructor.<computed> [as appendChild] (trace_viewer_index.js:5507)
    at HTMLDocument.Document.createElement (<anonymous>:41:65)
    at Document.<computed> [as createElement] (trace_viewer_index.js:9292)
    at HTMLDocument.constructor.<computed> [as createElement] (trace_viewer_index.js:5507)
    at overrideConstructor (trace_viewer_index.js:9640)
    at Array.forEach (<anonymous>)

The screenshot,

Again, thank you for your time! If there is any other information that you require, please let me know.

[psybuzz]

The screenshot and errors are helpful. There's a strange assertion error from the webcomponents polyfill code. Since the polyfill is at the start of the JS bundle in the trace viewer iframe, this should imply that we have properly received the JS bundle and started evaluating it.

Based on webcomponents/webcomponentsjs#301 (a similar error), it seems like this could happen if someone other than our JS bundle evaluates DOM APIs before the polyfill loads.

It might be possible for a Chrome extension to run scripts in the iframe before our JS bundle. Could you please trying this out in Incognito (usually has no extensions enabled) or in a browser profile without extensions enabled?

If that doesn't work, could you also try this out in Firefox, if available? I'm curious whether it is something browser specific.

cc @qiuminxu @ydko for visibility, and if they have any ideas

[Rubix982]

@psybuzz , It works with Incognito!

Though I still see some warnings and errors in red as well, still.
I'm not sure which extension of mine is exactly causing the issue, but whatever it maybe, a temporary solution would be to just add in a note on that tutorial's page to disable your extensions or open that in incognito to see a proper result.

[Rubix982]

@psybuzz any updates on this?

[stephanwlee]

As of latest TensorBoard and https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras, the profile plugin should work on Firefox, Chrome, and others. Please give the new plugin a try. Thanks!