fix: evaluateAsync behavior #1037

patrickhulce · 2016-11-23T23:25:08Z

addresses #1000 and #976

Ensures all Runtime.evaluate calls result in the native Promise
Transforms all errors produced during the evaluation to a standard object that can be rejected by our driver wrapper

brendankenny · 2016-11-23T23:37:48Z

I think there are two issues coming out of this that we should investigate if WAI from the debugger protocol:

little information from rejected promise (evaluateAsync doesn't return rejected promises properly #976). Ideally the exceptionDetails would indicate the promise was rejected but also have what was rejected (error object or string or whatever)
when JS is disabled (via Emulation.setScriptExecutionDisabled), scripts run via Runtime.evaluate do run, but it appears that scripts added by Page.addScriptToEvaluateOnLoad do not run. Is this on purpose?

For this PR, for instance, __nativePromise won't be defined in the js disabled pass, so the evaluateAsync wrapper has to grab Promise in that case (which is correct, but still unexpected that that script doesn't run)

patrickhulce · 2016-11-23T23:53:31Z

when JS is disabled (via Emulation.setScriptExecutionDisabled), scripts run via Runtime.evaluate do run, but it appears that scripts added by Page.addScriptToEvaluateOnLoad do not run. Is this on purpose?

yeah this threw me for a loop at first, but it started to make sense from the perspective of "load this JS on the page" should be off when JS is off, but later commanding you specifically to run JS will still work (which is critical for our gatherer anyway)

patrickhulce · 2016-11-23T23:54:57Z

little information from rejected promise (#976). Ideally the exceptionDetails would indicate the promise was rejected but also have what was rejected (error object or string or whatever)

and there is an object populated but it has mostly information about where the error occurred in the script just not the usual suspects you would use to typically report a user-friendly message. any first steps where I should poke around to see if we can easily add those without disrupting a larger beast?

brendankenny

some preliminary notes.

Since we're now wrapping everything we evaluate in another promise layer anyways, we could move back to supporting sync expressions in evaluateAsync, which would get rid of some noise in gatherers calling it

brendankenny · 2016-11-23T23:41:17Z

lighthouse-cli/test/fixtures/dobetterweb/dbw_tester.html

@@ -280,5 +280,7 @@
 }
 </script>

+<!-- Import zone.js to test Promise polyfill -->
+<script src="https://unpkg.com/zone.js?main=browser"></script>


we've been avoiding any network for the plain smoke tests. Any way we could do a quick Promise wrapper that would just defer to native promise fns but fail the Promise === __nativePromise check?

brendankenny · 2016-11-23T23:46:12Z

lighthouse-core/gather/gather-runner.js

@@ -90,6 +90,7 @@ class GatherRunner {
    return driver.assertNoSameOriginServiceWorkerClients(options.url)
      .then(_ => driver.beginEmulation(options.flags))
      .then(_ => driver.enableRuntimeEvents())
+      .then(_ => driver.evaluateScriptOnLoad('window.__nativePromise = Promise;'))


need to doc this in outline at top

brendankenny · 2016-11-23T23:47:48Z

lighthouse-core/gather/driver.js

+      // information such as name, message, and stack trace of the error when it's wrapped in a
+      // promise. Instead, map to a successful object that contains this information.
+      /* istanbul ignore next */
+      const errorWrapper = err => {


maybe move this below (outside of the class) with captureJSCallUsage

brendankenny · 2016-11-23T23:50:42Z

lighthouse-core/gather/driver.js

      this.sendCommand('Runtime.evaluate', {
-        expression: asyncExpression,
+        expression: `(function wrapInNativePromise () {


no space after function name

brendankenny · 2016-11-23T23:51:13Z

lighthouse-core/gather/driver.js

      this.sendCommand('Runtime.evaluate', {
-        expression: asyncExpression,
+        expression: `(function wrapInNativePromise () {
+          var __nativePromise = window.__nativePromise || Promise;


const (here and below)

brendankenny · 2016-11-23T23:51:21Z

lighthouse-core/gather/driver.js

-        expression: asyncExpression,
+        expression: `(function wrapInNativePromise () {
+          var __nativePromise = window.__nativePromise || Promise;
+          return new __nativePromise(function (resolve) {


no space after function

brendankenny · 2016-11-24T00:28:18Z

yeah this threw me for a loop at first, but it started to make sense from the perspective of "load this JS on the page" should be off when JS is off, but later commanding you specifically to run JS will still work (which is critical for our gatherer anyway)

that makes some sense, but it's not something we document for evaluateScriptOnLoad, might be kind of surprising if you assume your code runs before page load every time and you weren't aware a gatherer was switching off js, and appears to leave no option for if you really do need some js snippet to run before the page starts loading

brendankenny · 2016-11-24T00:31:52Z

any first steps where I should poke around to see if we can easily add those without disrupting a larger beast?

not sure. I did notice the devtools tests for awaitPromise: true test for exactly this limited output on rejection, so not sure how much of this was intended or was intended for improvement later.

addresses GoogleChrome#1000 and GoogleChrome#976 1. Ensures all `Runtime.evaluate` calls result in the native Promise 2. Transforms all errors produced during the evaluation to a standard object that can be rejected by our driver wrapper

paulirish · 2016-11-28T23:39:54Z

@ak239 could you take a look at this?

It looks like you do similar stuff in native: https://cs.chromium.org/chromium/src/v8/src/inspector/v8-runtime-agent-impl.cc?q=f:v8-runtime+wrapEvaluateResultAsync&sq=package:chromium&l=215&dr=CSs .. and I want to make sure we're being robust in handling the error cases.

And specifically you probably have a decent answer for @brendankenny's questions here: #1037 (comment)

alexkozy · 2016-11-29T23:30:12Z

Runtime.evaluate method with awaitPromise flag returns rejected value in result.result object.
If it's primitive value then you can just use its value, to wrap other types of values you can use following code:

Protocol.Runtime.evaluate({ expression: 'Promise.reject(new Error())', awaitPromise: true })
  .then(wrapErrorIfNeeded)
  .then(console.log);

function wrapResultIfNeeded(message) {
  if (message.error) {
    // if protocol returns internal error - return it.
    return `Internal error: ${message.error.message}`;
  }
  if (message.result.result.value) {
    // if value has primitive type then return its value.
    return message.result.result.value;
  }
  if (message.result.result.unserializableValue) {
    // if value can't be represented in json (e.g. NaN) - evaluate unserializableValue.
    return eval(message.result.result.unserializableValue);
  }
  if (message.result.result.objectId) {
    // if values is object ..
    if (message.result.result.subtype === "error") {
      // .. and it's an error - call function on this object which will return JSON.stringifiable object.
      return Protocol.Runtime.callFunctionOn({ objectId: message.result.result.objectId, functionDeclaration: wrapError.toString(), returnByValue: true }).then(wrapResultIfNeeded);
    } else {
      // .. otherwise do something, we can try to stringify it via protocol:
      return Protocol.Runtime.callFunctionOn({ objectId: message.result.result.objectId, functionDeclaration: "function foo() { return this; }", returnByValue: true }).then(wrapResultIfNeeded);    
    }
  }
  // should be not reached.
  return "Unknown error";
}

I think you can use this to avoid try .. catch.

patrickhulce · 2016-11-30T01:49:13Z

discussed with @ak239 in person, summary:

callFunctionOn still fails to capture details about errors in promises since no objectId is given and JSON.stringify(new Error()) === '{}' so we're going to stick with the hackier try/catch for now

@paulirish any other comments here?

brendankenny · 2016-11-30T01:52:23Z

lighthouse-cli/test/fixtures/dobetterweb/promise_polyfill.js

@@ -0,0 +1,257 @@
+/*
+ * @license


this needs to be in lighthouse-core/third_party/ (or better, the npm version...should be fine since just a dev dependency)

@brendankenny suggestions on getting this loaded by the static server? just add static check to fetch this from third party?
and i wish we could just do the npm version but this polyfill won't work on its own since chrome already defines Promise so I modified it, we could try to find another one that storms over it instead?

ya lets put in third_party. the static server can just look for a request to promise_polyfill and reach across to adjust the absoluteFilePath

brendankenny · 2016-11-30T01:56:44Z

lighthouse-core/gather/driver.js

      this.sendCommand('Runtime.evaluate', {
-        expression: asyncExpression,
+        expression: `(function wrapInNativePromise() {


this feels like it could be simplified, but are there reasons for everything? Why wrap in a promise constructor and try/catch and Promise.resolve() instead of doing a single promise wrapper?

should also update the function docs

yes each has a purpose

try/catch - for errors that happen outside of promises
Promise.resolve - to enable sync executions
new Promise - to ensure the promise returned is indeed a native promise + avoid inconsistent error handling between sync and async paths

but as I'm typing this just remembering that we opted for Promise.resolve().then( => ), will fix

wanna add a comment to document this? we're definitely gonna be headscratching if we need to touch this code again. :)

brendankenny · 2016-11-30T02:05:04Z

lighthouse-core/gather/driver.js

        } else {
-          resolve(result.result.value);
+          resolve(value);


we can save for another issue (maybe this is also #941), but the fact that there are three cases here

success

failure due to driver error

failure (of some sort) due to the evaluated expression

makes it feel like we shouldn't be conflating the last two, but I'm not exactly sure of an elegant way to do this, or even what gatherers should do (catch or re-throw) if we do differentiate by type

I think the differentiation is just done by what error message and stack trace results. Unless we think driver error should be fatal?

yeah, I think @paulirish is addressing this in his latest comments in #941. We basically need a way to say "this is an error I expected (fetch rejected on offline request or whatever)" vs "whoooops". For this, the caller of driver.evaluateAsync will have to tell the difference for now and we can revisit as we figure out error paths

alexkozy · 2016-12-01T16:19:19Z

I've another idea how to remove try catch:

Call Runtime.evaluate without returnByValue: true,
In then callback - check type of object and if it's not an error then just callFunctionOn(objectId, "() => this", returnByValue: true), otherwise callFunctionOn(objectId, wrapError, returnByValue: true).

Code snippet to demonstrate approach:

this.sendCommand('Runtime.evaluate', {
  expression: `(function wrapInNativePromise() {
    const __nativePromise = window.__nativePromise || Promise;
    return __nativePromise.resolve()
      .then(_ => ${asyncExpression});
  }())`,
  includeCommandLineAPI: true,
  awaitPromise: true
  }).then(wrapResultIfNeeded.bind(this, undefined));

function wrapResultIfNeeded(isError, message) {
  if (typeof isError !== 'boolean')
  	isError = !!message.exceptionDetails;
  if (message.error) {
    // if protocol returns internal error - return it.
    return Promise.reject(`Internal error: ${message.error.message}`);
  }
  if (message.result.result.value) {
    // if value has primitive type then return its value.
    if (isError)
	  return Promise.reject(message.result.result.value);
	else
	  return Promise.resolve(message.result.result.value);
  }
  if (message.result.result.unserializableValue) {
    // if value can't be represented in json (e.g. NaN) - evaluate unserializableValue.
    if (isError)
      return Promise.reject(eval(message.result.result.unserializableValue));
    else
      return Promise.resolve(eval(message.result.result.unserializableValue));
  }
  if (message.result.result.objectId) {
    // if values is object ..
    if (message.result.result.subtype === "error") {
      // .. and it's an error - call function on this object which will return JSON.stringifiable object.
      return Protocol.Runtime.callFunctionOn({ objectId: message.result.result.objectId, functionDeclaration: wrapError.toString(), returnByValue: true }).then(wrapResultIfNeeded.bind(this, isError));
    } else {
      // .. otherwise do something, we can try to stringify it via protocol:
      return Protocol.Runtime.callFunctionOn({ objectId: message.result.result.objectId, functionDeclaration: "function foo() { return this; }", returnByValue: true }).then(wrapResultIfNeeded.bind(this, isError));    
    }
  }
  // should be not reached.
  return "Unknown error";
}

function wrapError()
{
  // this - Error object, do something smart and return stringifiable object.
  return ({
    customError: true,
    name: this.name || 'Error',
    message: this.message || 'fallbackMessage',
    stack: this.stack || (new Error()).stack,
    __failedInBrowser: true
  });
}

patrickhulce · 2016-12-01T17:17:01Z

@ak239 what's the benefit to using that approach over the new .catch(theStringifyFunction)?

seems like either way we have to execute wrapError on it and the current way is done without multiple protocol calls

brendankenny

sorry, a few more things. Also need to update evaluateAsync jsdocs to drop the must evaluate to a promise part

brendankenny · 2016-12-02T00:39:02Z

lighthouse-cli/test/fixtures/static-server.js

@@ -24,7 +24,9 @@ const parseURL = require('url').parse;
 function requestHandler(request, response) {
  const filePath = parseURL(request.url).pathname;
  const queryString = parseURL(request.url).search;
-  const absoluteFilePath = path.join(__dirname, filePath);
+  const absoluteFilePath = filePath === '/dobetterweb/promise_polyfill.js' ?


maybe add a comment here for why this is done?

I also feel like this is also going to be a long term resident in static-server, so a full if statement is probably better so it's a bit quicker mentally parse

brendankenny · 2016-12-02T01:08:04Z

lighthouse-core/gather/driver.js

        } else {
-          resolve(result.result.value);
+          resolve(value);


yeah, I think @paulirish is addressing this in his latest comments in #941. We basically need a way to say "this is an error I expected (fetch rejected on offline request or whatever)" vs "whoooops". For this, the caller of driver.evaluateAsync will have to tell the difference for now and we can revisit as we figure out error paths

brendankenny · 2016-12-02T01:09:05Z

lighthouse-core/gather/driver.js

        if (result.exceptionDetails) {
-          reject(result.exceptionDetails.exception.value);
+          // An error occurred before we could even enter our try block, should be *very* rare


no more try block

brendankenny · 2016-12-02T01:18:17Z

lighthouse-core/third_party/promise-polyfill/promise.js

@@ -0,0 +1,257 @@
+/*


@rviscomi for another third_party/ thing to delete for import. Sorry about this, but we need to modify it to be able to use.

Good news is that this is for testing only, so you can just delete? Not sure how you deal with third_party/traceviewer-js/ today

brendankenny

LGTM. This will be great for evaluateAsync errors

1. Ensures all `Runtime.evaluate` calls result in the native Promise 2. Transforms all errors produced during the evaluation to a standard object that can be rejected by our driver wrapper

patrickhulce force-pushed the fix_evaluate_async branch from 88bfa18 to 8d29791 Compare November 23, 2016 23:27

brendankenny requested changes Nov 24, 2016

View reviewed changes

patrickhulce force-pushed the fix_evaluate_async branch 2 times, most recently from 8732398 to 5b10bf1 Compare November 24, 2016 01:30

fix: evaluateAsync behavior

46b8301

addresses GoogleChrome#1000 and GoogleChrome#976 1. Ensures all `Runtime.evaluate` calls result in the native Promise 2. Transforms all errors produced during the evaluation to a standard object that can be rejected by our driver wrapper

patrickhulce force-pushed the fix_evaluate_async branch from 5b10bf1 to 46b8301 Compare November 24, 2016 01:59

brendankenny mentioned this pull request Nov 29, 2016

DBW: Audit for notification permissions on start #1064

Merged

fix: consistency with feedback on other PRs

11ddfa7

patrickhulce force-pushed the fix_evaluate_async branch from 3494070 to 11ddfa7 Compare November 29, 2016 01:51

brendankenny requested changes Nov 30, 2016

View reviewed changes

more feedback

ca5091d

patrickhulce mentioned this pull request Nov 30, 2016

Extension Error: Cannot read property 'message' of undefined #1082

Closed

brendankenny requested changes Dec 2, 2016

View reviewed changes

patrickhulce added 4 commits December 2, 2016 09:34

more more feedback

c8e14c7

Merge remote-tracking branch 'upstream/master' into fix_evaluate_async

1343e61

Merge remote-tracking branch 'upstream/master' into fix_evaluate_async

a97a2ea

doh

7c5995f

brendankenny approved these changes Dec 3, 2016

View reviewed changes

brendankenny merged commit 83682cb into GoogleChrome:master Dec 3, 2016

patrickhulce mentioned this pull request Dec 12, 2016

Extension Error: Cannot read property 'message' of undefined #1147

Closed

This was referenced Dec 16, 2016

evaluateAsync fails if Promise has been polyfilled on target page #1000

Closed

evaluateAsync doesn't return rejected promises properly #976

Closed

Regression: Runtime.evaluate errors on polyfilled Promises #1173

Closed

paulirish mentioned this pull request Sep 13, 2019

tests: simplify serving of zone.js #9672

Merged

paulirish mentioned this pull request Sep 29, 2021

Runtime issue: TypeError: Cannot read property 'cells' of undefined dequelabs/axe-core#2674

Closed

fix: evaluateAsync behavior #1037

fix: evaluateAsync behavior #1037

Conversation

patrickhulce commented Nov 23, 2016

brendankenny commented Nov 23, 2016 • edited

patrickhulce commented Nov 23, 2016

patrickhulce commented Nov 23, 2016

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Nov 24, 2016

brendankenny commented Nov 24, 2016

paulirish commented Nov 28, 2016 • edited

alexkozy commented Nov 29, 2016 • edited

patrickhulce commented Nov 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexkozy commented Dec 1, 2016

patrickhulce commented Dec 1, 2016

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny left a comment

Choose a reason for hiding this comment

brendankenny commented Nov 23, 2016 •

edited

paulirish commented Nov 28, 2016 •

edited

alexkozy commented Nov 29, 2016 •

edited