Add an 'updateResult' utility function to upsert values into data-* files

[robhogan]

As far as I could tell, the current accepted practice to update data- files is to use a script to suggest the changes to a console and then to make manual updates.

This is a lot of work for new runtimes, and doesn't lend well to automating as part of a runtime release process.

This is a simple utility to parse and update res objects in place. It may be called from the test runner scripts - I used it to generate #1808.

Because it uses an actual parser it should be robust to syntax gotchas like code-in-strings, but it makes a few basic assumptions that hold for the data files as they are now, like:

• Data files have an exports.tests = [ /* ... */ ] section
• Tests must have an object-literal res property, or an array-literal subtests property.

[ljharb]

Most runtimes we support are browsers, so the primary workflow is to manually update results as they’re checked. Command-line runtimes sometimes have a script that updates the files automatically.

[ljharb]

How is this intended to be used, exactly?

[robhogan]

Thanks for looking at this so quickly. I work with the Hermes engine team so we have an interest in (semi-)automating updates to compat-table as part of the engine release process.

I've pushed my changes to hermes.js to show how I've used the utility - basically we modify the data file in place on a compat-table checkout. The changes can be reviewed by eye, footnotes manually added if necessary, and then pushed up for a PR. It eliminates the error-prone step of taking the console output and manually transcribing the results into the data files.

[ljharb]

https://github.com/kangax/compat-table/blob/gh-pages/duktape.js, https://github.com/kangax/compat-table/blob/gh-pages/jerryscript.js, https://github.com/kangax/compat-table/blob/gh-pages/nashorn.js, and https://github.com/kangax/compat-table/blob/gh-pages/rhino.js seem like they would benefit from the same approach?

[robhogan]

I’d say so. Happy to update those too if the approach looks good to you.

[robhogan]

(Done)

[robhogan]

Thanks for the review @ljharb - the reason for the retro syntax though is the jshint lint setup for this project. For example, applying one of the suggestions above (which I'd normally very much agree with!):

> npm run lint
update-result.js: line 147, col 52, 'arrow function syntax (=>)' is only available in ES6 (use 'esversion: 6').

I'm not exactly sure what that implies about runtime feature support, but I've tried to be as conservative as I remember how to be (my ES3 is a little rusty!), mostly for consistency's sake - hence avoiding Array.from, repeat etc - it feels a bit awkward to use newer runtime features while working with a linter that forbids const.

These constraints make sense in the test code where we obviously need to cater for the lowest common denominator, but probably shouldn't apply to scripts written for Node JS, like this one or the various engine harnesses. I'm not sure how best to reconcile that - different lint settings for test code vs scripts? Different linter? Exclude scripts from lint?

[ljharb]

ha, ok fair enough :-)

just ignore those suggestions for now; we should update from jshint to eslint anyways, and then we can use overrides so that scripts like this don't have to meet the same standards.

[robhogan]

Sounds good to me - I'll assume modern runtime support and apply all the suggestions the linter is happy with

[p-bakker]

Not sure why this PR went nowhere, but #1881 achieves the same, without having to use a parser to modify the module code AND will allow custom Compat Tables, like the (vary outdated) custom compat Table we have at https://rhino.github.io/compat/engines.html