Run EmberPerf benchmark

[metajack]

Get EmberPerf working in Servo. EmberPerf is here: http://emberperf.eviltrout.com/

This metabug is specifically just to get it running. Performance issues will be tracked in a separate metabug.

• ???

Part of #16717.

[metajack]

Marking this easy. The task here is to run Servo on this benchmark and file bugs for any errors you find that prevent it from running.

[highfive]

Hi! If you have any questions regarding this issue, feel free to make a comment here, or ask it in the #servo channel in IRC.

If you intend to work on this issue, then add @highfive: assign me to your comment, and I'll assign this to you. 😄

[jonathandturner]

Just to note, the latested (2.11.1) EmberPerf benchmark does not run successfully for me on any of the browsers I tested (Nightly Firefox, Chrome, etc).

Do we have a version we're aiming to compare against?

[metajack]

@cpeterso Do you know if this is working in Gecko currently?

[cpeterso]

@eviltrout, is there a problem with EmberPref benchmark hosted on emberperf.eviltrout.com?

The benchmark, even older versions, now stops at "17 test(s) remaining" in Firefox, Edge, and Chrome. I see the following JavaScript error in the devtools console:

TypeError: Ember.LinkView is undefined[Learn More] index.js:35:9
setup http://emberperf.eviltrout.com/benchmarks/link-to/active/index.js:35:9
runner http://emberperf.eviltrout.com/assets/test-client.js:265:30
one http://emberperf.eviltrout.com/assets/test-client.js:1209:9
apiLoadAsync/</< http://emberperf.eviltrout.com/assets/test-client.js:1425:25
load/< http://emberperf.eviltrout.com/assets/test-client.js:1461:13
process http://emberperf.eviltrout.com/assets/test-client.js:1557:17

AWFY continues to report new EmberPerf measurements.

[eviltrout]

Apologies - the Link related tests broke some time ago on newer Ember releases. I've removed them in the latest version and deployed it to emberperf.eviltrout.com

[bnjbvr]

@eviltrout Hi! As of today, the two following tests are still raising errors on Firefox Nightly and Chromium stable:

• Ember.Component.create
• Render bind-attr

Is there anything to be done with these tests?

[eviltrout]

Well that's embarrassing! As you might have figured out by now we don't tend to run all the tests. We somewhat got in the habit of sticking to the Complex List tests since we found them the most reflective of Discourse performance :)

I've removed those tests now and confirmed the suite now runs all tests for the latest versions of Ember.

[bnjbvr]

I've removed those tests now and confirmed the suite now runs all tests for the latest versions of Ember.

Thanks! That's great.

[mihajlija]

This is what I get on macOS 10.12.4 when running default settings (2.13.0)

ERROR:script::dom::bindings::error: Error at http://emberperf.eviltrout.com/benchmarks/ember-get/primed/index.js:13:9 Ember is not defined

The "latest release" test works though http://imgur.com/jNEc0lW :)

[eviltrout]

Is this a servo bug? I just ran the suite with default settings in safari and everything worked.

[mihajlija]

@eviltrout Servo I guess, I get the same error when running JetStream benchmark ("JetStream is not defined")

[jdm]

I cannot reproduce that JS error when running with the default settings. I have tried a debug build, a release build, and a nightly.

[jonathandturner]

I can't see the JS error, but when I run this benchmark, this is what I see:

[jonathandturner]

I take that back, if I let it run long enough, I do see:

ERROR:script::dom::bindings::error: Error at http://emberperf.eviltrout.com/benchmarks/ember-set/primed/index.js:7:9 Ember is not defined

[jonathandturner]

As the screenshot suggests, the benchmark doesn't seem to be working the same way as on other browsers, as there doesn't appear to be any status updates as the benchmarks run

[jonathandturner]

(apologies for the spamming)

I was able to get a successful run that didn't have the error. Not sure what causes it to occur, yet.

Here's Firefox 53 for comparison:

[jonathandturner]

I filed a follow-up bug around the EmberPerf performance numbers here: #17329

[tamird]

Is it time to close this? This issue appears to be about just getting it to run, which @jonathandturner seems to have done.

This issue still appears on ServoStarters.

[jonathandturner]

@tamird - I mention in my runs that I was getting errors with some runs and not others. I'll check and report back whether the intermittent error is still occurring.

[jonathandturner]

@tamird - I can confirm we're still seeing the "ERROR:script::dom::bindings::error: Error at http://emberperf.eviltrout.com/assets/test-client.js:341:5 Ember is not defined" on some runs (and not others).

@jdm - I don't know how we want to handle intermittent failures.