Adapt arewefastyet to report Servo benchmark results over time

[jdm]

https://github.com/h4writer/arewefastyet

[dhananjay92]

Looks interesting project to me.
Couple of questions...

• awfy compares MozFF with Chrome. What should Servo be compared against?
• Should all the benchmarks be supported for servo?! Since servo is still under development, I'm not sure if it can run all benchmarks.

Currently, I am trying to understand awfy and ultimately estimate the time required to do this project. I am not sure if it can be, but if it is doable in about 2-3 months, I will take it in the Fall semester. Also is it possible to get mentor for this project?

[jdm]

@dhananjay92 Ideally we would compare against both, but I'd be fine with just starting with Servo and no comparisons. The goal here would be to figure out the benchmarks that work and don't work in current Servo, and file issues about the ones that currently don't work while enabling any that do. We may end up trying to do this sooner than fall, but the future is unclear right now.

[dhananjay92]

@jdm Oh okay. Since, I can work during Fall only, please let me know if by end of summer this is not done. I will take it then! 😃

[pcwalton]

I think it's a bit premature to openly compare against other implementations, since we're obviously less complete than they are.

[dhananjay92]

@jdm, @pcwalton: Is this issue taken? I am free now. I can work on it during this semester.

[jdm]

@dhananjay92 Go for it!

[dhananjay92]

I changed the awfy's existing scripts (see 871cc2). Now using build.py, we can pull and build servo.

But, since I have 32-bit linux, the build fails with Issue #3899 as the reason. I tried doing it on my 64bit Windows machine, but there also it fails at the same place. The exact error:

Downloading Rust snapshot...
Download failed (404): Not Found - https://servo-rust.s3.amazonaws.com/7b7fc67dd453c470a48dbdcf64693a93293c9ab0/rustc-1.4.0-dev-x86_64-mingw64_nt-6.1.tar.gz

Is there any workaround to this? I can always install ubuntu-64bit and try if there isn't any workaround for this.

[jdm]

The workaround is for us to upload 32 bit linux snapshots like we're supposed to. I'll bring it up in the meeting tomorrow!

[dhananjay92]

Cool, Thanks.

[jdm]

FWIW I tried running that branch on OS X and got this:

>> Executing in /Users/jdm/src/arewefastyet/slave
./output/mach build --release
with: {'CC': 'gcc', 'PYTHONPATH': ':/usr/local/lib/python2.7/site-packages', 'SSH_AUTH_SOCK': '/tmp/launch-JAoEcw/Listeners', 'TMPDIR': '/var/folders/m7/9_hs2jb12kl4h0ww513v96cw0000gn/T/', 'SHELL': '/bin/bash', 'LOGNAME': 'jdm', 'USER': 'jdm', 'HOME': '/Users/jdm', 'ITERM_PROFILE': 'Default', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin', 'DISPLAY': '/tmp/launch-jbTNWw/org.macosforge.xquartz:0', '_': '/usr/local/bin/python', 'TERM_PROGRAM': 'iTerm.app', 'LANG': 'en_CA.UTF-8', 'TERM': 'xterm-256color', 'Apple_PubSub_Socket_Render': '/tmp/launch-crUzFU/Render', 'COLORFGBG': '7;0', 'SHLVL': '1', 'OLDPWD': '/Users/jdm/src/arewefastyet', 'PWD': '/Users/jdm/src/arewefastyet/slave', '__CF_USER_TEXT_ENCODING': '0x1F5:0:0', 'CXX': 'g++', 'ITERM_SESSION_ID': 'w0t7p0', 'EDITOR': 'nano', 'LINK': 'g++', '__CHECKFIX1436934': '1'}
output was: Could not find `Cargo.toml` in `/Users/jdm/src/arewefastyet/slave/output/support/android-rs-glue/glue`
Downloading Rust snapshot...
Extracting Rust snapshot...
Snapshot Rust ready.
Downloading Cargo nightly...
Extracting Cargo nightly...
Cargo ready.
Build completed in 0.05s

Command '['./output/mach', 'build', '--release']' returned non-zero exit status 101
Traceback (most recent call last):
  File "build.py", line 324, in <module>
    builder.build(puller)
  File "build.py", line 89, in build
    self.make()
  File "build.py", line 264, in make
    Run(args, self.env.get())
  File "/Users/jdm/src/arewefastyet/slave/utils.py", line 127, in Run
    raise e
subprocess.CalledProcessError: Command '['./output/mach', 'build', '--release']' returned non-zero exit status 101

[jdm]

The problem is the current working directory is incorrect for mach's needs; if I cd into output and run mach from there, we clone the submodule as necessary.

[dhananjay92]

So I need to cd into output first and then invoke mach. I didn't run into this one because I never reached that stage. Got the failures while downloading the rust snapshot itself. Once we have rust-snapshot solved, I will make all the necessary changes. Till that time, I am understanding all the parts of awfy.

BTW do we want to maintain repository for servo's benchmarking separately or should I keep merging my changes with @h4writer's original awfy repo?

[jdm]

If it works, it probably can't hurt to upstream. I don't know if there are ways to exclude certain builders, though, as I doubt we'll want to be tracking Servo publicly on AWFY immediately.

[dhananjay92]

Then let's keep this one separate. When we think servo is ready to be compared against other browsers, we can merge the changes with main AWFY.

[dhananjay92]

Opened Issue#1 to track the progress on this. From now on, I will open issues in my repository to track the progress for each major task.