Add a bisection command that uses nightly builds

[jdm]

http://servo-builds.s3.amazonaws.com/ hosts a long list of historical nightly builds. We should write a command that allows several modes of operation:

• ./mach bisect tests/wpt/web-platform-tests/foo.html - runs WPT tests using downloaded nightly builds, suggests good/bad bisection based on test result
• ./mach bisect path/to/file.html - runs downloaded nightlies and loads file:// url for easy testing
• ./mach bisect http://foo.com - runs downloaded nightlies and loads url for easy testing
• ./mach bisect -b - runs downloaded nightlies with browser.html for easy testing

We should cache the downloaded nightlies to make it easier to do future bisections, too.

[highfive]

cc @wafflespeanut

[jdm]

git rev-list -n 1 --before="2017-01-01" master is a useful command to figure out a commit associated with a particular nightly.

[jdm]

Curiously, http://servo-builds.s3.amazonaws.com/ only shows mac builds up until 2016-10-25. There are plenty of linux and android builds for recent days, however.

[jdm]

Ah, <MaxKeys>1000</MaxKeys> is hiding the entries that I care about.

[jdm]

I use http://servo-builds.s3.amazonaws.com/?list-type=2&prefix=nightly/mac/2017 when investigating regression ranges for macOS builds from 2017.

[o0Ignition0o]

This one looks very interesting, could I please pick it? I think I can start working on it next week :)

[jdm]

That would be great!

[jdm]

@o0Ignition0o Have you made any progress with this? If not, I would like to turn this into a project for some students this term.

[o0Ignition0o]

Hi @jdm :) I haven't made too much progress with this yet, so you can assign it to someone else if you want to. I'll pick up an other task

[jdm]

That's fine; you're welcome to keep working on it!

[atouchet]

@jdm are Windows builds available with http://servo-builds.s3.amazonaws.com/? I see every other platform on there aside from Windows. Having historical nightly builds would be useful for testing purposes.

[jdm]

Note my comment earlier with the prefix argument. If you specify the right prefix, they should appear.

[atouchet]

http://servo-builds.s3.amazonaws.com/?list-type=2&prefix=nightly/windows-msvc/ appears to work, thanks.

[o0Ignition0o]

Hey there :) I've been able to write a rough try to download a (hardcoded for now) nightly version and extract it.

I put the bisect command as a "mach wrapper" for now so I can run any other mach command, eg :
./mach bisect run https://www.mozilla.org
./mach bisect help
./mach bisect test --all

I'm now looking for a way to set the downloaded servo bin as the target I'm supposed to use.

I have a feeling this should be an environment variable, but I haven't found it quite yet ^^'
Next steps for me would be parsing s3's XML files (with beautifulsoup or something similar) and binary searching my way through the ./mach commands exit codes, but I need to be able to set the servo target first.
If you have an idea please let me know :)

[jdm]

https://github.com/servo/servo/blob/master/python/servo/post_build_commands.py#L99 is the implementation for ./mach run, and it relies on get_binary_path. There's no built-in way to override the binary that is selected right now; adding an additional argument to the run command that passes a specific binary to use would probably make sense.

[o0Ignition0o]

Hi, thanks a lot for your response.

So here's what I'm going to do :

1 - Add a --bin flag to ./mach run to allow a user to specify the path to the executable to run

2 - Add a --nightly | -n flag to allow a user to specify a YYYY-MM-DD date. This flag will :
Figure out which OS the user is running
Check the nightlies folder to find out if the nightly version has already been downloaded and extracted
if not :
Try to fetch it from the S3 server
Raise an exception and print a message if there is no nightly for the specified date and platform
run the specified command with the nightly version
The default nightly folder will be PROJECT_ROOT/target/nightlies/ and should be customizable through an environment variable

3 - go back to the bisect command, which should be easier to implement once these steps are performed.

Each step will be a separate Pull Request, which I'll link to this issue.
Should I file separate issues for each step ?
Are there things I should reconsider before working on the implementation ?

[jdm]

That sounds great!

[o0Ignition0o]

Just pushed a rough try to handle the --nightly flag.
I really don't like the way I let the -n flag silently fail if an empty string is provided, I'll amend my commit asap ^^'
EDIT : This is better, it will explicitely fail if an empty date is provided.

[o0Ignition0o]

Ok so I chose to split my merge requests depending on the platforms, because I don't know how yo extract a .msi and a .dmg archive with python ^^'

If you have any hint pease let me know, I'll keep on digging :)

[o0Ignition0o]

Ok so it seems like passing the -qi flags to msiexec.exe (on windows) should allow me to silently install a servo build, but I'm not sure at all I wouldn't mess any registry keys / already installed versions of servo.
Could anyone please point me to someone knowledgeable on windows installers so I can get more insights ?

PS : In the mozilla-unified repository, we seem to use executables so it seems easier to handle, but I'm not quite sure how I should handle .msi files :/

[jdm]

According to https://superuser.com/questions/307678/how-do-i-extract-files-from-an-msi-package you should be able to use msiexec /a c:\testfile.msi /qb TARGETDIR=c:\temp\test to extract the contents of the MSI to the target directory, which avoids the issues with installing the package.

[o0Ignition0o]

Thanks a lot, I ll give it a try :)

[o0Ignition0o]

Just ran into github issues claiming running servo in a VM is not that easy, so I think the fact I can't run servo.exe might be related to my virtualbox setup.
I'll just remove the print occurrences and submit it for review.
I don't really know whether extracting only the executable is relevant or not, maybe as a followup bug if no other file is needed ?

Edit : I also don't have any IOS device so I might be stuck on providing mac os support for the flag :/
But I sure could help someone else building the feature ! (Maybe I could help someone ? :) )

[jdm]

If you add your ssh key to https://github.com/servo/saltfs/tree/master/admin/files/ssh and your username to https://github.com/servo/saltfs/blob/master/admin/map.jinja, you could get ssh access to our mac build machines if you wanted :)

[o0Ignition0o]

Oh this is great !
I'll file to get access then, thanks a lot !

[o0Ignition0o]

It looks like windows builds now produce an executable instead of a .msi archive, sothe -n flag doesnt work on windows anymore :( .

I'm trying to find a way to extract the executable without installing the application, but couldn't manage so far.

Also https://firefox-source-docs.mozilla.org/mozbase/mozinstall.html doesn't allow me to just extract a portable version, maybe I could submit a pr to mozinstall to allow people to just extract contents ?

[jdm]

Interesting point; that's an unfortunate consequence. If you want to change mozinstall, you probably need to submit a patch to mozilla-central, since I suspect that https://searchfox.org/mozilla-central/source/testing/mozbase/mozinstall is the authoritative source.

[o0Ignition0o]

Yeah, plus that would maybe allow us to use exclusively the mozinstall API on Windows Linux and macos, which would be an improvement. I'll try to submit a patch :)