Create a webdriver client for testing the server

[jdm]

The webdriver server lives in components/webdriver_server and is used for native, external control over Servo for automation purposes. Right now it's really challenging to ensure that this actually works - we should write a simple client that connects to a server (ie. Servo running with --webdriver) and takes input through stdin to send commands to servo. This will require using the webdriver-rust library (and potentially extending it to add a client interface) for maximal code reuse, but could be a great way to play with a small independent project in Rust.

[sbditto85]

I'd be interested in trying my hands at this, but I've never worked on servo before is this a good starter issue? Anything I should know before I start?

[jdm]

@sbditto85 I think this would be a good starter issue, since you don't have to know anything about Servo internals. You also won't learn anything about Servo internals by doing it, but you'll get used to the Servo contribution model! I believe everything you need to know is in the first comment; to get started, you'll probably want to create a new webdriver_client crate in components/ and add a Cargo.toml similar to the one in components/servo (ie. it must have a [[bin]] section), since we'll be wanting to create a runnable binary. I would then add a build-webdriver command to python/servo/build_commands.py using build-cef as a guide, and instead of ports/cef being the target use components/webdriver_client. That will allow you to build your client code using ./mach build-webdriver.

[sbditto85]

OK I'll try it go ahead and assign it to me

[jdm]

Great!

[jdm]

@sbditto85 Are you still working on this? Have you made any progress?

[sbditto85]

I was working on it and made some progress, then life got in the way. If you'd like I can attempt to start working on it again or reassign its up to you.

[jdm]

I don't know of anybody else looking to work on this, but if you could push your code to a branch and link it here then maybe someone else could come along and take over if you don't find time to continue it.

[jgraham]

There is an alternate option here, which I think is better; make a python-based testsuite that we can then share with other browsers implementing the WebDriver spec (e.g. Gecko, IE).

I made the start of such a testsuite at https://github.com/w3c/web-platform-tests/commits/jgraham/webdriver although that has bitrotted somewhat. There's an improved version of the webdriver client there in the servodriver branch of wptrunner: https://github.com/w3c/wptrunner/blob/master/wptrunner/executors/webdriver.py

So I guess the way to start here would be to make the webdriver client into a new package on pypi, and then resurrect the integration of webdriver tests into wptrunner, before actually writing some tests for the things that are implemented (this last part is probably the hardest part of the whole exercise).

[shinglyu]

I am interested in this.

So as far as I understand, there are two options:

1. Write a Rust-based webdriver command-line client
2. Use a python-based webdriver client

Then run it with wptrunner and write test cases.

@jdm @jgraham, do you find any value in writing a Rust-based webdriver client? Otherwise I'll go with the python way?

[jgraham]

FWIW I have the beginnings of a spec-compliant python based client: https://github.com/w3c/wptrunner/blob/master/wptrunner/executors/webdriver.py

But it would be good to make it possible to run webdriver spec tests directly from wptrunner; that would be a fair amount of work in that project. And then we would need some tests. You might want to talk to @andreastt who was going to look at this too.

[shinglyu]

@jgraham Sorry I am not familiar with wptrunner, so I am not quite sure what needs to be done. So do you mean we need to finish the webdriver.py? And I don't know what needs to be done to "run webdriver spec tests directly from wptrunner", what is the current way of doing this? Thank you.

[liviuba]

I can give this a try

[jdm]

Sorry, this is currently claimed by a group of students from NCSU this term. I'll update this issue if they stop working on it.

[krunal3103]

We are facing some trouble on understanding how to actually start the webdriver client in servo to execute the tests! We tried to test a few cases by using the selenium webdriver and creating a Firefox driver, but aren`t sure how to create one for servo for these tests

[jdm]

@krunal3103 There's two main steps - start Servo with the webdriver server enabled (--webdriver 7000), and running the python script that uses the client and connects to it (cd tests/wpt/harness/wptrunner/executors; python /path/to/test.py). What's the piece that's causing you difficulties?

[krunal3103]

1)Are we supposed to create the test.py under the executors directory or
under tests/webdriver directory?
2)Also the automated tests that we need to write to check the validity of
the url, do we need to just GET and use assert_equals to check what url has
been fetched?
3) the sample code for creating the session, the git page says to keep it
under executors directory. Is that right? We create a session and then call
the tests to execute the tests?

Thank you.

On Monday, March 21, 2016, Josh Matthews notifications@github.com wrote:

@krunal3103 https://github.com/krunal3103 There's two main steps -
start Servo with the webdriver server enabled (--webdriver 7000), and
running the python script that uses the client and connects to it (cd
tests/wpt/harness/wptrunner/executors; python /path/to/test.py). What's
the piece that's causing you difficulties?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5970 (comment)

Regards,
Krunal Gala

[jdm]

1. The first test.py can be anywhere; it's just for playing with webdriver and experimenting. The actual automated tests you write will live in tests/webdriver.
2. Yes.
3. The webdriver.py that lives in tests/wpt/harness/wptrunner/executors should stay there. The automated testing code should probably start a new instance of Servo and a new session for each test that gets run.

[andreastt]

The WPT WebDriver specification test harness work is being tracked in w3c/wptrunner#171 (and a few other integrated PRs). It enables Python-based tests to be run across browsers to test the sanity of remote end WebDriver implementations, such as the one in Gecko.

With a little bit more work it can be made to support Servo as well. To do this, the wdspec test type needs to be implemented in wptrunner.executors.executorservo. You can see the implementation for Marionette, the Gecko WebDriver implementation, in the patch at executormarionette.py:512. However, due to external dependency concerns in Gecko, the nosetest runner integration will likely need to swapped out. I will address that later this week when I get back from holiday.

The wdspec harness repurposes @jgraham’s WebDriver client with a few minor adjustments. I expect we will find problems with it as we begin to write tests for the WebDriver specification, but this can be addressed underway. I believe the client offers what we need for writing tests at the level of detail required for specification tests, such as the ability to send raw data and malformed instructions.

To run a test with the wdspec harness in wptrunner:

% wptrunner --metadata . \
  --binary /2/gecko/build/desktop-release/dist/bin/firefox-bin \
  --webdriver-binary /1/wires/target/debug/wires \
  --certutil-binary /2/gecko/build/desktop-release/security/nss/cmd/certutil/certutil \
  --prefs-root /2/gecko/src/testing/profiles/ \
  --tests . \
  --log-mach - \
  --test-types wdspec \
  --pdb --log-mach-level debug \
  --include /1/web-platform-tests/webdriver/smoke.py

The --webdriver-binary, --certutil-binary, and --prefs-root flags are likely not needed with Servo.

A test might look like (written from memory):

def test_session_not_started():
    assert session.session_id == None

def test_window_handle():
    handle = session.window_handle
    assert handle is not None
    assert isinstance(handle, int)
    assert handle > 0

With regards to what @krunal3103 said:

2)Also the automated tests that we need to write to check the validity of the url, do we need to just GET and use assert_equals to check what url has been fetched?

In theory, yes, but note that there are a number of steps for the Get command listed in the specification. Each one of these needs to be tested, if possible.

A non-exhaustive list of examples:

• Malformed body
• Different types for the url field
• With a missing top-level browsing context, that it returns a no such window error
• Missing url field
• HTTP 401 authentication
• Redirects
• Page loading strategies
• …

[jdm]

@andreastt For context, the questions that @krunal3103 asked were specifically about the initial steps in the project description I put together. I'm forgoing having them write wptrunner-compatible tests until I understand how it all fits together.

[andreastt]

@jdm Yes, any testing of the Servo WebDriver server is better than none. I expect it will easy enough to integrate what comes out of this in WPT when the time comes. I should clarify that what I said above is not on master yet. I can pick up the Servo integration after it lands.

[krunal3103]

Thank you for the clarification.
Just one more doubt, after i create a session for each test, how do i get
servo to start? I can do so hy executing the ./mach run --webdriver port
url command, but I'm unsure how do i do that inside a particular test?

On Monday, March 21, 2016, Andreas Tolfsen notifications@github.com wrote:

@jdm https://github.com/jdm Yes, any testing of the Servo WebDriver
server is better than none. I expect it will easy enough to integrate what
comes out of this in WPT when the time comes. I should clarify that what I
said above is not on master yet. I can pick up the Servo integration after
it lands.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5970 (comment)

Regards,
Krunal Gala

[jgraham]

That's part of what wptrunner will handle in the future. For now the best thing is probably to write a little bit of code that uses subprocess to start/stop servo for each test.

[krunal3103]

Okay. So basically for each test I'll create a new session and invoke the
browser with the help of wptrunner. Assert the url returned and end the
session. Am I right here?
I had difficulty invoking the browser from within the test.
When i run ./mach run url it opens the url in the browser. How do I open
the browser from within the test after a session is created. Is there any
specific function/method that I'm missing out here?

On Monday, March 21, 2016, jgraham notifications@github.com wrote:

That's part of what wptrunner will handle in the future. For now the best
thing is probably to write a little bit of code that uses subprocess to
start/stop servo for each test.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#5970 (comment)

Regards,
Krunal Gala

[jdm]

What do you mean by "invoking the browser"? When running the automated tests, there will be no need for the user to run Servo separately, since that will be the responsibility of the python code (using subprocess, as @jgraham mentioned). The browser will need to be started before a session can be created, since a session involves connecting to the server that is running in the browser. Does that make sense?

[jgraham]

I think the talk of wptrunner has confused the issue somewhat :)

I would write something like:

class ServoProcess(object):
     def __init__(self)
          self.path = "/path/to/servo"
          self.proc = None

    def __enter__(self):
         self.proc = subprocess.Popen([self.path, "--webdriver", "4000"])

    def __exit__(self, *args, **kwargs):
        self.proc.kill()

def test_get():
     with ServoProcess():
         with webdriver.Session() as session:
             session.get("http://google.com")
             assert session.url == "http://google.com"

Obviously this is a bit messy, but in the medium term wptrunner will remove most of the boilerplate.

[asajeffrey]

There's now Servo webdriver support in wptrunner: w3c/wptrunner#188. The main remaining issue is that Servo doesn't support multiple sessions: #11253.

[jdm]

Fixed by the support for webdriver tests in wptrunner.