Make use of Jenkins for the TPS CI system

[whimboo]

Right now the tool has no queuing mechanism, that means in case of tests are running, we spawn a new process in parallel. Given that we are using a single account at the moment different tests would interfere with each other.

@jonallengriffin shall I find a way to dynamically create a random account via the tps extension which will be removed at the end of the test (if that is possible)?

[jonallengriffin]

It's probably better to add a queuing mechanism. We can also configure the tests to run against m-c for now instead of m-i, which has a much lower probability of generating builds that will cause multiple concurrent runs.

I think this is easier than getting concurrent runs to work: there are lots of assumptions in the code that would need to be changed, e.g., https://github.com/jonallengriffin/coversheet/blob/master/coversheet/subproc.py#L112

[whimboo]

So what about Jenkins here? It's kinda easy to setup and it shouldn't take that long as implementing all that on our own.

[jonallengriffin]

Do we have Jenkins runs for something else that get triggered when builds are available, to model from? I don't object to moving to Jenkins if it's less work.

I don't think it would be too hard to use a separate thread and a Queue to handle the actual test invocation; this would allow the pulse handler to return right away and prevent messages from building up. But if Jenkins is easier, I'd be fine with that too.

[whimboo]

Yes, we use it for mozmill-ci. With a bit of refactoring, and taking some code from mozmill-ci it should be easy to setup.

With Jenkins we have a way to kill jobs, queue new ones, and do all that via a web frontend. No need to login via SSH or VNC.

[jonallengriffin]

Sounds like a good plan then!

[whimboo]

Given that we want to use a couple of nodes and several branches, we will have to run the tests in parallel. Given that we might not want to create a list of existent Firefox accounts, @edmoz gave me a great link to a Python tool which allows us to dynamically create and delete accounts. We should include it in the upcoming CI. It is https://github.com/mozilla/fxa-python-client/