Use the new Docker queue on Travis

[joshk]

faster better stronger

let's see how it copes with this test suite

[doctrinebot]

Hello,

thank you for creating this pull request. I have automatically opened an issue
on our Jira Bug Tracker for you. See the issue link:

http://www.doctrine-project.org/jira/browse/DDC-3268

We use Jira to track the state of pull requests and the versions they got
included in.

[joshk]

just as I feared, this test suite is disk IO heavy due to chatting to the DBs. the Docker VMs no longer use tmpfs for the DB dirs, which explains the test speed change.

we may have something for this in the future though, stay tuned......

[stof]

@joshk so are you expecting this to be merged, or should it wait for more changes on Travis side ?

[joshk]

i'm not sure tbh, still thinking, maybe ping me next week as I have a phone call today which might give me some answers.

[Ocramius]

@joshk thanks for looking into this btw: the test suite is getting really slow as of lately, and we can't parallelize tests due to the dbms instance being shared across tests.

[joshk]

@Ocramius sorry you need to give me some more context, when was it fast and when did it become slow? Why has it become slow? Why can't you parallelise tests? Why is the dbms instance shared across tests?

I think it would be better to send a support email as this PR is about something different.

[Ocramius]

Why has it become slow?

we started testing more PGSQL versions, as well as an implementation with the second level cache (which is basically a full run of the tests, but with a different flag).

Additionally, the single suite runs used to take about one hour to run (around 1 year ago): now we're reaching 3 hours. The entire suite (excluding installation) takes 8 seconds to run on my machine (i7 box, high end MBP).

Why can't you parallelise tests?

The ORM relies on the DB state, including tests about failed transactions and multiple subsequent transactions. There's no reliable way to parallelize tests that do perform I/O on the DB except for spinning up a container per test, which is just a bad idea.

Why is the dbms instance shared across tests?

Setup is done only once per suite run, as doing it each time would increase run times by way too much, even with docker containers. Yes, this means that tests are responsible for cleanup, and failures may propagate incorrectly, but we don't currently have any failures at all (not even allowed failures).

I think it would be better to send a support email as this PR is about something different.

Well, hopefully moving away from a fully virtualized environment would already solve most performance issues. If the build times become an issue I will mail the travis folks for sure: so far it's just been observations done on the build time trend.

[joshk]

Bigger test suites will take more time, there isn't much we can do to fix that sorry as it is more an issue with the size and complexity of your test suite.

As for parallelising your test suite, there are ways to achieve this, but they are probably fairly complex too. For example, you could create multiple DBs and have different tests use different DBs. That said, this is more of an issue with your test suite. The same issue applies to the 3rd question too.

I am not sure I understand what you mean by moving away from a fully virtualised environment.

[Ocramius]

I am not sure I understand what you mean by moving away from a fully virtualised environment.

Hypervisor vs VM :-)

[joshk]

You mean container based virtualisation vs hypervisor based virtualisation?

You are already using OpenVZ, which is container based virtualisation.

The Docker setup uses LXC, which is a different type of container based virtualisation with some extras added to it. (copy on write)

[Ocramius]

@joshk ah, I was convinced travis was running VMs. My bad then: what's exactly the purpose here then?

[stof]

@Ocramius probably testing their new stack against existing projects

[joshk]

@stof confirm

[Ocramius]

@joshk is this still a thing?

[joshk]

It is, but since the new setup can't do hhvm-nightly at the moment, I'm going to close this for now.

[Ocramius]

@joshk thanks for the update!

[stof]

@Ocramius we should check whether the testsuite is much faster on the docker setup or no. If it is a clear win, we could disable the hhvm-nightly build instead.

[Ocramius]

This was tested some time ago and didn't produce any noticeable performance improvements since we are doing a lot of I/O...

[joshk]

One thing to note though is that we can scale the Docker infrastructure and increase it on demand, something we can't do with the private linux queue cloud.

If the build times are the same, I would recommend switching, as the boot times are 1-2 secs vs 30-120 secs.