Add .travis.yml to run tests from 3.3 to 5.0 on OS X and Linux

[pquentin]

This is an extended version of pull request #295 which adds more LibreOffice versions and extends the tests to OS X. I did not close #295 yet in case you want to keep it instead. Here's the Travis output for this pull request: https://travis-ci.org/pquentin/unoconv/builds/79622720.

For some reason, the tests on OS X don't pass. They're still valuable because they highlight the regression from #292: starting from 4.4, environment is not correctly detected.

I've discussed with @nthiebaud on #tdf-infra who kindly offered to host the continuous integration on a TDF server, which would:

1. make tests much faster since the files would be hosted locally
2. allow to test the master branch to detect issues early
3. prevent hitting the servers too hard.

However, it would be more work to setup and maintain, so starting with Travis might be wiser.

Also, which versions do you want to support? Going back to 3.3 is maybe too much.

[dagwieers]

I like the idea to do automated testing, however I have some concerns:

• the reliability of the tests
• the tests themselves (files, options, etc.)
• the overhead of testing (i.e. downloading/installing LibreOffice)
• the maintenance of the testing scripts over time

All of these concerns can be addressed, or may disappear over time ;-)
What are your thoughts ?

[pquentin]

1. I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
2. Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.
3. Are you talking about the time it takes on Travis or the overhead on TDF servers?

   1. For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

   2. For the servers, here's what @nthiebaud (shm_get on IRC) said:

      based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
      if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
      or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
      that would be 9 downloads per commit

      I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.

4. Without tests we cannot be confident that all options and filters work. I hope that in the long term it will help avoid issues like Cannot find a suitable office installation on Mac OS 10.9.3 #211 and OS X, LibreOffice 4.4 and 5.0: no suitable office installation #292. But if the tests start failing for bad reasons and nobody's here to fix them, then remove them. It's only one file (.travis.yml) and one directory (ci/).

Basically, what I'm saying is:

• the automated tests are easy to remove if the experiment does not work out.
• it can only make things better.

[ghost]

On Fri, Sep 11, 2015 at 12:01 AM, Quentin Pradet
notifications@github.com wrote:

I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.

Are you talking about the time it takes on Travis or the overhead on TDF servers?

For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

For the servers, here's what @nthiebaud (shm_get on IRC) said:

based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
that would be 9 downloads per commit

I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.
the difference is between 'supported version, which are mirrored
copiously and EOL version which are not.

Norbert

[dagwieers]

@pquentin Let's do it. And re-evaluate later.

[pquentin]

@dagwieers Great, thanks!

[pquentin]

@dagwieers You need to set up dagwieers/unoconv on travis, by the way.