New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add .travis.yml to run tests from 3.3 to 5.0 on OS X and Linux #296

Merged
merged 3 commits into from Sep 16, 2015

Conversation

Projects
None yet
2 participants
@pquentin
Contributor

pquentin commented Sep 10, 2015

This is an extended version of pull request #295 which adds more LibreOffice versions and extends the tests to OS X. I did not close #295 yet in case you want to keep it instead. Here's the Travis output for this pull request: https://travis-ci.org/pquentin/unoconv/builds/79622720.

For some reason, the tests on OS X don't pass. They're still valuable because they highlight the regression from #292: starting from 4.4, environment is not correctly detected.

I've discussed with @nthiebaud on #tdf-infra who kindly offered to host the continuous integration on a TDF server, which would:

  1. make tests much faster since the files would be hosted locally
  2. allow to test the master branch to detect issues early
  3. prevent hitting the servers too hard.

However, it would be more work to setup and maintain, so starting with Travis might be wiser.

Also, which versions do you want to support? Going back to 3.3 is maybe too much.

@dagwieers

This comment has been minimized.

Show comment
Hide comment
@dagwieers

dagwieers Sep 10, 2015

Owner

I like the idea to do automated testing, however I have some concerns:

  • the reliability of the tests
  • the tests themselves (files, options, etc.)
  • the overhead of testing (i.e. downloading/installing LibreOffice)
  • the maintenance of the testing scripts over time

All of these concerns can be addressed, or may disappear over time ;-)
What are your thoughts ?

Owner

dagwieers commented Sep 10, 2015

I like the idea to do automated testing, however I have some concerns:

  • the reliability of the tests
  • the tests themselves (files, options, etc.)
  • the overhead of testing (i.e. downloading/installing LibreOffice)
  • the maintenance of the testing scripts over time

All of these concerns can be addressed, or may disappear over time ;-)
What are your thoughts ?

@pquentin

This comment has been minimized.

Show comment
Hide comment
@pquentin

pquentin Sep 11, 2015

Contributor
  1. I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
  2. Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.
  3. Are you talking about the time it takes on Travis or the overhead on TDF servers?
    1. For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

    2. For the servers, here's what @nthiebaud (shm_get on IRC) said:

      based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
      if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
      or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
      that would be 9 downloads per commit

      I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.

  4. Without tests we cannot be confident that all options and filters work. I hope that in the long term it will help avoid issues like #211 and #292. But if the tests start failing for bad reasons and nobody's here to fix them, then remove them. It's only one file (.travis.yml) and one directory (ci/).

Basically, what I'm saying is:

  • the automated tests are easy to remove if the experiment does not work out.
  • it can only make things better.
Contributor

pquentin commented Sep 11, 2015

  1. I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
  2. Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.
  3. Are you talking about the time it takes on Travis or the overhead on TDF servers?
    1. For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

    2. For the servers, here's what @nthiebaud (shm_get on IRC) said:

      based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
      if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
      or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
      that would be 9 downloads per commit

      I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.

  4. Without tests we cannot be confident that all options and filters work. I hope that in the long term it will help avoid issues like #211 and #292. But if the tests start failing for bad reasons and nobody's here to fix them, then remove them. It's only one file (.travis.yml) and one directory (ci/).

Basically, what I'm saying is:

  • the automated tests are easy to remove if the experiment does not work out.
  • it can only make things better.
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 11, 2015

On Fri, Sep 11, 2015 at 12:01 AM, Quentin Pradet
notifications@github.com wrote:

I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.

Are you talking about the time it takes on Travis or the overhead on TDF servers?

For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

For the servers, here's what @nthiebaud (shm_get on IRC) said:

based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
that would be 9 downloads per commit

I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.
the difference is between 'supported version, which are mirrored
copiously and EOL version which are not.

Norbert

ghost commented Sep 11, 2015

On Fri, Sep 11, 2015 at 12:01 AM, Quentin Pradet
notifications@github.com wrote:

I launched the tests many times in two days, and I only got one unexpected Linux failure and that was for an old LibreOffice version, so the tests don't appear to be flaky.
Sure, there's a lot of room for improvement here since I'm only testing basic functionality. In the future it would be great to add unit tests, check that the conversion went well, etc.

Are you talking about the time it takes on Travis or the overhead on TDF servers?

For Travis, downloading LibreOffice does slow down the tests, especially for the old versions (~ 1MB/s instead of 10MB/s). But < 1 minute for current versions and < 5 minutes for old versions is still reasonable, I think.

For the servers, here's what @nthiebaud (shm_get on IRC) said:

based on the activity of the underlying project... 6 downloads per commit will really not be a problem.
if you limit the version downloaded to not EOL version I'm sure that should not cause to much trouble even with the current travis setup.
or maybe non-EOL + 1 like the latest 4.3 4.4 and 5.0
that would be 9 downloads per commit

I believe that if 9 downloads are not an issue, then 17 downloads will be fine too. But if you agree, we can remove some of the old versions that are not in RHEL or Debian stable.
the difference is between 'supported version, which are mirrored
copiously and EOL version which are not.

Norbert

@dagwieers

This comment has been minimized.

Show comment
Hide comment
@dagwieers

dagwieers Sep 16, 2015

Owner

@pquentin Let's do it. And re-evaluate later.

Owner

dagwieers commented Sep 16, 2015

@pquentin Let's do it. And re-evaluate later.

dagwieers added a commit that referenced this pull request Sep 16, 2015

Merge pull request #296 from pquentin/ci
Add .travis.yml to run tests from 3.3 to 5.0 on OS X and Linux

@dagwieers dagwieers merged commit f86cd16 into dagwieers:master Sep 16, 2015

@pquentin

This comment has been minimized.

Show comment
Hide comment
@pquentin

pquentin Sep 16, 2015

Contributor

@dagwieers Great, thanks!

Contributor

pquentin commented Sep 16, 2015

@dagwieers Great, thanks!

@pquentin

This comment has been minimized.

Show comment
Hide comment
@pquentin

pquentin Sep 16, 2015

Contributor

@dagwieers You need to set up dagwieers/unoconv on travis, by the way.

Contributor

pquentin commented Sep 16, 2015

@dagwieers You need to set up dagwieers/unoconv on travis, by the way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment