Add multiple jobs per repository

[crunis]

In big projects, it would be useful to be able to create multiple jobs that test different parts of the project or simply that are more or less exhaustive, so we have small jobs that can run quick tests for PR feedback and other bigger jobs that can be run once a day and have a bigger code coverage.

The ability to run test periodically has already been requested in another feature request.

Great tool ! Keep the good work !

[joshk]

Thanks for the feature request.

Sorry, I'm confused by the feature you are asking for. Can you explain again please.

[crunis]

Sure joshk, thanks for your interest.

Imagine I have a repository called ProjectX . This corresponds to a huge project with lots of contributors and many things to test. Then I have two sets of test, TESTS_A and TESTS_B:

• TESTS_A is a small set of tests that is perfect for quickly testing a Pull Request and have a feedback to see if it breaks something.
• TESTS_B is a very thorough set of tests, it gives big confidence that the repository is not broken, but takes 2 hours to execute, so it should only be executed every night.

So I'd like to create two "jobs" in travisCI, each one to execute TESTS_A and the other one to execute TESTS_B, and specify different trigger conditions for each one.

Ideally I would like that the more time expensive TESTS_B to be executed periodically every night, and thats why I mentioned that is another feature request that has already been issued around here.

Is it now more clear?

[Aaron1011]

@crunis You could use environment variables to create jobs that only execute a subset of the tests. As for periodically kicking of a build, the only way to trigger them currently is through a push to GitHub (I think).

[stof]

Using environment variables and the build matrix indeed allows to do this. However, if your setup for the tests need to be different, it gives you 2 choices:

• wrap all setup commands in a check on the environment to decide whether it should run or no. this works fine for small changes, but makes it totally unreadable for bigger differences
• move the setup to dedicated scripts and swap the script being used based on the environment variable. This works better, but we loose the output of the different commands being executed in the logs, making it harder to know what failed in the setup (and we get a single fold and a single execution time for the whole script).

It could be good to allow defining different sets of before_install/install/scripts and having jobs running one of the other (depending on a setting in the job)

[roidrage]

@stof Thank you for outlining the possible workaround, I think those are a viable solution to the issue.

[crunis]

Thanks for your feedback!

[stof]

@roidrage I discovered an even better way to do this thanks to an undocumented behavior of the matrix configuration: it allows to overwrite the config of build steps (even changing the VM being used). Can you tell me whether it is something which can continue to work in the future or just a side-effect of an implementation detail for which you don't want to preserve compatibility in the future ?
See the change done in qa-tools/qa-tools#125 for the way it works, or https://github.com/Elgg/Elgg/blob/4cf881a071fa5fd73cd6a84e820413845b9a208d/.travis.yml for another usage (which is where I discovered it)

[stof]

Note that I find undocumented feature awesome. It allows to have different job configurations very easily and avoids having lots of unnecessary commands in the build output (which is the drawback of my previous workaround based on commands wrapped in environment variable checks)

[thedrow]

@stof Oh wow this is awesome.
Are you sure it's working as you expect it to?

[thedrow]

I just checked and it seems like it works.
I think we should document this.

[stof]

@thedrow yes, currently it works. However, given it is not documented, I suspect it is a side-effect of the current implementation and may not work anymore in the future. This is why I'm asking @roidrage whether the Travis team would consider making this feature official

[joshk]

I will look into this further.

This is partially expected, and not expected. The syntax is likely to change though, and it is possible we may limit this current syntax once we move to a changed syntax/api.

Thanks for updating this issue though. Please do not promote this just yet as we need to evaluate if this will be support longterm.

[stof]

@joshk I suggest reopening the issue to keep track of progress on this

[stof]

@joshk any plan to reopen this issue to track the progress ?

[joshk]

Hi @stof, I will not reopen this issue at the moment until I have had time to talk to the team and figure out where it sits in our roadmap.

[stof]

@joshk any news ?

[brunoric]

+1 for the feature (and for the workaround do not be removed in the meantime).

[pietv]

Due to the 50 minute limit on MacOSX, multiple jobs feature is very welcome!

[esmail]

@pietv, you might want to check out the "Parallelizing your builds across virtual machines" appraoch. According to the docs: There is no timeout for a build; a build will run as long as all the jobs do as long as each job does not timeout.

[pietv]

@esmail thanks! yep, solved it along those lines:

env:
  global:
    - ECM_BRANCH=develop
    - EPM_BRANCH=develop
  matrix:
    - SKIP_STACK=true
    - SKIP_PACKAGES=true

So no longer really a problem!

[keithy]

I don't get it, this is a no-brainer!

I have a bash script, I want to test it in a bash setup.
I also have a set of db containers, I need to test those in a docker environment.
I also have a set of php packages, I need to test those in a php environment.
etc etc.

So, I need separate test setups for each of perhaps 15 scenarios. So .travis.yml .travis.2.yml .travis.3.yml would do this perfectly. Or .travis.bash.yml .travis.php.yaml etc etc.

Hacking the matrix to achieve simple functionality seems too much work.