Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multiple jobs per repository #2646

Closed
crunis opened this issue Aug 7, 2014 · 21 comments
Closed

Add multiple jobs per repository #2646

crunis opened this issue Aug 7, 2014 · 21 comments

Comments

@crunis
Copy link

@crunis crunis commented Aug 7, 2014

In big projects, it would be useful to be able to create multiple jobs that test different parts of the project or simply that are more or less exhaustive, so we have small jobs that can run quick tests for PR feedback and other bigger jobs that can be run once a day and have a bigger code coverage.

The ability to run test periodically has already been requested in another feature request.

Great tool ! Keep the good work !

@joshk

This comment has been minimized.

Copy link
Member

@joshk joshk commented Aug 7, 2014

Thanks for the feature request.

Sorry, I'm confused by the feature you are asking for. Can you explain again please.

@crunis

This comment has been minimized.

Copy link
Author

@crunis crunis commented Aug 10, 2014

Sure joshk, thanks for your interest.

Imagine I have a repository called ProjectX . This corresponds to a huge project with lots of contributors and many things to test. Then I have two sets of test, TESTS_A and TESTS_B:

  • TESTS_A is a small set of tests that is perfect for quickly testing a Pull Request and have a feedback to see if it breaks something.
  • TESTS_B is a very thorough set of tests, it gives big confidence that the repository is not broken, but takes 2 hours to execute, so it should only be executed every night.

So I'd like to create two "jobs" in travisCI, each one to execute TESTS_A and the other one to execute TESTS_B, and specify different trigger conditions for each one.

Ideally I would like that the more time expensive TESTS_B to be executed periodically every night, and thats why I mentioned that is another feature request that has already been issued around here.

Is it now more clear?

@Aaron1011

This comment has been minimized.

Copy link

@Aaron1011 Aaron1011 commented Aug 13, 2014

@crunis You could use environment variables to create jobs that only execute a subset of the tests. As for periodically kicking of a build, the only way to trigger them currently is through a push to GitHub (I think).

@stof

This comment has been minimized.

Copy link

@stof stof commented Aug 13, 2014

Using environment variables and the build matrix indeed allows to do this. However, if your setup for the tests need to be different, it gives you 2 choices:

  • wrap all setup commands in a check on the environment to decide whether it should run or no. this works fine for small changes, but makes it totally unreadable for bigger differences
  • move the setup to dedicated scripts and swap the script being used based on the environment variable. This works better, but we loose the output of the different commands being executed in the logs, making it harder to know what failed in the setup (and we get a single fold and a single execution time for the whole script).

It could be good to allow defining different sets of before_install/install/scripts and having jobs running one of the other (depending on a setting in the job)

@roidrage

This comment has been minimized.

Copy link
Contributor

@roidrage roidrage commented Aug 20, 2014

@stof Thank you for outlining the possible workaround, I think those are a viable solution to the issue.

@roidrage roidrage closed this Aug 20, 2014
@crunis

This comment has been minimized.

Copy link
Author

@crunis crunis commented Aug 21, 2014

Thanks for your feedback!

@stof

This comment has been minimized.

Copy link

@stof stof commented Mar 5, 2015

@roidrage I discovered an even better way to do this thanks to an undocumented behavior of the matrix configuration: it allows to overwrite the config of build steps (even changing the VM being used). Can you tell me whether it is something which can continue to work in the future or just a side-effect of an implementation detail for which you don't want to preserve compatibility in the future ?
See the change done in qa-tools/qa-tools#125 for the way it works, or https://github.com/Elgg/Elgg/blob/4cf881a071fa5fd73cd6a84e820413845b9a208d/.travis.yml for another usage (which is where I discovered it)

@stof

This comment has been minimized.

Copy link

@stof stof commented Mar 5, 2015

Note that I find undocumented feature awesome. It allows to have different job configurations very easily and avoids having lots of unnecessary commands in the build output (which is the drawback of my previous workaround based on commands wrapped in environment variable checks)

@thedrow

This comment has been minimized.

Copy link

@thedrow thedrow commented Mar 5, 2015

@stof Oh wow this is awesome.
Are you sure it's working as you expect it to?

@thedrow

This comment has been minimized.

Copy link

@thedrow thedrow commented Mar 5, 2015

I just checked and it seems like it works.
I think we should document this.

@stof

This comment has been minimized.

Copy link

@stof stof commented Mar 5, 2015

@thedrow yes, currently it works. However, given it is not documented, I suspect it is a side-effect of the current implementation and may not work anymore in the future. This is why I'm asking @roidrage whether the Travis team would consider making this feature official

@joshk

This comment has been minimized.

Copy link
Member

@joshk joshk commented Mar 5, 2015

I will look into this further.

This is partially expected, and not expected. The syntax is likely to change though, and it is possible we may limit this current syntax once we move to a changed syntax/api.

Thanks for updating this issue though. Please do not promote this just yet as we need to evaluate if this will be support longterm.

@stof

This comment has been minimized.

Copy link

@stof stof commented Mar 6, 2015

@joshk I suggest reopening the issue to keep track of progress on this

@stof

This comment has been minimized.

Copy link

@stof stof commented Apr 7, 2015

@joshk any plan to reopen this issue to track the progress ?

@joshk

This comment has been minimized.

Copy link
Member

@joshk joshk commented Apr 7, 2015

Hi @stof, I will not reopen this issue at the moment until I have had time to talk to the team and figure out where it sits in our roadmap.

@stof

This comment has been minimized.

Copy link

@stof stof commented Aug 1, 2015

@joshk any news ?

@brunoric

This comment has been minimized.

Copy link

@brunoric brunoric commented Nov 5, 2015

+1 for the feature (and for the workaround do not be removed in the meantime).

@pietv

This comment has been minimized.

Copy link

@pietv pietv commented Apr 11, 2016

Due to the 50 minute limit on MacOSX, multiple jobs feature is very welcome!

@esmail

This comment has been minimized.

Copy link

@esmail esmail commented May 24, 2016

@pietv, you might want to check out the "Parallelizing your builds across virtual machines" appraoch. According to the docs: There is no timeout for a build; a build will run as long as all the jobs do as long as each job does not timeout.

@pietv

This comment has been minimized.

Copy link

@pietv pietv commented May 24, 2016

@esmail thanks! yep, solved it along those lines:

env:
  global:
    - ECM_BRANCH=develop
    - EPM_BRANCH=develop
  matrix:
    - SKIP_STACK=true
    - SKIP_PACKAGES=true

So no longer really a problem!

@sbeyer sbeyer mentioned this issue Apr 12, 2017
sbeyer added a commit to banditcpp/snowhouse that referenced this issue Apr 12, 2017
More CI builds

This pull request adds more builds to AppVeyor and Travis CI.
Different compilers and different standards are tested.
The build environments for Travis CI are realized using Docker images.

A clang-format-4.0-based style check is added to the CI.
This is realized by exploiting undocumented Travis CI behavior, see
travis-ci/travis-ci#2646 (comment)

`cross_compile.sh` is removed and some new shell scripts are
added in a new `util/` directory.
@keithy

This comment has been minimized.

Copy link

@keithy keithy commented Jan 13, 2019

I don't get it, this is a no-brainer!

I have a bash script, I want to test it in a bash setup.
I also have a set of db containers, I need to test those in a docker environment.
I also have a set of php packages, I need to test those in a php environment.
etc etc.

So, I need separate test setups for each of perhaps 15 scenarios. So .travis.yml .travis.2.yml .travis.3.yml would do this perfectly. Or .travis.bash.yml .travis.php.yaml etc etc.

Hacking the matrix to achieve simple functionality seems too much work.

BrainBlasted added a commit to writeas/writeas-gtk that referenced this issue Aug 9, 2019
Based on the tips from travis-ci/travis-ci#2646

For T678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.