New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multiple jobs per repository #2646

Closed
crunis opened this Issue Aug 7, 2014 · 20 comments

Comments

Projects
None yet
9 participants
@crunis

crunis commented Aug 7, 2014

In big projects, it would be useful to be able to create multiple jobs that test different parts of the project or simply that are more or less exhaustive, so we have small jobs that can run quick tests for PR feedback and other bigger jobs that can be run once a day and have a bigger code coverage.

The ability to run test periodically has already been requested in another feature request.

Great tool ! Keep the good work !

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk Aug 7, 2014

Member

Thanks for the feature request.

Sorry, I'm confused by the feature you are asking for. Can you explain again please.

Member

joshk commented Aug 7, 2014

Thanks for the feature request.

Sorry, I'm confused by the feature you are asking for. Can you explain again please.

@crunis

This comment has been minimized.

Show comment
Hide comment
@crunis

crunis Aug 10, 2014

Sure joshk, thanks for your interest.

Imagine I have a repository called ProjectX . This corresponds to a huge project with lots of contributors and many things to test. Then I have two sets of test, TESTS_A and TESTS_B:

  • TESTS_A is a small set of tests that is perfect for quickly testing a Pull Request and have a feedback to see if it breaks something.
  • TESTS_B is a very thorough set of tests, it gives big confidence that the repository is not broken, but takes 2 hours to execute, so it should only be executed every night.

So I'd like to create two "jobs" in travisCI, each one to execute TESTS_A and the other one to execute TESTS_B, and specify different trigger conditions for each one.

Ideally I would like that the more time expensive TESTS_B to be executed periodically every night, and thats why I mentioned that is another feature request that has already been issued around here.

Is it now more clear?

crunis commented Aug 10, 2014

Sure joshk, thanks for your interest.

Imagine I have a repository called ProjectX . This corresponds to a huge project with lots of contributors and many things to test. Then I have two sets of test, TESTS_A and TESTS_B:

  • TESTS_A is a small set of tests that is perfect for quickly testing a Pull Request and have a feedback to see if it breaks something.
  • TESTS_B is a very thorough set of tests, it gives big confidence that the repository is not broken, but takes 2 hours to execute, so it should only be executed every night.

So I'd like to create two "jobs" in travisCI, each one to execute TESTS_A and the other one to execute TESTS_B, and specify different trigger conditions for each one.

Ideally I would like that the more time expensive TESTS_B to be executed periodically every night, and thats why I mentioned that is another feature request that has already been issued around here.

Is it now more clear?

@Aaron1011

This comment has been minimized.

Show comment
Hide comment
@Aaron1011

Aaron1011 Aug 13, 2014

@crunis You could use environment variables to create jobs that only execute a subset of the tests. As for periodically kicking of a build, the only way to trigger them currently is through a push to GitHub (I think).

Aaron1011 commented Aug 13, 2014

@crunis You could use environment variables to create jobs that only execute a subset of the tests. As for periodically kicking of a build, the only way to trigger them currently is through a push to GitHub (I think).

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Aug 13, 2014

Using environment variables and the build matrix indeed allows to do this. However, if your setup for the tests need to be different, it gives you 2 choices:

  • wrap all setup commands in a check on the environment to decide whether it should run or no. this works fine for small changes, but makes it totally unreadable for bigger differences
  • move the setup to dedicated scripts and swap the script being used based on the environment variable. This works better, but we loose the output of the different commands being executed in the logs, making it harder to know what failed in the setup (and we get a single fold and a single execution time for the whole script).

It could be good to allow defining different sets of before_install/install/scripts and having jobs running one of the other (depending on a setting in the job)

stof commented Aug 13, 2014

Using environment variables and the build matrix indeed allows to do this. However, if your setup for the tests need to be different, it gives you 2 choices:

  • wrap all setup commands in a check on the environment to decide whether it should run or no. this works fine for small changes, but makes it totally unreadable for bigger differences
  • move the setup to dedicated scripts and swap the script being used based on the environment variable. This works better, but we loose the output of the different commands being executed in the logs, making it harder to know what failed in the setup (and we get a single fold and a single execution time for the whole script).

It could be good to allow defining different sets of before_install/install/scripts and having jobs running one of the other (depending on a setting in the job)

@roidrage

This comment has been minimized.

Show comment
Hide comment
@roidrage

roidrage Aug 20, 2014

Member

@stof Thank you for outlining the possible workaround, I think those are a viable solution to the issue.

Member

roidrage commented Aug 20, 2014

@stof Thank you for outlining the possible workaround, I think those are a viable solution to the issue.

@roidrage roidrage closed this Aug 20, 2014

@crunis

This comment has been minimized.

Show comment
Hide comment
@crunis

crunis Aug 21, 2014

Thanks for your feedback!

crunis commented Aug 21, 2014

Thanks for your feedback!

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Mar 5, 2015

@roidrage I discovered an even better way to do this thanks to an undocumented behavior of the matrix configuration: it allows to overwrite the config of build steps (even changing the VM being used). Can you tell me whether it is something which can continue to work in the future or just a side-effect of an implementation detail for which you don't want to preserve compatibility in the future ?
See the change done in qa-tools/qa-tools#125 for the way it works, or https://github.com/Elgg/Elgg/blob/4cf881a071fa5fd73cd6a84e820413845b9a208d/.travis.yml for another usage (which is where I discovered it)

stof commented Mar 5, 2015

@roidrage I discovered an even better way to do this thanks to an undocumented behavior of the matrix configuration: it allows to overwrite the config of build steps (even changing the VM being used). Can you tell me whether it is something which can continue to work in the future or just a side-effect of an implementation detail for which you don't want to preserve compatibility in the future ?
See the change done in qa-tools/qa-tools#125 for the way it works, or https://github.com/Elgg/Elgg/blob/4cf881a071fa5fd73cd6a84e820413845b9a208d/.travis.yml for another usage (which is where I discovered it)

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Mar 5, 2015

Note that I find undocumented feature awesome. It allows to have different job configurations very easily and avoids having lots of unnecessary commands in the build output (which is the drawback of my previous workaround based on commands wrapped in environment variable checks)

stof commented Mar 5, 2015

Note that I find undocumented feature awesome. It allows to have different job configurations very easily and avoids having lots of unnecessary commands in the build output (which is the drawback of my previous workaround based on commands wrapped in environment variable checks)

@thedrow

This comment has been minimized.

Show comment
Hide comment
@thedrow

thedrow Mar 5, 2015

@stof Oh wow this is awesome.
Are you sure it's working as you expect it to?

thedrow commented Mar 5, 2015

@stof Oh wow this is awesome.
Are you sure it's working as you expect it to?

@thedrow

This comment has been minimized.

Show comment
Hide comment
@thedrow

thedrow Mar 5, 2015

I just checked and it seems like it works.
I think we should document this.

thedrow commented Mar 5, 2015

I just checked and it seems like it works.
I think we should document this.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Mar 5, 2015

@thedrow yes, currently it works. However, given it is not documented, I suspect it is a side-effect of the current implementation and may not work anymore in the future. This is why I'm asking @roidrage whether the Travis team would consider making this feature official

stof commented Mar 5, 2015

@thedrow yes, currently it works. However, given it is not documented, I suspect it is a side-effect of the current implementation and may not work anymore in the future. This is why I'm asking @roidrage whether the Travis team would consider making this feature official

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk Mar 5, 2015

Member

I will look into this further.

This is partially expected, and not expected. The syntax is likely to change though, and it is possible we may limit this current syntax once we move to a changed syntax/api.

Thanks for updating this issue though. Please do not promote this just yet as we need to evaluate if this will be support longterm.

Member

joshk commented Mar 5, 2015

I will look into this further.

This is partially expected, and not expected. The syntax is likely to change though, and it is possible we may limit this current syntax once we move to a changed syntax/api.

Thanks for updating this issue though. Please do not promote this just yet as we need to evaluate if this will be support longterm.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Mar 6, 2015

@joshk I suggest reopening the issue to keep track of progress on this

stof commented Mar 6, 2015

@joshk I suggest reopening the issue to keep track of progress on this

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Apr 7, 2015

@joshk any plan to reopen this issue to track the progress ?

stof commented Apr 7, 2015

@joshk any plan to reopen this issue to track the progress ?

@joshk

This comment has been minimized.

Show comment
Hide comment
@joshk

joshk Apr 7, 2015

Member

Hi @stof, I will not reopen this issue at the moment until I have had time to talk to the team and figure out where it sits in our roadmap.

Member

joshk commented Apr 7, 2015

Hi @stof, I will not reopen this issue at the moment until I have had time to talk to the team and figure out where it sits in our roadmap.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Aug 1, 2015

@joshk any news ?

stof commented Aug 1, 2015

@joshk any news ?

@brunoric

This comment has been minimized.

Show comment
Hide comment
@brunoric

brunoric Nov 5, 2015

+1 for the feature (and for the workaround do not be removed in the meantime).

brunoric commented Nov 5, 2015

+1 for the feature (and for the workaround do not be removed in the meantime).

@pietv

This comment has been minimized.

Show comment
Hide comment
@pietv

pietv Apr 11, 2016

Due to the 50 minute limit on MacOSX, multiple jobs feature is very welcome!

pietv commented Apr 11, 2016

Due to the 50 minute limit on MacOSX, multiple jobs feature is very welcome!

@esmail

This comment has been minimized.

Show comment
Hide comment
@esmail

esmail May 24, 2016

@pietv, you might want to check out the "Parallelizing your builds across virtual machines" appraoch. According to the docs: There is no timeout for a build; a build will run as long as all the jobs do as long as each job does not timeout.

esmail commented May 24, 2016

@pietv, you might want to check out the "Parallelizing your builds across virtual machines" appraoch. According to the docs: There is no timeout for a build; a build will run as long as all the jobs do as long as each job does not timeout.

@pietv

This comment has been minimized.

Show comment
Hide comment
@pietv

pietv May 24, 2016

@esmail thanks! yep, solved it along those lines:

env:
  global:
    - ECM_BRANCH=develop
    - EPM_BRANCH=develop
  matrix:
    - SKIP_STACK=true
    - SKIP_PACKAGES=true

So no longer really a problem!

pietv commented May 24, 2016

@esmail thanks! yep, solved it along those lines:

env:
  global:
    - ECM_BRANCH=develop
    - EPM_BRANCH=develop
  matrix:
    - SKIP_STACK=true
    - SKIP_PACKAGES=true

So no longer really a problem!

@sbeyer sbeyer referenced this issue Apr 12, 2017

Merged

More CI builds #33

sbeyer added a commit to banditcpp/snowhouse that referenced this issue Apr 12, 2017

Merge pull request #33 from banditcpp/docker
More CI builds

This pull request adds more builds to AppVeyor and Travis CI.
Different compilers and different standards are tested.
The build environments for Travis CI are realized using Docker images.

A clang-format-4.0-based style check is added to the CI.
This is realized by exploiting undocumented Travis CI behavior, see
travis-ci/travis-ci#2646 (comment)

`cross_compile.sh` is removed and some new shell scripts are
added in a new `util/` directory.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment