Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-13102][travis] Optimize some travis stages by skipping mvn verify if possible #9065

Open
wants to merge 4 commits into
base: master
from

Conversation

@1u0
Copy link
Contributor

commented Jul 10, 2019

What is the purpose of the change

This PR minimizes total Travis CI time for pull request triggered builds by skipping mvn verify steps for non python and misc jobs in Travis test stage.

Brief change log

  • added script to build mvn dependencies tree and list all changed files in a PR build run. This happens at the very first compile stage;
  • added script to detect if mvn verify can be skipped for given list of maven modules and list of changed files.

Verifying this change

This change can be verified as follows:

  • run different PRs (build in Travis CI) with this changes included and manually verifying that build is correct and some tests are not run because changes in a PR are not affecting them.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
@flinkbot

This comment has been minimized.

Copy link

commented Jul 10, 2019

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 2e0e53b (Tue Aug 06 15:40:31 UTC 2019)

Warnings:

  • No documentation files were touched! Remember to keep the Flink docs up to date!

Mention the bot in a comment to re-run the automated checks.

Review Progress

  • 1. The [description] looks good.
  • 2. There is [consensus] that the contribution should go into to Flink.
  • 3. Needs [attention] from.
  • 4. The change fits into the overall [architecture].
  • 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.


The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands
The @flinkbot bot supports the following commands:

  • @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
  • @flinkbot approve all to approve all aspects
  • @flinkbot approve-until architecture to approve everything until architecture
  • @flinkbot attention @username1 [@username2 ..] to require somebody's attention
  • @flinkbot disapprove architecture to remove an approval you gave earlier
@1u0

This comment has been minimized.

Copy link
Contributor Author

commented Jul 10, 2019

CC: @zentol

Some notes:

With this PR, there is a code duplication in tools/travis/stage.sh and tools/travis/stages.py (they both have mapping which maven modules to build for a given stage). I want to remove it (my personal preference keeping python version), but would like to check first if python is fine for others.

The dependencies detection heuristic relies on that inner module (artifacts) dependencies in Flink are explicit (declared in pom.xml files).
The dependencies check can be further improved by taking into account maven test and non-test dependencies. But for now, the implementation in favor of simplicity and to avoid false skippings (it's ok if we run tests, even if changes are not relevant; it's not ok, if we don't run tests, if changes are relevant and detection missed it).

@1u0 1u0 force-pushed the 1u0:flink-13102-travis-mvn-test-optimization branch from a9b1441 to a25a575 Jul 10, 2019

@1u0 1u0 force-pushed the 1u0:flink-13102-travis-mvn-test-optimization branch from a25a575 to 2e0e53b Jul 10, 2019

@flinkbot

This comment has been minimized.

Copy link

commented Jul 10, 2019

CI report for commit a9b1441: FAILURE Build

@flinkbot

This comment has been minimized.

Copy link

commented Jul 11, 2019

CI report for commit a25a575: FAILURE Build

@flinkbot

This comment has been minimized.

Copy link

commented Jul 11, 2019

CI report for commit 2e0e53b: FAILURE Build

@zentol zentol self-assigned this Jul 11, 2019

@1u0

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2019

@zentol, fyi, unfortunately, this optimization doesn't take effect for the new CI setup (running via /flink-ci/flink), because the new setup performs builds as standalone branches rather than as PRs (from Travis CI point of view).

The TRAVIS_COMMIT_RANGE environment variable would be empty in that case. And I don't know yet how to figure out git commits ranges to list changed files.

@zentol

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2019

this isn't the only issue in regards to the new setup; the is_pr_build function won't work as expected.

I have ideas for fixing both issues, but I'll need some time to figure things out, and I'm not sure if I will have it this or next week with the release coming up.

@flinkbot

This comment has been minimized.

Copy link

commented Aug 6, 2019

CI report:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.