[FLINK-13978][build system] Add experimental support for building on Azure Pipelines

[rmetzger]

What is the purpose of the change

This pull request adds support for building and testing Flink on Azure Pipelines.
There are two entry files (build definitions) in this pull request
a) azure-pipelines.yml for the use with free Azure Pipelines accounts (executes all tests with default parameters)
b) tools/azure-pipelines/build-apache-repo.yml is a build definition specifically for the apache/flink repo. It uses a custom build machines through the flink-ci/flink repository. These machines have a lot of performance, thus providing faster build times.

Both entry files rely on the same build job definitions in tools/azure-pipelines/jobs-template.yml.
In addition to that, there are some utilities for running tests on Azure added in this PR. Travis test execution should be unaffected by this change.

Verifying this change

• create a AZP account & setup a AZP pipelines for your Flink fork
• Push the changes in this PR to a branch in your GH repo
• validate that the building for pushes is working
• trigger another build, with "MODE=e2e" to see end to end test execution on AZP.

Note: e2e test execution does currently fail, due to an issue with minikube.

Does this pull request potentially affect one of the following parts:

• this PR only affects the build system, potentially Travis.

Documentation

(work in progress) documentation is available here: https://cwiki.apache.org/confluence/display/FLINK/%5Bpreview%5D+Azure+Pipelines

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 322bf18 (Thu Jan 30 15:17:40 UTC 2020)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• 322bf18 Travis: FAILURE
• 0e9afb2 Travis: FAILURE
• 438cbcb Travis: FAILURE
• 5a81004 Travis: CANCELED
• 4f4267a Travis: SUCCESS
• 0d869f3 Travis: CANCELED
• 56d01e5 Travis: SUCCESS
• 93946ab Travis: SUCCESS Azure: FAILURE
• 2656304 UNKNOWN
• af8c961 Travis: SUCCESS Azure: SUCCESS
• da5a1bf Travis: FAILURE Azure: SUCCESS
• 7c444b5 UNKNOWN

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[zentol]

Did a quick skim, will do a more thorough review another time.

[zentol]

Do we have to make any changes to CiBot?

[rmetzger]

Do we have to make any changes to CiBot?

Not that I'm aware of. If we want to trigger e2e tests through CiBot, the variable "MODE=e2e" needs to be passed. But I can change that to whatever you like.

[zentol]

second round

[rmetzger]

Thanks a lot for your review.
I have fixed the end to end test execution since the last time I responded to your comments.
Once #11067 is merged, all tests should pass on Azure.

Please let me know if you have further comments.
Subsequent changes are in separate commits, for easier review.

[rmetzger]

I believe I have reacted to all your comments. Please let me know if there's more feedback.
I re-enabled the google mirror. Let's see if this build is passing: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5129&view=results

Do we have to make any changes to CiBot?

Maybe you actually need to do a minor change in there, as the name of the builds has changed.

[zentol]

Maybe you actually need to do a minor change in there, as the name of the builds has changed.

Good to know; it's just a simple configuration change that I can already apply. Will they be called like in the build you linked? (compile_ci_build, e2e_ci_build, test_ci_*)

[zentol]

I believe I have reacted to all your comments.

You missed #10976 (comment)

[rmetzger]

I have updated this, the GitHub UI just doesn't show it next to your review comment:

[zentol]

I have seen that, but that's not a reason. You're explaining what you are doing, not why.

[zentol]

[FAIL] 'State Migration end-to-end test from 1.6' failed after 0 minutes and 0 seconds! Test exited with exit code 1

/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/common.sh: line 414: /bin/stop-cluster.sh: No such file or directory
rm: cannot remove '/lib/terminfo/x/xterm-mono': Permission denied
rm: cannot remove '/lib/terminfo/x/xterm-debian': Permission denied

You weren't kidding when you said the tests are invasive :P

[rmetzger]

Maybe you actually need to do a minor change in there, as the name of the builds has changed.

Good to know; it's just a simple configuration change that I can already apply. Will they be called like in the build you linked? (compile_ci_build, e2e_ci_build, test_ci_*)

Yes

[zentol]

I have adjusted CiBot; if everything is setup correctly it should start linking azure builds in this PR for new commits.

[rmetzger]

Awesome, thanks

[zentol]

+1