Restore support for Elasticsearch OSS distributions 7.0.x-7.10.2

[VijayanB]

What does this PR do?

This PR restores the support for Elasticsearch OSS distributions 7.0.x-7.10.2

Why is it important?

Change #25351 had a side effect of removing compatibility with OSS distributions of Elasticsearch 7.0.x-7.10.2. This change restores Beats compatibility outlined in the product compatibility matrix by adding version-awareness into the license check. Also resolves #25865

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Filebeat 7.13 is not compatible with elastichsearch-oss 7.10.2 #25865
• Relates Validate that required functionality in Elasticsearch is available upon initial connection #25351

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #26305 opened

• Reason: The PR is not allowed to run in the CI yet

• Start Time: 2021-06-14T20:32:02.605+0000

• Duration: 4 min 3 sec

• Commit: dc6ceb1

Trends 🧪

Steps errors

Expand to view the steps failures

Load a resource file from a shared library

• Took 0 min 0 sec . View more details on here
• Description: approval-list/elastic/beats.yml

Error signal

• Took 0 min 0 sec . View more details on here
• Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)

Log output

Expand to view the last 100 lines of log output

[2021-06-14T20:34:06.573Z]  > git config --get remote.origin.url # timeout=10
[2021-06-14T20:34:06.577Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-14T20:34:06.580Z]  > git merge cefaeafb0004a4c35cae6fde1e69d61f6fa05dc1 # timeout=10
[2021-06-14T20:34:06.589Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-14T20:34:06.596Z]  > git config core.sparsecheckout # timeout=10
[2021-06-14T20:34:06.599Z]  > git checkout -f dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa # timeout=15
[2021-06-14T20:34:10.975Z] Commit message: "Restore support for Elasticsearch OSS distributions 7.0.x-7.10.2"
[2021-06-14T20:34:10.975Z] First time build. Skipping changelog.
[2021-06-14T20:34:10.976Z] Cleaning workspace
[2021-06-14T20:34:11.130Z]  > git --version # timeout=10
[2021-06-14T20:34:11.132Z]  > git --version # 'git version 2.17.1'
[2021-06-14T20:34:11.136Z] fatal: bad object bbed8ab717206e63fa6cc1cef7d35f4a13f717ab
[2021-06-14T20:34:10.978Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-14T20:34:10.982Z] Resetting working tree
[2021-06-14T20:34:10.982Z]  > git reset --hard # timeout=10
[2021-06-14T20:34:11.058Z]  > git clean -fdx # timeout=10
[2021-06-14T20:34:12.091Z] Timeout set to expire in 3 hr 0 min
[2021-06-14T20:34:12.129Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-06-14T20:34:12.511Z] [INFO] Number of builds to be searched 10
[2021-06-14T20:34:13.281Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-06-14T20:34:13.310Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26305@tmp
[2021-06-14T20:34:13.357Z] [INFO] gitCheckout: Checkout SCM PR-26305 with default customisation from the Item.
[2021-06-14T20:34:13.404Z] [INFO] Override default checkout
[2021-06-14T20:34:13.488Z] Sleeping for 10 sec
[2021-06-14T20:34:23.523Z] The recommended git tool is: git
[2021-06-14T20:34:23.570Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-06-14T20:34:23.580Z] Wiping out workspace first.
[2021-06-14T20:34:23.587Z] Cloning the remote Git repository
[2021-06-14T20:34:23.587Z] Using shallow clone with depth 10
[2021-06-14T20:34:23.587Z] Avoid fetching tags
[2021-06-14T20:34:23.601Z] Cloning repository git@github.com:elastic/beats.git
[2021-06-14T20:34:23.625Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-26305@tmp # timeout=10
[2021-06-14T20:34:23.630Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-06-14T20:34:23.630Z]  > git --version # timeout=10
[2021-06-14T20:34:23.634Z]  > git --version # 'git version 2.17.1'
[2021-06-14T20:34:23.634Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-14T20:34:23.638Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-06-14T20:34:43.309Z] Cleaning workspace
[2021-06-14T20:34:43.320Z] Using shallow fetch with depth 10
[2021-06-14T20:34:43.320Z] Pruning obsolete local branches
[2021-06-14T20:34:44.243Z] Merging remotes/origin/master commit cefaeafb0004a4c35cae6fde1e69d61f6fa05dc1 into PR head commit dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa
[2021-06-14T20:34:43.296Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-06-14T20:34:43.299Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-06-14T20:34:43.306Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-06-14T20:34:43.311Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-14T20:34:43.314Z] No valid HEAD. Skipping the resetting
[2021-06-14T20:34:43.314Z]  > git clean -fdx # timeout=10
[2021-06-14T20:34:43.322Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-06-14T20:34:43.322Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-14T20:34:43.325Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/26305/head:refs/remotes/origin/PR-26305 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-06-14T20:34:44.246Z]  > git config core.sparsecheckout # timeout=10
[2021-06-14T20:34:44.249Z]  > git checkout -f dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa # timeout=15
[2021-06-14T20:34:45.733Z] Merge succeeded, producing dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa
[2021-06-14T20:34:45.733Z] Checking out Revision dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa (PR-26305)
[2021-06-14T20:34:46.866Z] Commit message: "Restore support for Elasticsearch OSS distributions 7.0.x-7.10.2"
[2021-06-14T20:34:46.866Z] Cleaning workspace
[2021-06-14T20:34:47.015Z]  > git --version # timeout=10
[2021-06-14T20:34:47.018Z]  > git --version # 'git version 2.17.1'
[2021-06-14T20:34:47.021Z] fatal: bad object bbed8ab717206e63fa6cc1cef7d35f4a13f717ab
[2021-06-14T20:34:45.708Z]  > git remote # timeout=10
[2021-06-14T20:34:45.715Z]  > git config --get remote.origin.url # timeout=10
[2021-06-14T20:34:45.718Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-14T20:34:45.721Z]  > git merge cefaeafb0004a4c35cae6fde1e69d61f6fa05dc1 # timeout=10
[2021-06-14T20:34:45.729Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-14T20:34:45.735Z]  > git config core.sparsecheckout # timeout=10
[2021-06-14T20:34:45.738Z]  > git checkout -f dc6ceb10fd75fcbaa73cf6616b7b22e4488f51fa # timeout=15
[2021-06-14T20:34:46.868Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-14T20:34:46.871Z] Resetting working tree
[2021-06-14T20:34:46.871Z]  > git reset --hard # timeout=10
[2021-06-14T20:34:46.948Z]  > git clean -fdx # timeout=10
[2021-06-14T20:34:48.658Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-06-14T20:34:49.302Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-06-14T20:35:57.138Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26305@tmp/.git
[2021-06-14T20:35:57.399Z] Archiving artifacts
[2021-06-14T20:35:58.186Z] + git rev-parse HEAD
[2021-06-14T20:35:58.531Z] + git rev-parse HEAD
[2021-06-14T20:35:58.848Z] + git rev-parse origin/pr/26305
[2021-06-14T20:35:58.923Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-06-14T20:35:59.414Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-06-14T20:36:00.902Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/26305/reviews return 0 elements
[2021-06-14T20:36:00.985Z] [INFO] githubPrCheckApproved: Title: Restore support for Elasticsearch OSS distributions 7.0.x-7.10.2 - User: VijayanB - Author Association: FIRST_TIME_CONTRIBUTOR
[2021-06-14T20:36:01.399Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet
[2021-06-14T20:36:01.399Z] ERROR: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.)
[2021-06-14T20:36:01.500Z] [INFO] Let's stop build #1. The PR is not allowed to run in the CI yet
[2021-06-14T20:36:01.527Z] Sleeping for 5 sec
[2021-06-14T20:36:02.924Z] Stage "Lint" skipped due to earlier failure(s)
[2021-06-14T20:36:03.042Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-06-14T20:36:03.138Z] Stage "Extended" skipped due to earlier failure(s)
[2021-06-14T20:36:03.234Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-14T20:36:03.332Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-14T20:36:03.478Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26305/src/github.com/elastic/beats
[2021-06-14T20:36:04.005Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26305
[2021-06-14T20:36:04.416Z] [INFO] getVaultSecret: Getting secrets
[2021-06-14T20:36:04.472Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-14T20:36:05.557Z] + chmod 755 generate-build-data.sh
[2021-06-14T20:36:05.557Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26305/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26305/runs/1 ABORTED 242667
[2021-06-14T20:36:05.557Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26305/runs/1/steps/?limit=10000 -o steps-info.json
[2021-06-14T20:36:05.807Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26305/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-06-14T20:36:05.807Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-06-14T20:36:06.760Z] Retry 2/3 exited 22, retrying in 2 seconds...

[ph]

We introduced that check into the default distribution of Beats several years ago, as a simple way to check that the expected features would be available, so we could have good defaults that take advantage of all the free and open features in Elasticsearch, like ILM, new data types in Elasticsearch required by the Elastic Common Schema, integrated monitoring, etc.. Going forward, that check allows us to ensure that all users of Beats can take advantage of all the free & open features available in Elasticsearch. This applies to new versions of course, but it also enables us to unlock support for all the free & open features that have been available in Elasticsearch in 6.x and 7.0+.

If you are using the oss-only distribution of Elasticsearch 7.10 or earlier, I would encourage you to upgrade to the default distribution, which is free and open and has significantly more functionality. It’s under the Elastic License 2.0, which allows free use, redistribution, modification, derivative works, with only a few simple limitations. If you can’t upgrade for some reason, you can continue to use older releases of Beats.