Test each integration with the oldest supported version #1541

jsoriano · 2021-08-18T12:08:21Z

What does this PR do?

Test each package with the oldest supported version of the stack. This will help making the builds a bit more deterministic, while ensuring that no changes are introduced that break the condition of supported version.

If the version doesn't exist yet, it will try to use the closer snapshot, for example if 7.14.1 doesn't exist, it will try to use 7.14.1-SNAPSHOT or 7.x-SNAPSHOT, this will allow to develop integrations with unreleased features.

Some packages using 7.14.0 have been modified to require at least 7.14.1 as they were affected by elastic/beats#27299.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2021-08-18T14:09:11Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-26T10:35:16.425+0000
Duration: 70 min 37 sec
Commit: e1722f5

Test stats 🧪

Test	Results
Failed	0
Passed	2896
Skipped	3
Total	2899

Trends 🧪

…version

elasticmachine · 2021-08-23T17:00:34Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-08-23T17:00:35Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

ruflin

I really like the idea that we test with the oldest version supported.

Later on, I wonder if we could also do a test run with the newest version supported. This might be unstable and maybe only run ones every day / week to show where we are standing.

ruflin · 2021-08-24T06:49:28Z

packages/bluecoat/manifest.yml

 description: This Elastic integration collects Blue Coat Director logs
 categories: ["network", "security"]
 release: experimental
 license: basic
 type: integration
 conditions:
-  kibana.version: "^7.14.0"
+  kibana.version: "^7.14.1"


Is this change by itention? Same for the other packaegs?

Yes, this is intentional, these packages fail to install in 7.14.0, probably due to elastic/beats#27299.

Got it. Sounds like we should start working on elastic_agent.version: ... as this one is not Kibana related.

Yes, I think we are going to need conditions for elastic agent soon, but this will imply more changes.

jsoriano · 2021-08-24T07:49:00Z

Later on, I wonder if we could also do a test run with the newest version supported. This might be unstable and maybe only run ones every day / week to show where we are standing.

Yep, agree, I was thinking on adding a jenkins parameter so we can launch all the tests with an specific version, this would allow for example to run the tests against a build candidate. This could be run with master/7.x on a daily basis.

.ci/Jenkinsfile

endorama · 2021-08-24T14:30:35Z

.ci/Jenkinsfile

+}
+
+def findStackVersion(String versionCondition) {
+  def version = versionCondition.substring(1)


Is this line supposed to remove the initial v from the version string? I would add a note about it's behaviour.

Agree, this is a bit obscure. Victor is moving this function to the shared library and we are being more explicit about this there (explicitly checking that first character is ^, documenting that we only support this operator here and testing this expectation). You can find it here: elastic/apm-pipeline-library#1248

endorama · 2021-08-24T14:34:19Z

.ci/Jenkinsfile

+  // Old minors may not be available in artifacts-api, if it is older
+  // than the others in the same major, return the version as is.


It is not clear to me what is happening here and why returning the version as is is the correct choice.
Does artifacts-api delete all minors apart from latest? (so for example it keeps 7.x and not 7.x-1)

Yes, older versions are not available in this API, you can see it here: https://artifacts-api.elastic.co/v1/versions. For example 7.13.0 is not there now.

In these cases I am assuming that these older versions existed, and their docker images are available.

.ci/Jenkinsfile

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

…version

…to integrations-stack-version

aspacca · 2021-08-26T11:02:24Z

.ci/Jenkinsfile

+  if (manifest != null && manifest["conditions"] != null) {
+    def kibanaVersionCondition = manifest["conditions"]["kibana.version"]
+    if (kibanaVersionCondition != null) {
+      def version = findOldestSupportedVersion(versionCondition: kibanaVersionCondition)


I cannot find findOldestSupportedVersion, where is it defined?

See https://github.com/elastic/apm-pipeline-library/blob/master/vars/findOldestSupportedVersion.groovy that was originally copied from 6f60c8d

We use a shared library for jenkinsfiles, this function has been moved there: elastic/apm-pipeline-library#1248

aspacca · 2021-08-26T12:19:18Z

prepareStack() LGTM, cannot say about conditions: kibana.version bump, I assume it is correct as well

jsoriano · 2021-08-26T13:01:57Z

cannot say about conditions: kibana.version bump, I assume it is correct as well

This is needed for packages that fail to install in 7.14.0 (see this comment #1541 (comment)).

Test each package with the oldest supported version of the stack. This will help making the builds a bit more deterministic, while ensuring that no changes are introduced that break the condition of supported version. If the version doesn't exist yet, it will try to use the closer snapshot, for example if 7.14.1 doesn't exist, it will try to use 7.14.1-SNAPSHOT or 7.x-SNAPSHOT, this will allow to develop integrations with unreleased features. Some packages using 7.14.0 have been modified to require at least 7.14.1 as they were affected by elastic/beats#27299.

jsoriano force-pushed the integrations-stack-version branch 2 times, most recently from f708e3c to 1b79f82 Compare August 18, 2021 12:15

Test each integration with the oldest supported version

10afe79

jsoriano force-pushed the integrations-stack-version branch from 1b79f82 to 10afe79 Compare August 18, 2021 12:16

Use readYaml to read the yaml

3157bb6

jsoriano force-pushed the integrations-stack-version branch 4 times, most recently from ce1d34c to beaf3e1 Compare August 18, 2021 15:58

Try to look for snapshot versions

aa483ef

jsoriano force-pushed the integrations-stack-version branch from beaf3e1 to aa483ef Compare August 18, 2021 16:00

jsoriano added 18 commits August 23, 2021 15:46

Bluecoat

f57e660

Cisco

55d5bb3

Cyberark

a752fc3

Cylance

33a7fbc

F5

1febf88

Fortinet

c4146ad

Imperva

444fcfd

Infoblox

855fd24

Juniper

bdd3fc9

Microsoft

04a5a64

Netscout

07b5e84

Proofpoint

1b59a3c

sonicwall

03eaaa0

Sophos

b7b9b26

Squid

cbcde35

Tomcat

db54b8a

ZScaler

083c4be

Merge remote-tracking branch 'origin/master' into integrations-stack-…

eda441c

…version

jsoriano added the Team:Security-External Integrations label Aug 23, 2021

jsoriano marked this pull request as ready for review August 23, 2021 17:00

jsoriano requested review from ruflin and andrewkroh August 23, 2021 17:00

Fix squid

76570a2

jsoriano force-pushed the integrations-stack-version branch from 0229fcd to 76570a2 Compare August 23, 2021 17:14

ruflin reviewed Aug 24, 2021

View reviewed changes

network_traffic

4036cff

jsoriano requested a review from a team August 24, 2021 11:15

v1v mentioned this pull request Aug 24, 2021

findOldestSupportedVersion step elastic/apm-pipeline-library#1248

Merged

v1v reviewed Aug 24, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

endorama reviewed Aug 24, 2021

View reviewed changes

v1v reviewed Aug 24, 2021

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

.ci/Jenkinsfile Outdated Show resolved Hide resolved

jsoriano and others added 4 commits August 24, 2021 17:29

Remove function moved to shared library.

6f60c8d

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Update function name.

740772e

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Merge remote-tracking branch 'origin/master' into integrations-stack-…

77b1e13

…version

Merge remote-tracking branch 'jsoriano/integrations-stack-version' in…

e1722f5

…to integrations-stack-version

aspacca reviewed Aug 26, 2021

View reviewed changes

aspacca approved these changes Aug 26, 2021

View reviewed changes

jsoriano merged commit f50eb7c into elastic:master Aug 26, 2021

jsoriano deleted the integrations-stack-version branch August 26, 2021 13:18

This was referenced Sep 7, 2021

Add daily jenkins pipeline to run tests with latest snapshots #1581

Closed

Add parameter to choose stack version to use in CI #1585

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test each integration with the oldest supported version #1541

Test each integration with the oldest supported version #1541

jsoriano commented Aug 18, 2021 •

edited

Loading

elasticmachine commented Aug 18, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

elasticmachine commented Aug 23, 2021

elasticmachine commented Aug 23, 2021

ruflin left a comment

ruflin Aug 24, 2021

jsoriano Aug 24, 2021

ruflin Aug 24, 2021

jsoriano Aug 24, 2021

jsoriano commented Aug 24, 2021

endorama Aug 24, 2021 •

edited

Loading

jsoriano Aug 24, 2021

endorama Aug 24, 2021

jsoriano Aug 24, 2021

aspacca Aug 26, 2021

v1v Aug 26, 2021

jsoriano Aug 26, 2021

aspacca Aug 26, 2021

aspacca commented Aug 26, 2021 •

edited

Loading

jsoriano commented Aug 26, 2021

		// Old minors may not be available in artifacts-api, if it is older
		// than the others in the same major, return the version as is.

Test each integration with the oldest supported version #1541

Test each integration with the oldest supported version #1541

Conversation

jsoriano commented Aug 18, 2021 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Aug 18, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

elasticmachine commented Aug 23, 2021

elasticmachine commented Aug 23, 2021

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Aug 24, 2021

endorama Aug 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aspacca commented Aug 26, 2021 • edited Loading

jsoriano commented Aug 26, 2021

jsoriano commented Aug 18, 2021 •

edited

Loading

elasticmachine commented Aug 18, 2021 •

edited

Loading

endorama Aug 24, 2021 •

edited

Loading

aspacca commented Aug 26, 2021 •

edited

Loading