Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure install scripts only install if needed #20349

Merged
merged 2 commits into from
Aug 4, 2020
Merged

Ensure install scripts only install if needed #20349

merged 2 commits into from
Aug 4, 2020

Conversation

cachedout
Copy link
Contributor

What does this PR do?

There are a series of scripts which are used by the CI.

In some cases, the workers already have the requisite software and so we don't need to bother downloading it. In other cases, we need to detect that we don't have an appropriate version installed on an image, so we can open an issue to get that done. This introduces an easy way to do by checking logs for the string UNMET DEP.

Why is it important?

Reduce pre-provsioning errors in testing.

@cachedout cachedout requested review from andrewkroh and a team July 30, 2020 13:20
@cachedout cachedout self-assigned this Jul 30, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 30, 2020
v1v
v1v approved these changes Jul 30, 2020
View reviewed changes
.ci/scripts/install-docker-compose.sh Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 30, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20349 updated]

  • Start Time: 2020-08-04T09:49:55.826+0000

  • Duration: 72 min 35 sec

Test stats 🧪

Test Results
Failed 0
Passed 17828
Skipped 1920
Total 19748

andrewkroh
andrewkroh approved these changes Aug 3, 2020
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Do we have Go preinstalled in any of the images? I thought GVM was installed, but not Go since we change Go versions frequently.

@cachedout
Copy link
Contributor Author

Do we have Go preinstalled in any of the images? I though GVM was installed, but not Go since we change Go versions frequently.

I'd like to see if we could build out an Ansible role that could at least install it and then we can start a discussion about whether or not it's worth the overhead to keep that pre-installed version of Go synced with the supported version(s) in Beats. I'm still a bit on the fence about it.

@andrewkroh
Copy link
Member

+1 I think it would be good to have Go installed. As part of Ansible it could run gvm $version to install Go into $HOME so that it's already downloaded the next time that gvm $version is used.

To determine what $versions we need, probably bootstrapping with the versions from 7.x and master would be a good start. The versions can be read from .go-version file in each branch we care about (e.g. https://raw.githubusercontent.com/elastic/beats/master/.go-version).

@cachedout cachedout merged commit 3a9ce71 into elastic:master Aug 4, 2020
v1v added a commit to v1v/beats that referenced this pull request Aug 6, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
6fa6d3a 
…ne-2.0

* upstream/master:
  [docs] Promote ingest management to beta (elastic#20295)
  Upgrade elasticsearch client library used in tests (elastic#20405)
  Disable logging when pulling on python integration tests (elastic#20397)
  Remove pillow from testing requirements.txt (elastic#20407)
  [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440)
  [Ingest Manager] Send datastreams fields (elastic#20402)
  Add event.ingested to all Filebeat modules (elastic#20386)
  [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426)
  Improve cgroup_regex docs with examples (elastic#20425)
  Makes `metrics` config option required in app_insights (elastic#20406)
  Ensure install scripts only install if needed (elastic#20349)
  Update container name for the azure filesets (elastic#19899)
  Group same timestamp metrics values in app_insights metricset (elastic#20403)
  add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767)
  Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547)
  [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396)
  Update Suricata dashboards (elastic#20394)
  [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359)
  Prepare home directories for docker images in a different stage (elastic#20356)
v1v added a commit to v1v/beats that referenced this pull request Aug 6, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into fix/ci-tools-inst…
6b99d98 
…allation

* upstream/master: (23 commits)
  [docs] Promote ingest management to beta (elastic#20295)
  Upgrade elasticsearch client library used in tests (elastic#20405)
  Disable logging when pulling on python integration tests (elastic#20397)
  Remove pillow from testing requirements.txt (elastic#20407)
  [Filebeat][ATP Module]Setting user agent field required by the API (elastic#20440)
  [Ingest Manager] Send datastreams fields (elastic#20402)
  Add event.ingested to all Filebeat modules (elastic#20386)
  [Elastic Agent] Fix agent control socket path to always be less than 107 characters (elastic#20426)
  Improve cgroup_regex docs with examples (elastic#20425)
  Makes `metrics` config option required in app_insights (elastic#20406)
  Ensure install scripts only install if needed (elastic#20349)
  Update container name for the azure filesets (elastic#19899)
  Group same timestamp metrics values in app_insights metricset (elastic#20403)
  add_process_metadata processor adds container id even if process metadata not accessible (elastic#19767)
  Support "cluster" scope in Metricbeat elasticsearch module (elastic#18547)
  [Filebeat][SophosXG Module] Renaming module and fileset (elastic#20396)
  Update Suricata dashboards (elastic#20394)
  [Elastic Agent] Improve version, restart, enroll CLI commands (elastic#20359)
  Prepare home directories for docker images in a different stage (elastic#20356)
  New multiline mode in Filebeat: while_pattern (elastic#19662)
  ...
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@cachedout @melchiormoulin
Ensure install scripts only install if needed (elastic#20349)
bbe4635 
* Ensure install scripts only install once

* Add unmet dep str
v1v pushed a commit to v1v/beats that referenced this pull request Oct 16, 2020
@cachedout @v1v
Ensure install scripts only install if needed (elastic#20349)
10667d3 
* Ensure install scripts only install once

* Add unmet dep str
@v1v v1v mentioned this pull request Oct 16, 2020
Merged
v1v added a commit that referenced this pull request Oct 30, 2020
@v1v
[CI] 2.0 pipeline for 6.8 (#21859)
62f1115 
* cherry-pick aad07ad
* Add stages to Jenkins pipeline (#14484)
* ci: avoid to modify go.mod file (#16842)
* Ensure install scripts only install if needed (#20349)
* ci: fix warnings with wildcards and archive system-tests (#18695)
* ci: run test on Windows (#15570)
* [CI] fail if not possible to install python3 (#19164)
* [CI] lint stage doesn't produce test reports (#21888)
* [CI] Add stage name in the step (#21887)
* [CI] kind setup fails sometimes (#21857)
* [CI] refactor runbld post build action (#21256)
* [CI] Archive build reasons (#21347)
* [CI] Fix runbld when workspace does not exist (#21350)
* [CI] Send slack message with build status (#21428)
* [CI] Setup git config globally (#21562)
* [CI] fix 'no matches found within 10000' (#21466)
* [CI] Change notification channel (#21559)
* [CI] Support skip-ci label (#21377)
* [BUILD][CI] fetch dependencies with retry (#21614)
* Disable x-pack/metricbeat in linux since build is broken #21962
* Disable tests for windows in auditbeat. See #21958
* Disable integration testing for libbeat. See #21959
* Disable integration testing for metricbeat. See #21961
* Disable windows build for auditbeat. See #21958
* Disable build for libbeat. See #21959
* Disable build for metricbeat. See #21961
* Disable windows build for metricbeat. See #22257
* Disable build for x-pack/metricbeat. See #21962
* Disable windows build for x-pack/winlogbeat. See #22263
* Disable windows build for x-pack/filebeat. See #21963
* integration-tests-environment is not available for winlogbeat. see #22265
* integration-tests-environment is not available for packetbeat. see #22266
* Build not available for x-pack/metricbeats in 6.8. See #22269
* Cherry-pick 96c60d0
This was referenced Feb 24, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@v1v v1v v1v approved these changes

@andrewkroh andrewkroh andrewkroh approved these changes

Assignees

@cachedout cachedout

Labels
automation ci Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@cachedout @elasticmachine @andrewkroh @v1v