Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch snapshots automation #53706

Merged
merged 38 commits into from Jan 7, 2020

Conversation

@brianseeders
Copy link
Contributor

brianseeders commented Dec 20, 2019

Closes #52016

There's a couple of small TODOs still there, but this is finished up for the most part.

Need to make sure proper initial snapshots exist before merging this.

Jenkinsfiles:

  • trigger_build_es triggers ES build jobs for each tracked branch
  • build_es builds ES, and uploads the created artifacts to a GCS daily snapshot bucket along with a manifest file. Snapshots in this bucket are kept for 60 days. This job then triggers the verify job
  • verify_es runs most of the standard pipeline, but uses the ES snapshot created from the last job. If the tests succeed, the snapshot is "promoted" by copying its manifest file to the default location used by kbn-es for that ES version, and also copying all of the artifacts to the permanent snapshot bucket. If it fails, an e-mail will be sent out, and the last verified snapshot will continue to be used by kbn-es.

kbn-es:

  • release-manager snapshot functionality removed
  • by default, check the daily gcs snapshot bucket for the last verified snapshot manifest for the required version of ES
  • if the daily snapshot 404s because it expired, fall back to the permanent snapshot storage bucket (this gets us long-lived snapshot storage for old branches and automatic daily artifact cleanup without having to maintain a cleanup process)
  • once a manifest has been downloaded, find the artifact that matches the requested version/license/platform, and continue as normal
brianseeders added 25 commits Dec 13, 2019
…mation
…mation
wip
…mation
…mation
…shot handling
…mation
…ches yaml
Copy link
Member

tylersmalley left a comment

LGTM - just need to change the build failure email back before merging.

@mistic

This comment has been minimized.

Copy link
Member

mistic commented Jan 6, 2020

@elasticmachine merge upstream

elasticmachine and others added 5 commits Jan 6, 2020
…mation
…ana into es-snapshots-automation
…sts run
@brianseeders

This comment has been minimized.

Copy link
Contributor Author

brianseeders commented Jan 6, 2020

@elasticmachine merge upstream

@brianseeders

This comment has been minimized.

Copy link
Contributor Author

brianseeders commented Jan 6, 2020

@elasticmachine merge upstream

@kibanamachine

This comment has been minimized.

Copy link

kibanamachine commented Jan 6, 2020

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@brianseeders brianseeders merged commit 4b6c2f8 into elastic:master Jan 7, 2020
55 checks passed
55 checks passed
API integration tests node scripts/functional_tests --config test/api_integration/config.js --bail --debug
Details
Browser tests yarn run grunt test:browser-ci
Details
Build kbn_tp_sample_panel_action yarn build
Details
CLA All commits in pull request signed
Details
Check core API changes node scripts/check_core_api_changes
Details
Check file casing node scripts/check_file_casing --quiet
Details
Check licenses node scripts/check_licenses --dev
Details
Example functional tests node scripts/functional_tests --config test/examples/config.js --bail --debug
Details
Firefox smoke test node scripts/functional_tests --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --include-tag smoke --config test/functional/config.firefox.js
Details
Functional tests / Group 1 yarn run grunt run:functionalTests_ciGroup1
Details
Functional tests / Group 10 yarn run grunt run:functionalTests_ciGroup10
Details
Functional tests / Group 11 yarn run grunt run:functionalTests_ciGroup11
Details
Functional tests / Group 12 yarn run grunt run:functionalTests_ciGroup12
Details
Functional tests / Group 2 yarn run grunt run:functionalTests_ciGroup2
Details
Functional tests / Group 3 yarn run grunt run:functionalTests_ciGroup3
Details
Functional tests / Group 4 yarn run grunt run:functionalTests_ciGroup4
Details
Functional tests / Group 5 yarn run grunt run:functionalTests_ciGroup5
Details
Functional tests / Group 6 yarn run grunt run:functionalTests_ciGroup6
Details
Functional tests / Group 7 yarn run grunt run:functionalTests_ciGroup7
Details
Functional tests / Group 8 yarn run grunt run:functionalTests_ciGroup8
Details
Functional tests / Group 9 yarn run grunt run:functionalTests_ciGroup9
Details
Internationalization check node scripts/i18n_check --ignore-missing
Details
Interpreter functional tests node scripts/functional_tests --config test/interpreter_functional/config.ts --bail --debug --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64-2
Details
Jest integration tests yarn run grunt test:jest_integration
Details
Jest tests yarn run grunt test:jest
Details
Kibana accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/kibana/build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64 --config test/accessibility/config.ts
Details
Mocha tests node scripts/mocha
Details
Plugin functional tests node scripts/functional_tests --config test/plugin_functional/config.js --bail --debug
Details
Project tests yarn run grunt test:projects
Details
Type check node scripts/type_check
Details
TypeScript - all files belong to a TypeScript project node scripts/check_ts_projects
Details
Verify NOTICE.txt node scripts/notice --validate
Details
Verify dependency versions yarn run grunt verifyDependencyVersions
Details
X-Pack Chrome Functional tests / Group 1 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-2 --include-tag ciGroup1
Details
X-Pack Chrome Functional tests / Group 10 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-11 --include-tag ciGroup10
Details
X-Pack Chrome Functional tests / Group 2 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-3 --include-tag ciGroup2
Details
X-Pack Chrome Functional tests / Group 3 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-4 --include-tag ciGroup3
Details
X-Pack Chrome Functional tests / Group 4 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-5 --include-tag ciGroup4
Details
X-Pack Chrome Functional tests / Group 5 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-6 --include-tag ciGroup5
Details
X-Pack Chrome Functional tests / Group 6 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-7 --include-tag ciGroup6
Details
X-Pack Chrome Functional tests / Group 7 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-8 --include-tag ciGroup7
Details
X-Pack Chrome Functional tests / Group 8 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-9 --include-tag ciGroup8
Details
X-Pack Chrome Functional tests / Group 9 node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-10 --include-tag ciGroup9
Details
X-Pack Jest node scripts/jest --ci --verbose
Details
X-Pack Karma Tests yarn test:browser
Details
X-Pack SIEM cyclic dependency test node legacy/plugins/siem/scripts/check_circular_deps
Details
X-Pack accessibility tests node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana --config test/accessibility/config.ts
Details
X-Pack firefox smoke test node scripts/functional_tests --debug --bail --kibana-install-dir /dev/shm/workspace/install/kibana-1 --include-tag smoke --config test/functional/config.firefox.js
Details
elasticsearch-ci/docs Build finished.
Details
eslint node scripts/eslint --no-cache
Details
kibana-ci Build finished.
Details
prbot:outdated
prbot:release note labels
prbot:release version labels
sasslint node scripts/sasslint
Details
Kibana CI automation moved this from In progress to Done Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
brianseeders added a commit to brianseeders/kibana that referenced this pull request Jan 7, 2020
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 7, 2020
…le-saved-objects

* 'master' of github.com:elastic/kibana: (86 commits)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  [SIEM] Enable eslint react/no-children-prop (elastic#53985)
  [DOCS][Reporting] Adds info about using a custom reporting index (elastic#54052)
  Changing background color to align with EUI color (elastic#54060)
  Fix linting issues (elastic#54068)
  NP Migration: Move doc views registry and existing doc views into discover plugin (elastic#53465)
  [ML] DF Analytics job creation: Add 'excludes' input field to form (elastic#53856)
  EMT-issue-65: add endpoint list api (elastic#53861)
  [SIEM] Fix doubled drag handles in Timeline (elastic#52679)
  Functional tests: refactor visualize_page (elastic#53845)
  [Dashboard] Redesign empty screen in readonly mode (elastic#54073)
  [ML] Support search for partitions on Single Metric Viewer (elastic#53879)
  update apm index pattern (elastic#54095)
  Add data ui for envoyproxy Metricbeat Module (elastic#53476)
  [ML] persist the brush when expanded to full width (elastic#54020)
  Skip failing test (elastic#54100)
  [APM] Show errors on the timeline instead of under the transaction (elastic#53756)
  ...

# Conflicts:
#	src/legacy/core_plugins/console/public/np_ready/application/contexts/services_context.tsx
#	src/legacy/core_plugins/console/public/np_ready/application/index.tsx
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 7, 2020
…/kibana into feature/console-saved-objects

* 'feature/console-saved-objects' of github.com:jloleysens/kibana: (86 commits)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  [SIEM] Enable eslint react/no-children-prop (elastic#53985)
  [DOCS][Reporting] Adds info about using a custom reporting index (elastic#54052)
  Changing background color to align with EUI color (elastic#54060)
  Fix linting issues (elastic#54068)
  NP Migration: Move doc views registry and existing doc views into discover plugin (elastic#53465)
  [ML] DF Analytics job creation: Add 'excludes' input field to form (elastic#53856)
  EMT-issue-65: add endpoint list api (elastic#53861)
  [SIEM] Fix doubled drag handles in Timeline (elastic#52679)
  Functional tests: refactor visualize_page (elastic#53845)
  [Dashboard] Redesign empty screen in readonly mode (elastic#54073)
  [ML] Support search for partitions on Single Metric Viewer (elastic#53879)
  update apm index pattern (elastic#54095)
  Add data ui for envoyproxy Metricbeat Module (elastic#53476)
  [ML] persist the brush when expanded to full width (elastic#54020)
  Skip failing test (elastic#54100)
  [APM] Show errors on the timeline instead of under the transaction (elastic#53756)
  ...
brianseeders added a commit that referenced this pull request Jan 7, 2020
brianseeders added a commit that referenced this pull request Jan 7, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 8, 2020
* master: (55 commits)
  [ui/public/utils] Copy rarely used items to where they are consumed (elastic#53819)
  set AppArch team as an owner of the search endpoints (elastic#54131)
  Don't expose Elasticsearch client as Observable (elastic#53824)
  [SIEM] Cleanup unnecessary use of enzyme-to-json (elastic#53980)
  fix ui exports doc (elastic#54138)
  change markdown element title (elastic#54194)
  [Logs UI] Refactor log position to hooks (elastic#53540)
  [SIEM] Implement NP Plugin Setup (elastic#54030)
  [DOCS] Updates ML links (elastic#53613)
  sort renovate packages in config
  Spaces - fix flakey api tests (elastic#54154)
  Remove dependency that was causing effect to re-execute infinitely. (elastic#54160)
  [dev/run] expose unexpected flags as more than just names (elastic#54080)
  [DOCS] Moves index pattern doc to Discover (elastic#53347)
  [SIEM] Cleanup React imports (elastic#53981)
  Update eslint related packages (elastic#54107)
  [Uptime] Added date range filter into expanded list query (elastic#52609)
  [SIEM] Add react/display-name eslint rule (elastic#53107)
  [SIEM] Enable eslint prefer-template rule (elastic#53983)
  Elasticsearch snapshots automation (elastic#53706)
  ...
brianseeders added a commit that referenced this pull request Jan 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.