Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Change sql-release-workflow to use new staging bucket for artifacts #776

Merged
merged 32 commits into from
Feb 1, 2021

Conversation

camerski
Copy link
Contributor

@camerski camerski commented Oct 12, 2020

Description of changes:
The infrastructure team is separating the production and staging locations into different AWS accounts. Plugins need to modify their workflows to publish to the new locations.

This PR changes the sql-release-workflow to add a build number and write the zip, deb, and rpm plugin artifacts to staging.artifacts.opendistroforelasticsearch.amazon.com/snapshots/elasticsearch-plugins/sql/. The write to S3 currently fails because the secrets have not been updated, however a dummy run can be seen here: https://github.com/opendistro-for-elasticsearch/sql/actions/runs/296411387

This is the first change to the SQL plugin workflows. If this change is approved, I will update all the remaining workflows in a second PR. Secrets will be updated at the same time the PRs are merged.

See #777 for the second change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

penghuo and others added 22 commits June 17, 2020 09:17
* Bug fix, support long type for aggregation

* change to datetime to JDBC format
* prepare odfe 1.9

* Fix all ES 7.8 compile and build errors

* Revert changes as Lombok is working now

* Update CustomExternalTestCluster.java

* Fix license headers check

* Use splitFieldsByMetadata to separate fields when calling SearchHit constructor

* More fixes for ODFE 1.9

* Remove todo statement

* Add ODFE 1.9.0 release notes
* Revert "Rename release notes to use 4 digit versions (#547)"

This reverts commit 33c6d3e.

* Revert "Opendistro Release 1.9.0 (#532)"

This reverts commit 254f2e0.

* Revert "Bug fix, support long type for aggregation (#522)"

This reverts commit fb2ed91.
Merge develop to master
* merge all sql repos

* fix test and build workflows

* fix workbench and odbc path

* fix workbench and odbc path

* restructure workbench dir and fix workflows

* fix workbench workflow

* fix workbench workflow

* fix workbench workflow

* fix workbench workflow

* fix workbench workflow

* revert workbench directory structure

* fix workbench workflow

* fix workbench workflow

* fix workbench workflow

* fix workbench workflow

* update workbench workflow for release

* Delete .github/ in sql-workbench directory

* Add cypress to sql-workbench

* Sync latest ODBC commits

* Sync latest workbench commits (will add cypress in separate PR)

* Add ignored ODBC libs
* add date and time support

* update doc

* update doc
Merge develop to master for ODFE 1.9.0.1 release
Merge fixes for github release actions from develop to master
Fix odbc win32 release workflow for master
- add null check to avoid crashing if details not initialized
Merge develop branch into master to cut odfe1.10 release
Merge develop branch into master for od1.10 release
Merge workflow fix to master for od1.10 release
Merge develop to master for ODFE 1.10.1.0 release
Merge develop to master for ODFE 1.10.1
Copy link
Contributor

@abbashus abbashus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@abbashus
Copy link
Contributor

@camerski I would suggest creating a PR with a fork repo workflow in future instead of directly creating a branch on OpenDistro SQL repo.

@camerski
Copy link
Contributor Author

@camerski I would suggest creating a PR with a fork repo workflow in future instead of directly creating a branch on OpenDistro SQL repo.

Sure, no problem.

Copy link
Contributor

@penghuo penghuo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!
Could you help us understand how the GITHUB_RUN_NUMBER help the release workflow?

@camerski
Copy link
Contributor Author

Looks good to me!
Could you help us understand how the GITHUB_RUN_NUMBER help the release workflow?

The motivation for this change is that with the current workflows it is possible for an artifact to be overwritten during or after an ODFE release. That makes it impossible to guarantee that the ODFE release actually contains the artifacts that were tested, because they could be overwritten between the testing and final build phases. Even after a release, the artifacts can be overwritten which means we cannot reliably reproduce an ODFE build.

This change guarantees that each build is unique, making it much easier to provide reproducible ODFE builds and guarantee that we're releasing the right artifacts.

Long term, we plan to decouple ODFE distributions from plugin releases and treat it more like a Linux distribution: An ODFE release will just cherry-pick whatever public version of your plugin you tell us to use. This will allow you to define your own release process and not be dependent on the infrastructure team.

@codecov
Copy link

codecov bot commented Dec 21, 2020

Codecov Report

Merging #776 (74589a7) into develop (1dbf244) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             develop     #776   +/-   ##
==========================================
  Coverage      99.87%   99.87%           
  Complexity      2416     2416           
==========================================
  Files            234      234           
  Lines           5547     5547           
  Branches         359      359           
==========================================
  Hits            5540     5540           
  Misses             5        5           
  Partials           2        2           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dbf244...74589a7. Read the comment docs.

@gaiksaya gaiksaya marked this pull request as draft January 12, 2021 19:47
@peterzhuamazon peterzhuamazon changed the base branch from master to develop February 1, 2021 22:17
@peterzhuamazon peterzhuamazon marked this pull request as ready for review February 1, 2021 22:19
@peterzhuamazon peterzhuamazon merged commit bc40306 into develop Feb 1, 2021
@peterzhuamazon peterzhuamazon deleted the camerski branch February 1, 2021 22:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants