[GOAL2-558] Update S3 Usage #124

Karmastic · 2019-07-08T04:23:02Z

Builds on Eric's fork to address rough edges after testing all of the scenarios.
Tested everything I could. Algonet not tested but I'm working on a PR that updates it to set appropriate environment variables for nodecfg and log uploads (there's a new algonet IAM user).

There will be some manual work required to transition to the new buckets (need releases to exist in old and new buckets for the first new release, and need to update the bootstrappers). It should be transparent to most people.

Note that goal logging send is now secured -- aws doesn't support anonymous multi-part uploads so we'll have to distribute upload creds as-needed. This should almost eliminate our risk of S3 / billing abuse, with a slight inconvenience introduced when we need logs uploaded.

synch with master

variables S3_UPLOAD_BUCKET and S3_REGION.

create_and_deploy_recipe.sh script.

…sting file in util/s3.

Use consistent naming for bucket name constants.

Replaced remaining references to S3_UPLOAD_ID and S3_UPLOAD_SECRET with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Removed temporary recipe geo-test-small from testdata.

a flag or through S3_UPLOAD_BUCKET environment variable.

Removed obsolete fileIterator. Fixed scripts and rationalized usage of S3_UPLOAD_BUCKET Fixed anonymous credentials Logging now requires credentials - S3 doesn't allow anonymous MultiPartUploads

Standardized use of S3_RELEASE_BUCKET for builds; S3_UPLOAD_BUCKET only used for uploading logs.

Karmastic · 2019-07-08T05:09:41Z

See Confluence for details / usage scenarios: https://algorand.atlassian.net/wiki/spaces/DEVOPS/pages/420118549/S3+Buckets+and+Policies

egieseke

These changes look ok to me. I will retest with Algonet to make sure that works.

egieseke

I successfully built and deployed using Algonet.

winder

Looks great! Mostly nitpicks and minor suggestions.

winder · 2019-07-10T18:00:21Z

.travis.yml

  - name: release
-    if: branch =~ /^rel\/nightly/ AND type != pull_request
+    if: (branch =~ /^rel\/nightly/ OR branch =~ /^rel\/dev/) AND type != pull_request


I left this comment on https://github.com/algorand/go-algorand-internal/pull/3

To support more release branches than just rel/nightly, we should specifically exclude rel/stable rather than only include nightly:

This seems to work: branch != rel/stable AND branch =~ /^rel//

$ travis-conditions eval "branch != rel/stable AND branch =~ /^rel\//" --data '{"branch": "rel/stable"}' false $ travis-conditions eval "branch != rel/stable AND branch =~ /^rel\//" --data '{"branch": "rel/nightly"}' true $ travis-conditions eval "branch != rel/stable AND branch =~ /^rel\//" --data '{"branch": "master"}' false

Additionally, we could use the travis deployments for this rather than tasks. I'm using the github releases provider for SHD but we could use the script or s3 provider as well:
https://docs.travis-ci.com/user/deployment/script/
https://docs.travis-ci.com/user/deployment/s3/

winder · 2019-07-10T18:16:24Z

scripts/create_and_deploy_private_version.sh

+if [[ ! -z "${S3_RELEASE_BUCKET}" && -z "${BUCKET}" ]]; then
+    echo "Ignoring S3_RELEASE_BUCKET setting - defaulting to algorand-internal.  Use -b to override."
+fi
+S3_RELEASE_BUCKET="${BUCKET:-algorand-internal}"


nit: Could you set the algorand-internal default at the top of the file when the rest are initialized to empty? I think it will let you simplify the code a bit:

if [[ ! -z "${S3_RELEASE_BUCKET}" ]]; then echo "Ignoring S3_RELEASE_BUCKET setting - defaulting to algorand-internal. Use -b to override." fi # If we really need this it would always be. export S3_RELEASE_BUCKET="${BUCKET}"

Then just use -b "${BUCKET}" at the end.

Initializing prevents the ability to report this warning.

Given the users of this script I don't think it's a major point, but with the current logic S3_RELEASE_BUCKET could be silently overwritten with the default value.

winder · 2019-07-10T18:16:44Z

scripts/create_and_deploy_recipe.sh

+if [[ ! -z "${S3_RELEASE_BUCKET}" && -z "${BUCKET}" ]]; then
+    echo "Ignoring S3_RELEASE_BUCKET setting - defaulting to algorand-internal.  Use -b to override."
+fi
+S3_RELEASE_BUCKET="${BUCKET:-algorand-internal}"


Same as above

winder · 2019-07-10T18:17:19Z

scripts/deploy_private_version.sh

+if [[ ! -z "${S3_RELEASE_BUCKET}" && -z "${BUCKET}" ]]; then
+    echo "Ignoring S3_RELEASE_BUCKET setting - defaulting to algorand-internal.  Use -b to override."
+fi
+S3_RELEASE_BUCKET="${BUCKET:-algorand-internal}"


Same as above

winder · 2019-07-10T18:17:48Z

scripts/travis/release_packages.sh

@@ -13,10 +13,12 @@ set -e
 CHANNEL=""
 FULLVERSION=""
 S3CMD="s3cmd"
+BUILD_BUCKET=${S3_RELEASE_BUCKET}
+RELEASE_BUCKET="algorand-releases"


This is the pattern I like :)

winder · 2019-07-10T18:24:59Z

scripts/travis/release_packages.sh

@@ -50,9 +52,10 @@ function promote_stable() {
    init_s3cmd

    # Copy the _CHANNEL_ pending 'node' files to _CHANNEL-canary_


Update comment

What comment update is needed?

It's also copying the files to a new bucket

Remove pending_ from 'node' files, rename to CHANNEL-canary, and copy to the RELEASE_BUCKET

winder · 2019-07-10T18:25:58Z

scripts/upload_config.sh

@@ -31,5 +31,5 @@ TARFILE=${TEMPDIR}/config_${CHANNEL}_${FULLVERSION}.tar.gz
 cd $1
 tar -zcf ${TARFILE} * >/dev/null 2>&1

-${GOPATH}/bin/updater send -s ${TEMPDIR} -c ${CHANNEL}
+${GOPATH}/bin/updater send -s ${TEMPDIR} -c ${CHANNEL} -b "${S3_RELEASE_BUCKET}"


Has this environment variable been validated by another script which calls this one?

winder · 2019-07-10T18:27:06Z

util/s3/fileIterator.go

 	next      struct {
 		path string
 		f    *os.File
 	}
 	err error
 }

-func makeFileIterator(files []string, bucket string) s3manager.BatchUploadIterator {
+func makeFileIterator(files []string, bucket string, subFolder string) s3manager.BatchUploadIterator {


The other function named this targetFolder, might as well keep them consistent.

winder · 2019-07-10T18:29:44Z

util/s3/s3Helper.go

+
+func checkCredentialsRequired(action string, bucketName string) (required bool) {
+	required = true
+    if action == downloadAction && bucketName == s3DefaultReleaseBucket {


spaces -> tab

winder · 2019-07-10T18:47:01Z

util/s3/s3Helper_test.go

+		})
+	}
+}
+


Is it possible to check the session credentials in gotHelper.session? Would be nice to check directly that the anonymous credentials work. I guess it's implicitly tested in test5/test9 - maybe give those tests more specific names?

We no longer prefix with "pending_" and instead move from the build bucket to the release bucket.

winder

LGTM

Add the opcode plusw.

egieseke and others added 19 commits June 24, 2019 16:11

Merge pull request #1 from algorand/master

c055ddb

synch with master

Update to support use of Algorand's AWS S3 Service.

296b126

Removed updatekey.json file and references to it.

91b1d96

Support overriding default upload bucket and region with environment

1653f20

variables S3_UPLOAD_BUCKET and S3_REGION.

Added require for S3_UPLOAD_BUCKET environment variable in

2f35439

create_and_deploy_recipe.sh script.

Removed redundant s3Helper.go from logging, and consolidated with exi…

b83079e

…sting file in util/s3.

Fixed lint issue in s3Helper.go

f49fbbb

Check for empty bucket name.

9a7ee2f

Use consistent naming for bucket name constants.

Removed redundant updater/s3Helper.go.

ab981a8

Replaced remaining references to S3_UPLOAD_ID and S3_UPLOAD_SECRET with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Removed temporary recipe geo-test-small from testdata.

Simplified S3Helper upload and download methods.

3eddc28

Updated logic for checking for bucket specification which can be through

b966e3a

a flag or through S3_UPLOAD_BUCKET environment variable.

Added -b bucket flags to nodecfg apply and get commands.

9af6dbf

Added unit test for s3Helper.go

85d95ae

Updated to handle public/private scenarios

97e522d

Removed obsolete fileIterator. Fixed scripts and rationalized usage of S3_UPLOAD_BUCKET Fixed anonymous credentials Logging now requires credentials - S3 doesn't allow anonymous MultiPartUploads

More fixes with more testing

186e0b6

Standardized use of S3_RELEASE_BUCKET for builds; S3_UPLOAD_BUCKET only used for uploading logs.

Fix build, fix release_packages

f6feb9f

Merge remote-tracking branch 'origin/master' into rel/test

022a840

Fix usage of RELEASE_BUCKET

21c87d4

Fixed prompt_nightly

d9a972d

Karmastic requested review from winder, egieseke and tsachiherman July 8, 2019 04:23

Karmastic mentioned this pull request Jul 8, 2019

[GOAL2-558] main net s3 bucket strategy #98

Closed

Karmastic added 2 commits July 8, 2019 01:29

Fix s3 uploads test

5cc77eb

Stop uploading files from gen/

661b194

egieseke reviewed Jul 8, 2019

View reviewed changes

David Shoots and others added 3 commits July 9, 2019 09:53

Fixed script syntax error

911bf0e

Fixed to support overriding bucket

25bf391

Fixed algonet/nodecfg support for new bucket strategy

c6eefb7

Merge branch 'master' into david/newS3Buckets

05e378f

egieseke previously approved these changes Jul 10, 2019

View reviewed changes

winder reviewed Jul 10, 2019

View reviewed changes

David Shoots added 2 commits July 25, 2019 16:30

Merge branch 'master' into david/newS3Buckets

7a56f1e

addressed PR feedback

efc9d70

Karmastic dismissed egieseke’s stale review via efc9d70 July 25, 2019 20:54

Reviewed release naming and updated

119f174

We no longer prefix with "pending_" and instead move from the build bucket to the release bucket.

winder previously approved these changes Jul 29, 2019

View reviewed changes

Karmastic added 2 commits July 29, 2019 17:28

Merge remote-tracking branch 'origin/master' into david/newS3Buckets

6ee3863

Fix syntax error

c21f44d

Karmastic dismissed winder’s stale review via c21f44d July 29, 2019 21:34

winder approved these changes Jul 30, 2019

View reviewed changes

Karmastic merged commit 7588e41 into master Jul 30, 2019

Karmastic deleted the david/newS3Buckets branch July 30, 2019 13:55

algorandskiy pushed a commit to algorandskiy/go-algorand that referenced this pull request Jun 15, 2020

Merge pull request algorand#124 from derbear/plusw

ebab9fe

Add the opcode plusw.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GOAL2-558] Update S3 Usage #124

[GOAL2-558] Update S3 Usage #124

Karmastic commented Jul 8, 2019

Karmastic commented Jul 8, 2019

egieseke left a comment

egieseke left a comment

winder left a comment

winder Jul 10, 2019

winder Jul 10, 2019

winder Jul 10, 2019

Karmastic Jul 25, 2019

winder Jul 26, 2019

winder Jul 10, 2019

winder Jul 10, 2019

winder Jul 10, 2019

winder Jul 10, 2019

Karmastic Jul 25, 2019

winder Jul 26, 2019

winder Jul 10, 2019

winder Jul 10, 2019

winder Jul 10, 2019

winder Jul 10, 2019

winder left a comment

		@@ -50,9 +52,10 @@ function promote_stable() {
		init_s3cmd

		# Copy the _CHANNEL_ pending 'node' files to _CHANNEL-canary_

[GOAL2-558] Update S3 Usage #124

[GOAL2-558] Update S3 Usage #124

Conversation

Karmastic commented Jul 8, 2019

Karmastic commented Jul 8, 2019

egieseke left a comment

Choose a reason for hiding this comment

egieseke left a comment

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment