Fix code_deploy to properly register S3 rev #732

holyjak · 2018-01-08T19:45:32Z

Register a revision based on the latest (presumabely just uploaded)
file (when using S3) so that CodeDeploy will deploy the correct archive.
Otherwise it would deploy the latest registered revision, not the
latest uploaded archive.

As AWS the documentation
says:

If you've already called the push command to push an application
revision to Amazon S3, you don't need to register the revision.
However, if you upload a revision to Amazon S3 through other means and
want the revision to appear in the AWS CodeDeploy console or through
the AWS CLI, follow these steps to register the revision first.

Notice that we should also do this for GitHub:

If you've pushed an application revision to a GitHub repository and
want the revision to appear in the AWS CodeDeploy console or through
the AWS CLI, you must also follow these steps.

but I do not have the capacity to do that.

Tested both with versioned and non-versioned (no version, still has
etag) buckets.

BanzaiMan

Please fix the specs, and add build log URL(s) that shows deployment(s) using this code.

Thanks.

holyjak · 2018-01-10T17:48:50Z

How do I run the specs? I guess I need to start by bundle install but that fails due to ruby/json#229 when installing json-1.8.1. (Ubuntu 16.04, Ruby 2.3.3p222)

Build log: https://travis-ci.com/TeliaSoneraNorge/rapidshop-b2b/builds/63097181

BanzaiMan · 2018-01-10T18:49:27Z

You can see the failure here: https://travis-ci.org/travis-ci/dpl/jobs/326692313#L1686

BanzaiMan · 2018-01-10T18:52:34Z

Also, the build you indicated used the release 1.8.43, not your code. Could you review https://github.com/travis-ci/dpl/blob/master/TESTING.md and try again? Thanks.

holyjak · 2018-01-10T22:51:00Z

I will. Can you please tell me how to run specs locally? As described above, bundle indtall fails :-( ons. 10. jan. 2018 kl. 19.52 skrev Hiro Asari <notifications@github.com>:

…

Also, the build you indicated used the release 1.8.43, not your code. Could you review https://github.com/travis-ci/dpl/blob/master/TESTING.md and try again? Thanks. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#732 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmJPip4txYRt3wnDKXkXqYKwZ0R8Uy7ks5tJQb0gaJpZM4RW2_-> .

holyjak · 2018-01-11T10:27:00Z

Build log: https://travis-ci.com/TeliaSoneraNorge/rapidshop-b2b/builds/63176068

holyjak · 2018-01-11T13:33:17Z

Here is build log for the latest version: https://travis-ci.com/TeliaSoneraNorge/rapidshop-b2b/builds/63190665

Is there anything else I should fix?

PS: I would still appreciate instructions for how to get bundle and rspec to run locally. (As mentioned above, bundle install fails on json-1.8.1). Ideally a working Vagrantfile for VirtualBox where I could just run gem install bundler; bundle install :)

BanzaiMan

Over all it seems good, but I have some stylistic suggestions and one question about unclear piece of data.

lib/dpl/provider/code_deploy.rb

@@ -61,6 +76,16 @@ def github_revision
      end

      def push_app
+        rev = revision()
+        if rev[:s3_location]
+            revInfo = rev[:s3_location]


lib/dpl/provider/code_deploy.rb

+        end
+        code_deploy.register_application_revision({
+            revision:               rev,
+            application_name:       options[:application]      || option(:application_name),


lib/dpl/provider/code_deploy.rb

+          s3obj = s3api.get_object({
+              bucket: option(:bucket),
+              key: s3_key,
+              range: "bytes=0-1"


spec/provider/code_deploy_spec.rb

@@ -89,7 +89,9 @@
      s3_location: {
        bucket:      'bucket',
        bundle_type: 'tar',
-        key:         'key'
+        key:         'key',
+        version:     'ObjectVersionId',


holyjak · 2018-01-16T17:47:50Z

Thanks, will fix. I still wait for instructions how to run rspec locally (see bundle install issue mentioned above) :-) tir. 16. jan. 2018 kl. 15:45 skrev Hiro Asari <notifications@github.com>:

…

***@***.**** requested changes on this pull request. Over all it seems good, but I have some stylistic suggestions and one question about unclear piece of data. ------------------------------ In lib/dpl/provider/code_deploy.rb <#732 (comment)>: > @@ -61,6 +76,16 @@ def github_revision end def push_app + rev = revision() + if rev[:s3_location] + revInfo = rev[:s3_location] rev_info ------------------------------ In lib/dpl/provider/code_deploy.rb <#732 (comment)>: > @@ -61,6 +76,16 @@ def github_revision end def push_app + rev = revision() + if rev[:s3_location] + revInfo = rev[:s3_location] + log "Registering app revision with version=#{revInfo[:version]}, etag=#{revInfo[:e_tag]}" + end + code_deploy.register_application_revision({ + revision: rev, + application_name: options[:application] || option(:application_name), We can squash the spaces around ||. ------------------------------ In lib/dpl/provider/code_deploy.rb <#732 (comment)>: > @@ -39,13 +43,24 @@ def revision end end + def revision_version_info + s3obj = s3api.get_object({ + bucket: option(:bucket), + key: s3_key, + range: "bytes=0-1" Is there a reason for this value? Why is it hard coded? ------------------------------ In spec/provider/code_deploy_spec.rb <#732 (comment)>: > @@ -89,7 +89,9 @@ s3_location: { bucket: 'bucket', bundle_type: 'tar', - key: 'key' + key: 'key', + version: 'ObjectVersionId', Use object_version_id, and etag. The names are inconsequential here, so lower the cost of skimming the code by mimicking other constants in this bit. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#732 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmJPryvxUXbflQdDiXZc-SJD9lLq7LHks5tLLYYgaJpZM4RW2_-> .

Register a revision based on the latest (presumabely just uploaded) file when using S3 or the given commit when using GH so that CodeDeploy will deploy the correct archive. Otherwise it would deploy the latest _registered revision_, not the latest _uploaded archive_. As [AWS the documentation says](https://docs.aws.amazon.com/codedeploy/latest/userguide/application-revisions-register.html): > If you've already called the push command to push an application > revision to Amazon S3, you don't need to register the revision. > However, if you upload a revision to Amazon S3 through other means and > want the revision to appear in the AWS CodeDeploy console or through > the AWS CLI, follow these steps to register the revision first. > If you've pushed an application revision to a GitHub repository and > want the revision to appear in the AWS CodeDeploy console or through > the AWS CLI, you must also follow these steps. Tested both with versioned and non-versioned (no version, still has etag) buckets.

holyjak · 2018-01-17T15:17:52Z

Here is build log for the latest deployer update: https://travis-ci.com/TeliaSoneraNorge/rapidshop-b2b/builds/63617699

BanzaiMan · 2018-01-18T20:22:48Z

rake should run specs. However, I suggest using a Docker image or some other means of separation, as it may perform some system-level changes.

holyjak · 2018-01-19T06:40:34Z

That wasnt really helpful. As described above, I have tried to use a Vagrantbmachine but ‘bundle install’ failed due to issues with json-1.8.1. What Docker image do you use? How do tou mitigate this failure? tor. 18. jan. 2018 kl. 21:22 skrev Hiro Asari <notifications@github.com>:

…

rake should run specs. However, I suggest using a Docker image or some other means of separation, as it may perform some system-level changes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#732 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmJPtQPuACBaChVScxXrndsvEwte44Xks5tL6gZgaJpZM4RW2_-> .

holyjak · 2018-01-19T15:14:14Z

Can you guys rerun travis? The single version of ruby that failed as an empty log so it is hard to troubleshoot. Perhaps just a temporary hitch?

holyjak · 2018-01-22T19:22:26Z

@BanzaiMan I think I have addressed all you wanted? Can this be merged now? Thanks!

BanzaiMan · 2018-01-22T19:36:05Z

Thank you!

holyjak · 2018-01-22T20:31:51Z

Thank you for the help and patience! (Though I would still appreciate also advice on how to bundle install the damn thing ;-))

Any idea when will this be released and availablie at travis-ci? Thanks!

BanzaiMan · 2018-01-22T20:33:25Z

I'm thinking of (hopefully) the last 1.8.x release tomorrow.

holyjak force-pushed the master branch from cf2e263 to bc84279 Compare January 9, 2018 06:49

BanzaiMan suggested changes Jan 10, 2018

View reviewed changes

holyjak force-pushed the master branch 2 times, most recently from c24e64e to 3667bc9 Compare January 11, 2018 10:23

holyjak force-pushed the master branch 3 times, most recently from 9e406e3 to f6fd837 Compare January 11, 2018 12:14

BanzaiMan suggested changes Jan 16, 2018

View reviewed changes

holyjak force-pushed the master branch 2 times, most recently from 11cebf9 to 7eac080 Compare January 17, 2018 14:16

holyjak force-pushed the master branch from 7eac080 to 3da5680 Compare January 17, 2018 14:54

Merge branch 'master' into master

f383678

BanzaiMan merged commit 6c79e0c into travis-ci:master Jan 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix code_deploy to properly register S3 rev #732

Fix code_deploy to properly register S3 rev #732

holyjak commented Jan 8, 2018

BanzaiMan left a comment

holyjak commented Jan 10, 2018

BanzaiMan commented Jan 10, 2018

BanzaiMan commented Jan 10, 2018

holyjak commented Jan 10, 2018 via email

holyjak commented Jan 11, 2018

holyjak commented Jan 11, 2018

BanzaiMan left a comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

holyjak commented Jan 16, 2018 via email

holyjak commented Jan 17, 2018

BanzaiMan commented Jan 18, 2018

holyjak commented Jan 19, 2018 via email

holyjak commented Jan 19, 2018

holyjak commented Jan 22, 2018

BanzaiMan commented Jan 22, 2018

holyjak commented Jan 22, 2018

BanzaiMan commented Jan 22, 2018

Fix code_deploy to properly register S3 rev #732

Fix code_deploy to properly register S3 rev #732

Conversation

holyjak commented Jan 8, 2018

BanzaiMan left a comment

Choose a reason for hiding this comment

holyjak commented Jan 10, 2018

BanzaiMan commented Jan 10, 2018

BanzaiMan commented Jan 10, 2018

holyjak commented Jan 10, 2018 via email

holyjak commented Jan 11, 2018

holyjak commented Jan 11, 2018

BanzaiMan left a comment

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

holyjak commented Jan 16, 2018 via email

holyjak commented Jan 17, 2018

BanzaiMan commented Jan 18, 2018

holyjak commented Jan 19, 2018 via email

holyjak commented Jan 19, 2018

holyjak commented Jan 22, 2018

BanzaiMan commented Jan 22, 2018

holyjak commented Jan 22, 2018

BanzaiMan commented Jan 22, 2018