gae: Reimplement provider for Google App Engine #316

lorenzleutgeb · 2015-08-22T14:01:11Z

This now uses Google's Cloud SDK towards a resolution for #94

I see that tests are missing, but I'd rather like some opinion on the options and install routine beforehand.

And please bear with me, these are the first lines of Ruby in my life.

waprin · 2015-08-24T21:00:01Z

Thanks for doing this, I am no Ruby expert either but I'll try to test it soon.

Gemfile

@@ -54,7 +54,6 @@ group :gcs do
 end

 group :gae do
-  gem 'rubyzip'
 end


lorenzleutgeb · 2015-08-25T11:10:33Z

Justification for dropping SHA1 check:

The zipped file comes with a CRC sum

$ gzip -lv google-cloud-sdk.tar.gz
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla e0c0beec Aug 19 17:18             2204347            10680320  79.4% google-cloud-sdk.tar

and this sum is honored when decompressing with zlib

The zlib compression library provides in-memory compression and decompression functions, including integrity checks of the uncompressed data.

The file is downloaded via a trusted authenticated channel (HTTP over TLS).

Update: zlib is gone, but gzip also checks for integrity.

BanzaiMan · 2015-08-27T18:09:46Z

Could you test this with:

deploy:
  provider: gae
  edge:
    source: flowlo/dpl
    branch: gae
  ⋮
  on:
    all_branches: true # if necessary

lorenzleutgeb · 2015-08-27T18:49:28Z

Tested it, fails because it cannot install the Google Cloud SDK, right here.

@BanzaiMan Could you please point me to docs of context.shell? I want to get the exit code and standard streams to see what's happening, but do not know how to accomplish that.

lib/dpl/provider/gae.rb

          end
        end
+
+        # Bootstrap the Google Cloud SDK.
+        context.shell("CLOUDSDK_CORE_DISABLE_PROMPTS=1 #{Dir.home}/#{NAME}/bin/bootstrapping/install.py --usage-reporting=false --command-completion=false --path-update=false --additional-components=preview")


BanzaiMan · 2015-08-27T19:59:33Z

The real error is indicated here: https://travis-ci.org/coduno/api/jobs/77558698#L280

Both #check_auth and #push_app call GCLOUD by absolute path, but only the latter fails when /home/travis/google-cloud-sdk/bin/gcloud is invoked.

In between, #cleanup is called, and I suspect that somehow it is removing gcloud. If you add skip_cleanup: true under deploy, does the deployment work?

lib/dpl/provider/gae.rb

-                entry.extract entry.name
-              end
+        Gem::Package::TarReader.new(Zlib::GzipReader.open(open(BASE + NAME + EXT, "rb"))).each do |entry|
+          target = File.join(Dir.home, entry.full_name)


lorenzleutgeb · 2015-08-27T21:59:14Z

I added skip_cleanup but still have the same symptom.

BanzaiMan · 2015-08-27T22:01:14Z

@FlowLo So bizarre! If you want to inject various debug output (to determine when /home/travis/google-cloud-sdk/bin/gcloud disappears), it might be instructive. If you can't do that immediately, I'll take a stab at it some time later.

lorenzleutgeb · 2015-08-27T22:18:01Z

@BanzaiMan I just threw in some File.exists, see this build ... And well, TBH I can't really make sense of that. The call to .../bootstrapping/install.py does not complain, but still, gcloud is missing right after that.

I'm in UTC+02, will get some sleep now.

gae: Reimplement provider for Google App Engine

BanzaiMan · 2015-08-28T02:30:17Z

I think I tracked this down to an I/O race condition of sorts.

As currently implemented in the PR, the control is passed on to the next method before SDK bootstrapping is complete. This means that, by the time dpl gets to run gcloud, it is not yet extracted; hence ENOENT.

A clean way I've found is to use wget and tar, but with Open3 and wait for the files we want are written to disk.

Instead of Gem::Package::TarReader and Zlib::GzipReader, we use 'wget' and 'tar' to download & extract SDK archive inside Open3.popen3. Similarly, we use Open3.popen3 to ensure that SDK bootstrapping is complete before invoking `gcloud`. See #316 (comment) for complete details.

BanzaiMan · 2015-08-28T02:35:01Z

See https://github.com/travis-ci/dpl/tree/ha-test-pr-316.

Here is an example build: https://travis-ci.org/BanzaiMan/travis_production_test/builds/77610416. The deployment fails because the app is not set up for GAE, but you can see that gcloud is invoked (https://travis-ci.org/BanzaiMan/travis_production_test/builds/77610416#L193).

lorenzleutgeb · 2015-08-28T09:12:40Z

I opted for curl | gzip | tar over Gem::Package::TarReader and Zlib::GzipReader for stability. Looks like downloading/unzipping/unarchiving is solid now. Sorry for the confusion.

Provider working smoothly:

Managed VM (travis.yml)
Classic (travis.yml)

If you see stuff like

ERROR: (gcloud.preview.app.deploy) Not enough VMs ready (0/1 ready, 1 still deploying).
Deployed Version: ...
This can happen when your application does not start successfully.

it's the application's fault 😄

This now uses Google's Cloud SDK

Generate an SSH key so gcloud will not hang asking for a password upon key pair generation.

gae: Reimplement provider for Google App Engine

lorenzleutgeb force-pushed the gae branch 7 times, most recently from ec0b1c6 to 85da2cc Compare August 22, 2015 14:55

BanzaiMan reviewed Aug 24, 2015
View reviewed changes

Gemfile

@@ -54,7 +54,6 @@ group :gcs do

end

group :gae do

gem 'rubyzip'

end

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

BanzaiMan self-assigned this Aug 24, 2015

BanzaiMan added the provider: gae (Google App Engine) label Aug 24, 2015

lorenzleutgeb force-pushed the gae branch from 85da2cc to 0b999bc Compare August 25, 2015 12:34

BanzaiMan reviewed Aug 27, 2015
View reviewed changes

lorenzleutgeb reviewed Aug 27, 2015
View reviewed changes

lib/dpl/provider/gae.rb

entry.extract entry.name

end

Gem::Package::TarReader.new(Zlib::GzipReader.open(open(BASE + NAME + EXT, "rb"))).each do |entry|

target = File.join(Dir.home, entry.full_name)

This comment was marked as spam.

Sign in to view

BanzaiMan added a commit that referenced this pull request Aug 28, 2015

Merge pull request #316 from flowlo/gae

29bc797

gae: Reimplement provider for Google App Engine

lorenzleutgeb force-pushed the gae branch 5 times, most recently from 66ec8f3 to 57c588a Compare August 28, 2015 07:47

lorenzleutgeb force-pushed the gae branch 6 times, most recently from a1faf30 to cbd6198 Compare August 28, 2015 09:05

lorenzleutgeb force-pushed the gae branch 6 times, most recently from e248da0 to c778292 Compare August 28, 2015 09:36

gae: Reimplement provider for Google App Engine

f6d59f8

This now uses Google's Cloud SDK

lorenzleutgeb force-pushed the gae branch from c778292 to f6d59f8 Compare August 28, 2015 09:36

gae: Work around issue with gcloud

7f5509d

Generate an SSH key so gcloud will not hang asking for a password upon key pair generation.

BanzaiMan added a commit that referenced this pull request Sep 2, 2015

Merge pull request #316 from flowlo/gae

6e556a9

gae: Reimplement provider for Google App Engine

BanzaiMan merged commit 6e556a9 into travis-ci:master Sep 2, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gae: Reimplement provider for Google App Engine #316

gae: Reimplement provider for Google App Engine #316

lorenzleutgeb commented Aug 22, 2015

waprin commented Aug 24, 2015

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

lorenzleutgeb commented Aug 25, 2015

BanzaiMan commented Aug 27, 2015

lorenzleutgeb commented Aug 27, 2015

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

BanzaiMan commented Aug 27, 2015

This comment was marked as spam.

lorenzleutgeb commented Aug 27, 2015

BanzaiMan commented Aug 27, 2015

lorenzleutgeb commented Aug 27, 2015

BanzaiMan commented Aug 28, 2015

BanzaiMan commented Aug 28, 2015

lorenzleutgeb commented Aug 28, 2015

gae: Reimplement provider for Google App Engine #316

gae: Reimplement provider for Google App Engine #316

Conversation

lorenzleutgeb commented Aug 22, 2015

waprin commented Aug 24, 2015

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

lorenzleutgeb commented Aug 25, 2015

BanzaiMan commented Aug 27, 2015

lorenzleutgeb commented Aug 27, 2015

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

BanzaiMan commented Aug 27, 2015

This comment was marked as spam.

lorenzleutgeb commented Aug 27, 2015

BanzaiMan commented Aug 27, 2015

lorenzleutgeb commented Aug 27, 2015

BanzaiMan commented Aug 28, 2015

BanzaiMan commented Aug 28, 2015

lorenzleutgeb commented Aug 28, 2015