build: bake aws into the builder image

[petermattis]

Release note: None

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

This change is 

[petermattis]

Gah, apparently this installs a version of aws that is too old:

> /home/agent/work/.go/bin/roachprod create teamcity-968270-kv95-encrypt-false-nodes-1 -n 2 --clouds=aws --aws-machine-type-ssd=m5d.2xlarge
Creating cluster teamcity-968270-kv95-encrypt-false-nodes-1 with 2 nodes
2018/10/16 00:06:48 imported ${HOME}/.ssh/id_rsa.pub as teamcity-runner-w3i2u7tBwUAjM4_HewnqvX8mV44= in region us-east-2
2018/10/16 00:06:49 imported ${HOME}/.ssh/id_rsa.pub as teamcity-runner-w3i2u7tBwUAjM4_HewnqvX8mV44= in region us-west-2
2018/10/16 00:06:49 usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

Unknown options: --tag-specifications, ResourceType=instance,Tags=[{Key=Lifetime,Value=12h0m0s},{Key=Name,Value=teamcity-968270-kv95-encrypt-false-nodes-1-0001},{Key=Roachprod,Value=true},]

I saw exactly this same error locally and had to upgrade aws. My apt-get fu is weak. How do I track down a newer apt-installable version?

[petermattis]

Gah, apparently this installs a version of aws that is too old:

> /home/agent/work/.go/bin/roachprod create teamcity-968270-kv95-encrypt-false-nodes-1 -n 2 --clouds=aws --aws-machine-type-ssd=m5d.2xlarge
Creating cluster teamcity-968270-kv95-encrypt-false-nodes-1 with 2 nodes
2018/10/16 00:06:48 imported ${HOME}/.ssh/id_rsa.pub as teamcity-runner-w3i2u7tBwUAjM4_HewnqvX8mV44= in region us-east-2
2018/10/16 00:06:49 imported ${HOME}/.ssh/id_rsa.pub as teamcity-runner-w3i2u7tBwUAjM4_HewnqvX8mV44= in region us-west-2
2018/10/16 00:06:49 usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

Unknown options: --tag-specifications, ResourceType=instance,Tags=[{Key=Lifetime,Value=12h0m0s},{Key=Name,Value=teamcity-968270-kv95-encrypt-false-nodes-1-0001},{Key=Roachprod,Value=true},]

I saw exactly this same error locally and had to upgrade aws. My apt-get fu is weak. How do I track down a newer apt-installable version?

[petermattis]

Switched to explicitly download the awslci-bundle from amazon and installing it. Easier than figuring out if there is an up to date apt-get source or using pip.

[petermattis]

Switched to explicitly download the awslci-bundle from amazon and installing it. Easier than figuring out if there is an up to date apt-get source or using pip.

[benesch]

Switched to explicitly download the awslci-bundle from amazon and installing it. Easier than figuring out if there is an up to date apt-get source or using pip.

That's exactly what I was about to recommend. I'm not seeing a maintained PPA for awscli, and getting pip to work right in the builder would be a hassle, so I think the bundle is the way to go.

[benesch]

Switched to explicitly download the awslci-bundle from amazon and installing it. Easier than figuring out if there is an up to date apt-get source or using pip.

That's exactly what I was about to recommend. I'm not seeing a maintained PPA for awscli, and getting pip to work right in the builder would be a hassle, so I think the bundle is the way to go.

[petermattis]

Ugh, what is this failure about. https://teamcity.cockroachdb.com/viewLog.html?buildId=968385&tab=buildLog

[01:20:43][Step 2/2] Status: Downloaded newer image for cockroachdb/builder:20181015-210651
[01:20:50][Step 2/2] Traceback (most recent call last):
[01:20:50][Step 2/2]   File "/usr/bin/../lib/google-cloud-sdk/lib/gcloud.py", line 20, in <module>
[01:20:50][Step 2/2]     from __future__ import absolute_import
[01:20:50][Step 2/2] ImportError: No module named __future__

It happened twice, so it isn't a fluke.

[petermattis]

Ugh, what is this failure about. https://teamcity.cockroachdb.com/viewLog.html?buildId=968385&tab=buildLog

[01:20:43][Step 2/2] Status: Downloaded newer image for cockroachdb/builder:20181015-210651
[01:20:50][Step 2/2] Traceback (most recent call last):
[01:20:50][Step 2/2]   File "/usr/bin/../lib/google-cloud-sdk/lib/gcloud.py", line 20, in <module>
[01:20:50][Step 2/2]     from __future__ import absolute_import
[01:20:50][Step 2/2] ImportError: No module named __future__

It happened twice, so it isn't a fluke.

[benesch]

It happened twice, so it isn't a fluke.

I think ./awscli-bundle/install -i /usr/local/ aws has a stray space in it and it therefore might have wiped the contents of /usr/local in weird and arbitrary ways.

[benesch]

It happened twice, so it isn't a fluke.

I think ./awscli-bundle/install -i /usr/local/ aws has a stray space in it and it therefore might have wiped the contents of /usr/local in weird and arbitrary ways.

[petermattis]

Ah, good catch. I've fixed that. Let me see if this gets me farther.

[petermattis]

Ah, good catch. I've fixed that. Let me see if this gets me farther.

[petermattis]

Ok, making progress: https://teamcity.cockroachdb.com/viewLog.html?buildId=968422&buildTypeId=Cockroach_Nightlies_RoachtestNightlyAws&tab=buildLog

[petermattis]

Ok, making progress: https://teamcity.cockroachdb.com/viewLog.html?buildId=968422&buildTypeId=Cockroach_Nightlies_RoachtestNightlyAws&tab=buildLog

[petermattis]

And we're running a roachtest on AWS!

[petermattis]

And we're running a roachtest on AWS!

[benesch]

Doesn't look like you actually pushed your latest revision?

[petermattis]

Just adding an extra level of difficulty to the review. Pushed.

[petermattis]

Just adding an extra level of difficulty to the review. Pushed.

[benesch]

LGTM!

[petermattis]

Thanks, though see #31416. Some aws initialization or something is missing. I think this PR is fine, but I'll hold off on merging it until we get that issue figured out.

[petermattis]

Thanks, though see #31416. Some aws initialization or something is missing. I think this PR is fine, but I'll hold off on merging it until we get that issue figured out.

[petermattis]

bors r=benesch

[petermattis]

bors r=benesch

[craig]

Build succeeded

• GitHub CI (Cockroach)

[craig]

Build succeeded

• GitHub CI (Cockroach)