Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Ubuntu Xenial (16.04) on staging #3113

Merged
merged 12 commits into from Jul 3, 2019

Conversation

@rbreslow
Copy link
Member

rbreslow commented Jul 1, 2019

Overview

  • Configure Gunicorn to log errors to syslog
  • Configure rsyslog to forward Celery logs to syslog
  • Configure rsyslog to forward syslog to Papertrail
  • Remove references to monitoring service
  • Reconcile VPC and DataPlane stack drift
  • Reconcile /var/lib/jenkins/.aws/staging.yml

Resolves #3103

Testing Instructions

See that the CloudFormation stacks are in-sync:

image

Verify feature/jrb/xenial-staging is on staging:

maiden:~ rbreslow$ http --body https://staging.modelmywatershed.org/version.txt
origin/feature/jrb/xenial-staging 1.24.2-54-g4691c0a833f5e4ed07e080fff4f6ac675ac7e9d8

Select a HUC-8 unit and view its analysis:

image

See the Papertrail logs:

image

@rbreslow rbreslow self-assigned this Jul 1, 2019
deployment/cfn/application.py Show resolved Hide resolved
deployment/cfn/data_plane.py Show resolved Hide resolved
deployment/cfn/utils/cfn.py Show resolved Hide resolved
def get_recent_ami(aws_profile, ami_name, owner="self", executable_by="self"):
conn = boto.connect_ec2(profile_name=aws_profile)
filters = {'name': ami_name}
def get_recent_ami(

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 1, 2019

Author Member

Pulling in updates to this method from Cicero.

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 2, 2019

Contributor

The structure of this method signature is a bit hard to follow. I believe the Python convention would be to structure it like this:

def get_recent_ami(aws_profile, filters={}, owner="self", region="us-east-1",
                   executable_by="self"):

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

I'll go through and make corrections to things. I had been using https://github.com/python/black (without tuning any of its parameters), although I agree the method signature you posted is much more pretty.

deployment/cfn/vpc.py Show resolved Hide resolved
deployment/cfn/vpc.py Show resolved Hide resolved
deployment/packer/driver.py Show resolved Hide resolved
@rbreslow rbreslow requested a review from hectcastro Jul 1, 2019
@rbreslow rbreslow marked this pull request as ready for review Jul 1, 2019
rbreslow and others added 6 commits Jul 1, 2019
This was to make life easier for the ELK log aggregation stack.
This removes an unnecessary field from the Papertrail log entries.
Copy link
Contributor

hectcastro left a comment

Looks good on staging. I left a handful of comments and made some commits to fix things that I came across while testing.

  • The tile deliver stack drift still appears to exist (S3 bucket reference was lost)
  • RWD service data volume is mounted, but prior to data availability; I think we need to make sure the Docker service is restarted instead of initctl emit rwd-ready
deployment/cfn/data_plane.py Show resolved Hide resolved
def get_recent_ami(aws_profile, ami_name, owner="self", executable_by="self"):
conn = boto.connect_ec2(profile_name=aws_profile)
filters = {'name': ami_name}
def get_recent_ami(

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 2, 2019

Contributor

The structure of this method signature is a bit hard to follow. I believe the Python convention would be to structure it like this:

def get_recent_ami(aws_profile, filters={}, owner="self", region="us-east-1",
                   executable_by="self"):
@@ -1,4 +1,5 @@
EC2_INSTANCE_TYPES = [
't3.micro',

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 2, 2019

Contributor

Single quotes were replaced in a bunch of files above in favor of double quotes, but not here. I think they should all remain consistent as single quotes, or all be replaced as double quotes.

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

It's probably worth remaining consistent as single quotes. Sorry for letting this stuff slip through.

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 3, 2019

Contributor

I think it makes sense to keep t3.micro in the EC2_INSTANCE_TYPES list, but the bastion is currently still a t3.micro. We should convert it back to the T2 instance family because we have 12 RIs within the account (we're going to be over no matter what because the bastion used to be a larger instance type).

self.region = self.get_input('Region')
self.availability_zones = get_availability_zones(self.aws_profile)
self.availability_zones = get_availability_zones(self.aws_profile,
['us-east-1b', 'us-east-1d'])

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 2, 2019

Contributor

We may have to parameterize the AZs, because the AZs in production (us-east-1b, us-east-1a) are not the same as staging.

deployment/cfn/worker.py Outdated Show resolved Hide resolved
deployment/cfn/data_plane.py Show resolved Hide resolved
@@ -81,28 +81,6 @@
"Environment": "{{user `stack_type`}}"
},
"associate_public_ip_address": true
},

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 2, 2019

Contributor

I recall you mentioning that you wanted to convert the Packer builder to the T3 instance family.

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

Thanks for surfacing this.

def get_recent_ami(aws_profile, ami_name, owner="self", executable_by="self"):
conn = boto.connect_ec2(profile_name=aws_profile)
filters = {'name': ami_name}
def get_recent_ami(

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

I'll go through and make corrections to things. I had been using https://github.com/python/black (without tuning any of its parameters), although I agree the method signature you posted is much more pretty.

@@ -1,4 +1,5 @@
EC2_INSTANCE_TYPES = [
't3.micro',

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

It's probably worth remaining consistent as single quotes. Sorry for letting this stuff slip through.

@@ -81,28 +81,6 @@
"Environment": "{{user `stack_type`}}"
},
"associate_public_ip_address": true
},

This comment has been minimized.

Copy link
@rbreslow

rbreslow Jul 2, 2019

Author Member

Thanks for surfacing this.

@rbreslow

This comment has been minimized.

Copy link
Member Author

rbreslow commented Jul 2, 2019

The tile deliver stack drift still appears to exist (S3 bucket reference was lost)

I had ignored this since we didn't need to make any modifications to that stack and it seems like a potential time sink with not a lot to gain from.

I spent an hour or so this afternoon trying to reconcile the drift and haven't been successful. Since the S3 bucket was deleted and a new one was created out-of-band, the CloudFormation stack update will fail because it tries to create a bucket that already exists. Same with the DNS record. I believe the only option is to delete the resource so CloudFormation can recreate it, which is something we wouldn't want to do in production.

@rbreslow rbreslow requested a review from hectcastro Jul 2, 2019
Copy link
Contributor

hectcastro left a comment

👍

deployment/cfn/worker.py Outdated Show resolved Hide resolved
@@ -1,4 +1,5 @@
EC2_INSTANCE_TYPES = [
't3.micro',

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jul 3, 2019

Contributor

I think it makes sense to keep t3.micro in the EC2_INSTANCE_TYPES list, but the bastion is currently still a t3.micro. We should convert it back to the T2 instance family because we have 12 RIs within the account (we're going to be over no matter what because the bastion used to be a larger instance type).

@rbreslow rbreslow merged commit b7dc892 into develop Jul 3, 2019
@rbreslow rbreslow deleted the feature/jrb/xenial-staging branch Jul 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.