Many Microservices, Handle It

Adoption coverage

The AWS layer now supports adoption of endpoints (API Gateway), nosqldbs (DynamoDB), search_domains (ElasticSearch), notifiers (SNS).

mu-adopt now supports a --pattern flag which can be used to filter against a resource's resolved name, discarding any that do not match.

$ mu-adopt -t endpoints nosqldbs search_domains notifiers jobs cdns buckets functions
<...>
Sep 02 01:34:59 - mu-adopt - Writing to muservices.yaml
Sep 02 01:35:01 - mu-adopt - 7 notifiers
Sep 02 01:35:01 - mu-adopt - 57 functions
Sep 02 01:35:01 - mu-adopt - 4 endpoints
Sep 02 01:35:01 - mu-adopt - 2 search_domains
Sep 02 01:35:01 - mu-adopt - 5 nosqldbs
Sep 02 01:35:01 - mu-adopt - 9 jobs
Sep 02 01:35:01 - mu-adopt - 6 cdns

$ mu-adopt -t endpoints nosqldbs search_domains notifiers jobs cdns buckets functions --pattern 'myapp'
<...>
Sep 02 01:37:42 - mu-adopt - 2 notifiers
Sep 02 01:37:42 - mu-adopt - 10 functions
Sep 02 01:37:42 - mu-adopt - 1 endpoints
Sep 02 01:37:42 - mu-adopt - 1 search_domains
Sep 02 01:37:42 - mu-adopt - 4 nosqldbs
Sep 02 01:37:42 - mu-adopt - 6 jobs
Sep 02 01:37:42 - mu-adopt - 2 cdns

New resource types Job and CDN

We've added a type to support AWS CloudWatch Events, called Job in deference to future implementations for other providers, and CDN, for AWS CloudFront.

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/jobs.html
https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/cdns.html

---
appname: jobtest
jobs:
- name: a-job-for-a-thing
  schedule:
    minute: '0'
    hour: '1'
    day_of_month: '1'
    month: "*"
    day_of_week: "?"
    year: "*"
  targets:
  - type: functions
    name: do-thing
functions:
- name: do-thing
<...>

---
appname: cloudfront
cdns:
- name: test
  origins:
  - name: default
    bucket:
      name: bucket
  certificate:
    name: "*.example.com" # a certificate we already have in ACM
  dns_records:
  - zone:
      name: example.com # a domain we already have in Route53
  behaviors:
  - origin: default
    forwarded_values:
      headers:
      - Origin
      - Access-Control-Request-Headers
      - Access-Control-Request-Method
      - Access-Control-Allow-Origin
buckets:
- name: bucket
<...>

AWS::Function

Added a path option to the code stanza, which can be set to a directory on disk. This will be transparently zipped into a deployment package a la the existing zip_file option.

Commonly-used attributes of sibling resources, such as URLs, IP addresses, and ARNs, are now automatically exposed to Lambda functions as environment variables.

API Gateway integration should now behave correctly when specified either as one of an AWS Lambda Function's triggers, or from the endpoint side under methods => integrate_with.

Updates to code on disk should behave as expected when regrooming AWS Lambda resources.

The new invoke_on_completion stanza can be used to run an AWS Lambda function once its groom phase is complete.

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/functions.html

AWS::Endpoint

API Gateways are now taggable.

Method integrations are now more robust and support more functionality, including request parameters, timeouts (timeout_in_millis), . Asynchronous requests have a special async shortcut keyword to enable this behavior, as do AWS_PROXY integrations with the proxy keyword.

The cors keyword, instead of being a boolean, is now a string so you can specify the domain names that should match, e.g. * or foo.example.com.

Logging to CloudWatch now exposed to Basket of Kittens for configuration, via the log_requests keyword and access_logs stanza.

Alias endpoints are now supported with the domain_names stanza, and automatically integrate with Route53 when possible.

endpoints:
- name: api
  deploy_to: production
  log_requests: true
  methods:
  - path: "/"
    type: POST
    cors: "*"
    responses:
    - code: 200
      body:
      - is_error: false
        content_type: application/json
    integrate_with:
      name: some-lambda
      type: functions
      integration_http_method: POST
      async: true
      backend_http_method: POST
      passthrough_behavior: WHEN_NO_MATCH
  domain_names:
  - dns_record:
      zone:
        name: example.com
    certificate:
      name: "*.example.com"

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/endpoints.html

AWS::Bucket

The upload stanza is now available for populating an S3 bucket during the groom phase, and we have exposed S3's CORS (Cross-origin resource sharing) controls.

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/buckets.html

---
appname: s3test
buckets:
- name: mybucket
  web: true
  cors:
  - allowed_methods:
    - GET
    - POST
    allowed_origins:
    - "*"
  upload:
  - source: "code/build"
    destination: "/"

AWS::NoSQLDB

Added the populate stanza, which allows batch-insertion of records at creation time.

https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/nosqldbs.html

---
appname: dynamotest
nosqldbs:
- name: mytable
  read_capacity: 100
  write_capacity: 1
  attributes:
  - name: business_owner
    type: S
    primary_sort: true
  - name: domain
    type: S
    primary_partition: true
  populate:
  - business_owner: foo
    staff_division: bar
    operational_division: baz
    domain: example.gov
  - business_owner: bing
    staff_division: None
    operational_division: bong
    domain: example.com

Quality of Life Improvements

• AWS::SearchDomain now exposes access policies via our standard language with the policies stanza, available in addition to the existing raw access_policies. See https://cloudamatic.gitlab.io/mu/MU/Config/BasketofKittens/search_domains/policies.html
• AWS::ContainerCluster - Blind cleanups and lookups of EKS images should be significantly faster
• New CentOS 6 and CentOS 7 base images for AWS
• Even better Slack and console messaging from mu-deploy
• AWS::ServerPool: Modernize lookup of ALB target groups

Bugfixes

• AWS::NoSQLDB will actually apply tags to Dynamo tables
• AWS: pagination of large API result sets should be closer to universal
• MommaCat daemon startup should be somewhat better guarded
• AWS::MsgQueue and AWS::Notifier: improved resilience in their .find methods
• AWS::ContainerCluster myriad bitrot issues with ECS fixed
• AWS::Database bitrot in tests (t2.small instances seem deprecated) fixed by moving to t2.small
• Google::Role adoption bugfix for folders with nil bindings
• AWS::Role: Be sure we're binding custom inline policies properly on groom
• AWS::Function Adopt tags correctly