Updated docs for ECR

- Added documentation for ECR
- Removed the pre-generated code from the docs as it made searching worse

Dockerfile

@@ -3,9 +3,26 @@ FROM public.ecr.aws/lambda/python:3.12
 
 ENV LAMBDA_TASK_ROOT=/var/runtime
 
-# Copy Starfleet over (just a quick and dirty sample, not the cleanest):
+# Copy Starfleet over:
 COPY ./ ${LAMBDA_TASK_ROOT}/starfleet
 
+# Perform cleanup of things the Docker doens't need:
+RUN rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/tests && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/.tox && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/.pytest_cache && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/deploy && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/site && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/.kubelr && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/build && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/.coverage && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/*.ini && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/*.yaml && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/*.md && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/sample_samconfig.toml && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/Dockerfile && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/venv && \
+    rm -Rf ${LAMBDA_TASK_ROOT}/starfleet/env
+
 # Install the specified packages:
 RUN cd ${LAMBDA_TASK_ROOT}/starfleet && \
   pip install . && \

mkdocs.yml

@@ -38,9 +38,6 @@ theme:
 plugins:
   - search
   - blog
-  - gen-files:
-      scripts:
-        - mkdocs/gen_ref_pages.py
   - literate-nav:
       nav_file: SUMMARY.md
   - mkdocstrings
@@ -87,6 +84,7 @@ nav:
     - Set up IAM: installation/IAM.md
     - Prepare Configuration: installation/PrepareConfiguration.md
     - Prepare Payload Templates: installation/PreparePayload.md
+    - Set up ECR: installation/SetupECR.md
     - Run AWS SAM: installation/RunSAM.md
     - Make the Account Index: installation/MakeAccountIndex.md
     - Wrapping Up: installation/WrapUp.md
@@ -135,4 +133,3 @@ nav:
   - Extras:
     - Overview: Links.md
     - Attributions: Attributions.md
-    - Code Reference: reference/

mkdocs/Links.md

@@ -6,6 +6,3 @@ See [the attributions](Attributions.md) page for additional attributions for thi
 ## GitHub
 1. Main repository link: [https://github.com/gemini-oss/starfleet](https://github.com/gemini-oss/starfleet)
 1. Other cool Gemini OSS projects: [https://github.com/gemini-oss](https://github.com/gemini-oss)
-
-## Code Reference
-We also have the [code to reference here](/starfleet/reference/starfleet/).

mkdocs/blog/posts/ecr.md

@@ -0,0 +1,58 @@
+---
+date: 2024-04-08
+authors:
+  - mikegrima
+categories:
+  - General Announcement
+---
+
+# ECR/Dockerized Lambda Support
+We made a small update to the docs to highlight deploying Starfleet with a Dockerized Lambda. This was necessary as Starfleet's dependencies made it larger than the `.zip` file size limit for Lambda.
+
+The main steps to convert to this from a non-ECR set up is to:
+
+1. Create the Private ECR repo in the same account/region you have Starfleet deployed in.
+2. Update the `samconfig.toml` file.
+3. Update the SAM template.
+4. Re-build and deploy.
+
+The Dockerized Lambda is built with the included `Dockerfile`, which should have everything needed and ready to go. You can test that the container builds by running `docker build .` from within the Starfleet directory.
+
+## Update SAM Config
+Once you create your ECR Repo, you then need to update your `samconfig.toml` file to include the line:
+```
+image_repository = "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/REPONAME"
+```
+... under your `.deploy.parameters` sections.
+
+## Update your SAM Template
+The SAM template also needs to be updated. Included now are 2 sample SAM Templates: `test_sam_template.yaml`, which is the ECR sample template, and `test_sam_template_NO_ECR.yaml`, which is the original non-ECR version.
+
+You would need to update all the function entires to change it from:
+```yaml
+  StarbaseEventBridgeFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      CodeUri: ./src
+      Handler: starfleet.starbase.entrypoints.eventbridge_timed_lambda_handler
+      Runtime: python3.12
+```
+over to:
+
+```yaml
+  StarbaseEventBridgeFunction:
+    Type: AWS::Serverless::Function
+    Metadata:
+      DockerTag: starfleet
+      DockerContext: ./
+      Dockerfile: Dockerfile
+    Properties:
+      PackageType: Image
+      ImageConfig:
+        Command:
+          - starfleet.starbase.entrypoints.eventbridge_timed_lambda_handler
+```
+
+Update all the Lambda definitions to include the `Metadata` section as shown above (this is the same for all the functions) and update the `Properties` section to have the fields above. Remove the old fields. Note: the `Command` field is where you place the Lambda handler path - this is unique to each Lambda function.
+
+For more information see [the ECR setup documentation](../../installation/SetupECR.md) and also [AWS's documentation here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html).

mkdocs/installation/RunSAM.md

@@ -6,11 +6,17 @@ Now you are ready to run AWS SAM and get the infrastructure components deployed!
     This section could use some assistance from the community for simplifying the installation process and documentation. We would love for assistance on the creation of things like an installation script, for example that would set all of this up for you.
 
 ## The Test Template
-We include a file called `test_sam_template.yaml`, so you can later clone it for production with production specific values in. The commands below will assume that the template is named `test_sam_template.yaml`.
+We include a file called `test_sam_template.yaml` (for ECR) and `test_sam_template_NO_ECR.yaml`(for non-ECR), so you can later clone it for production with production specific values in. The commands below will assume that the template is named `test_sam_template.yaml`.
 
 ## Build Starfleet
 To build the code, in the main checked out directory you will run:
 
+If you are using ECR:
+```bash
+sam build --template-file test_sam_template.yaml --parameter-overrides ParameterKey=EnvironmentName,ParameterValue=TEST
+```
+
+If you are not using ECR:
 ```bash
 sam build --use-container --template-file test_sam_template.yaml --parameter-overrides ParameterKey=EnvironmentName,ParameterValue=TEST
 ```
@@ -59,6 +65,7 @@ We need to update the SAM configuration, _manually_, to allow it to create IAM r
 [TEST.deploy.parameters]
 # ...
 capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"]
+image_repository = "YOUR_ACCOUNT_ID.dkr.ecr.YOUR_REGION.amazonaws.com/REPO_NAME -- if you are using ECR"
 
 [TEST.validate.parameters]
 region = "YOUR_REGION_HERE"
@@ -137,6 +144,7 @@ At this point you will have the following:
 - [x] Starfleet worker IAM roles deployed everywhere
 - [x] The `configuration.yaml` file in `src/starfleet/configuration_files` modified with values unique to your environment
 - [x] A payload template (not stored as a configuration file) in a different place than your configuration that describes what the Starfleet Account Index Generator is supposed to do
+- [x] An optional ECR Repository set up to make dockerized Lambda builds
 - [x] And Now: AWS SAM:
     - [x] SAM's administrative resources deployed
     - [x] SAM's TEST deployment configuration all set up

mkdocs/installation/SetupECR.md

@@ -0,0 +1,52 @@
+# Set Up Private ECR (Elastic Container Registry)
+
+This is not required but highly recommended as the Starfleet package may be too big for Lambda to accept it as a `.zip` file. As such, we recommend setting up Starfleet with a Dockerized Lambda hosted in a private ECR.
+
+The steps to do this will largely involve setting up ECR, which you can find in [the AWS documentation here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html).
+
+When creating your registry please adhere to the following:
+
+1. You are creating a **PRIVATE** registry. **DO NOT MAKE THIS PUBLIC!**
+2. This must reside in the same account and region that you are deploying Starfleet to
+3. Recommend that you tag immutability
+4. Recommend that you set up a lifecycle policy to expire older images (recommend to expire any `Image count more than 3`)
+
+!!! danger "Public Access"
+    DO NOT make your ECR repository public!
+
+Once you have your ECR registry set up, there will be a URL associated with it. That URL is the where your image will be pushed to. You will need that in the next steps for AWS SAM.
+
+!!! note "Note: ECR URL"
+    You will need the ECR URL for the next sections. This is in the format of `aws_account_id.dkr.ecr.region.amazonaws.com`
+
+## Note about Dockerizing Starfleet
+Included in the main code base is a `Dockerfile` that is intended to build Starfleet. This file should for the most part be all good to go. This is set up to use the AWS official Lambda docker container for Python. This also:
+
+1. Copies the Starfleet code to the container in the `var/runtime` directory, which is where Lambda wants the code to live.
+2. Cleans up some unnecessary files.
+2. It runs `pip install` to install the Starfleet packages
+3. Lastly, and _very importantly_ it **removes** the built-in `boto3` and `botocore` packages that is shipped by default with the container. This is necessary because Lambda will prefer to use the preloaded version of boto instead of the Starfleet packaged one. The pre-loaded version can be out of date and can result in very bizarre errors with unsupported boto APIs - this is because you may be using boto features that are newer than the included boto packages. Thus, we remove them so only the Starfleet included boto packages are there.
+
+For more information about Dockerizing Lambda functions, [see the docs here](https://docs.aws.amazon.com/lambda/latest/dg/python-image.html).
+
+## Continuing
+
+At this point you will have the following:
+
+- [x] Enable AWS Organizations if you haven't already done so and move some accounts into it
+- [x] Pick out an AWS account for deploying a testing version of Starfleet
+- [x] Work on getting a read-only Starfleet IAM role deployed with the permissions outlined above in all your AWS accounts. This role is _not_ very permissive and is only able to describe the enabled regions for an account.
+    - [x] In the organization root, it has permissions to list the Organizations accounts.
+    - [x] If you use StackSets then you need to manually make the role in the org root since StackSets won't operate in the org root.
+    - [x] Important: Make sure that you have some IAM principal that you can use locally that can assume all these roles. This will be needed to run the Starfleet CLI. If you use AWS SSO, then use the ARN for the permissions set provisioned administrative role in the Starfleet account. See the note above for an example.
+- [x] AWS Account identified for deployment
+- [x] Starfleet worker IAM roles deployed everywhere
+- [x] The `configuration.yaml` file in `src/starfleet/configuration_files` modified with values unique to your environment
+- [x] A payload template (not stored as a configuration file) in a different place than your configuration that describes what the Starfleet Account Index Generator is supposed to do
+- [x] An optional ECR Repository set up to make dockerized Lambda builds
+- [x] And Now: AWS SAM:
+    - [x] SAM's administrative resources deployed
+    - [x] SAM's TEST deployment configuration all set up
+    - [x] Starfleet deployed in your environment
+
+While it's now deployed, it won't work without an Account Index. The next section describes how to get it set up.

sample_samconfig.toml

@@ -9,7 +9,7 @@ region = "REPLACEME"
 confirm_changeset = true
 capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"]  # Important -- you need these capabilities defined since this creates IAM roles
 parameter_overrides = "EnvironmentName=\"TEST\""
-image_repositories = []
+image_repository = "REPLACE ME - IF YOU ARE  NOT USING ECR DELETE THIS LINE"
 
 [TEST.validate.parameters]
 region = "REPLACEME"
@@ -31,7 +31,7 @@ region = "REPLACEME"
 confirm_changeset = true
 capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"]  # Important -- you need these capabilities defined since this creates IAM roles
 parameter_overrides = "EnvironmentName=\"PROD\""
-image_repositories = []
+image_repository = "REPLACE ME - IF YOU ARE  NOT USING ECR DELETE THIS LINE"
 
 [PROD.validate.parameters]
 region = "REPLACEME"

test_sam_template.yaml

@@ -1,3 +1,4 @@
+# This is the ECR version of the SAM template:
 AWSTemplateFormatVersion: '2010-09-09'
 Transform: AWS::Serverless-2016-10-31
 Description: >
@@ -126,10 +127,15 @@ Resources:
   # The Starbase EventBridge listener:
   StarbaseEventBridgeFunction:
     Type: AWS::Serverless::Function
+    Metadata:
+      DockerTag: starfleet
+      DockerContext: ./
+      Dockerfile: Dockerfile
     Properties:
-      CodeUri: ./src
-      Handler: starfleet.starbase.entrypoints.eventbridge_timed_lambda_handler
-      Runtime: python3.12
+      PackageType: Image
+      ImageConfig:
+        Command:
+          - starfleet.starbase.entrypoints.eventbridge_timed_lambda_handler
       Architectures:
         - arm64
       Events:
@@ -223,10 +229,15 @@ Resources:
 
   AccountIndexGenerator:
     Type: AWS::Serverless::Function
+    Metadata:
+      DockerTag: starfleet
+      DockerContext: ./
+      Dockerfile: Dockerfile
     Properties:
-      CodeUri: ./src
-      Handler: starfleet.worker_ships.plugins.account_index_generator.ship.lambda_handler
-      Runtime: python3.12
+      PackageType: Image
+      ImageConfig:
+        Command:
+          - starfleet.worker_ships.plugins.account_index_generator.ship.lambda_handler
       Architectures:
         - arm64
       MemorySize: 256
@@ -265,10 +276,15 @@ Resources:
 #
 #  AWSConfigWorker:
 #    Type: AWS::Serverless::Function
+#    Metadata:
+#      DockerTag: starfleet
+#      DockerContext: ./
+#      Dockerfile: Dockerfile
 #    Properties:
-#      CodeUri: ./src
-#      Handler: starfleet.worker_ships.plugins.aws_config.ship.lambda_handler
-#      Runtime: python3.12
+#      PackageType: Image
+#      ImageConfig:
+#        Command:
+#          - starfleet.worker_ships.plugins.aws_config.ship.lambda_handler
 #      Architectures:
 #        - arm64
 #      MemorySize: 128
@@ -305,10 +321,15 @@ Resources:
 #
 #  GitHubSyncWorker:
 #    Type: AWS::Serverless::Function
+#    Metadata:
+#      DockerTag: starfleet
+#      DockerContext: ./
+#      Dockerfile: Dockerfile
 #    Properties:
-#      CodeUri: ./src
-#      Handler: starfleet.worker_ships.plugins.github_sync.ship.lambda_handler
-#      Runtime: python3.12
+#      PackageType: Image
+#      ImageConfig:
+#        Command:
+#          - starfleet.worker_ships.plugins.github_sync.ship.lambda_handler
 #      Architectures:
 #        - arm64
 #      MemorySize: 128
@@ -346,10 +367,15 @@ Resources:
 #
 #  IamRoleWorker:
 #    Type: AWS::Serverless::Function
+#    Metadata:
+#      DockerTag: starfleet
+#      DockerContext: ./
+#      Dockerfile: Dockerfile
 #    Properties:
-#      CodeUri: ./starfleet/src
-#      Handler: starfleet.worker_ships.plugins.iam.role_ship.lambda_handler
-#      Runtime: python3.12
+#      PackageType: Image
+#      ImageConfig:
+#        Command:
+#          - starfleet.worker_ships.plugins.iam.role_ship.lambda_handler
 #      Architectures:
 #        - arm64
 #      MemorySize: 128
@@ -373,10 +399,15 @@ Resources:
     Type: AWS::Serverless::Function
     DependsOn:
       - AccountIndexGeneratorQueue
+    Metadata:
+      DockerTag: starfleet
+      DockerContext: ./
+      Dockerfile: Dockerfile
     Properties:
-      CodeUri: ./src
-      Handler: starfleet.starbase.entrypoints.fanout_payload_lambda_handler
-      Runtime: python3.12
+      PackageType: Image
+      ImageConfig:
+        Command:
+          - starfleet.starbase.entrypoints.fanout_payload_lambda_handler
       Architectures:
         - arm64
       Events: