Feature/custom image template

mlops-multi-account-cdk/mlops-infra/ADVANCED_TOPICS.md

@@ -0,0 +1,39 @@
+# Advanced topics
+The topics defined here assume you have already deployed the solution once following the steps in the main [README](README.md)
+
+- [Advanced topics](#advanced-topics)
+  - [Setup CodeCommit with this repository](#setup-codecommit-with-this-repository)
+
+
+## Setup CodeCommit with this repository
+You would wonder after you have cloned this repository and deployed the solution how would you then start to interact with your deployed CodeCommit repository and start using it as a main repository and push changes to it. You have 2 options for this:
+1. Clone the created CodeCommit repository and start treating it seperately from this repository
+2. Just use this folder as a repository
+
+For the second option, you can do the following (while you are in the folder `mlops-infra`):
+```
+git init
+```
+this will create a local git for this folder which would be separate from the main so you can treat it as any git repo and it would not impact the main repository git. So, add the CodeCommit Repository as a remote source:
+```
+git remote add origin https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/mlops-infra
+```
+Ensure you have configured your machine to connect to CodeCommit and make `git push` or `git pull` commands to it; follow [Step 3 from the AWS documentation](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html).
+
+Now you can interact with the CodeCommit repository as normal. You will need to do the following for the first commit:
+```
+git add -A
+git commit -m "first commit"
+export AWS_PROFILE=mlops-governance
+git push origin main
+make init   # this will enable precommit which will now block any further pushes to the main branch
+```
+
+Ensure that your git uses the branch name **main** by default, otherwise the push command might fail and you will need to create a main branch then push changes through it.
+
+If you want to push the changes you made back to the main repository this folder belongs to you can just run this command:
+```
+rm -fr .git
+```
+This will remove the git settings from this folder so it would go back to the main repository settings. You can then raise a PR to include your changes to the main repository in GitHub.
+

mlops-multi-account-cdk/mlops-sm-project-template/.pre-commit-config.yaml

@@ -0,0 +1,75 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# SPDX-License-Identifier: MIT-0
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy of this
+# software and associated documentation files (the "Software"), to deal in the Software
+# without restriction, including without limitation the rights to use, copy, modify,
+# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
+# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+repos:
+# General
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.3.0
+  hooks:
+  - id: check-case-conflict
+  - id: detect-private-key
+  - id: trailing-whitespace
+  - id: end-of-file-fixer
+  - id: mixed-line-ending
+    args:
+      - --fix=lf
+    exclude: /package-lock\.json$
+  - id: check-added-large-files
+    args:
+      - --maxkb=1000
+  - id: check-merge-conflict
+  - id: no-commit-to-branch
+    args:
+      - --branch
+      - main
+  - id: pretty-format-json
+    args:
+      - --autofix
+      - --indent=2
+      - --no-sort-keys
+    exclude: /package-lock\.json$
+# Secrets
+- repo: https://github.com/awslabs/git-secrets
+  rev: b9e96b3212fa06aea65964ff0d5cda84ce935f38
+  hooks:
+  - id: git-secrets
+    entry: git-secrets --scan
+    files: .
+-   repo: https://github.com/psf/black
+    rev: 22.6.0
+    hooks:
+    -   id: black
+        args: ["--line-length=120"]
+-   repo: https://gitlab.com/PyCQA/flake8
+    rev: 3.9.2
+    hooks:
+    -   id: flake8
+        args: ["--ignore=E231,E501,F841,W503,F403,E266,W605,F541,F401,E302", "--exclude=app.py", "--max-line-length=120"]
+-   repo: https://github.com/Lucas-C/pre-commit-hooks
+    rev: v1.2.0
+    hooks:
+    -   id: forbid-crlf
+    -   id: remove-crlf
+    -   id: insert-license
+        files: \.(py|yaml)$
+-   repo: local
+    hooks:
+    -   id: clear-jupyter-notebooks
+        name: clear-jupyter-notebooks
+        entry: bash -c 'find . -type f -name "*.ipynb" -exec jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace "{}" \; && git add . && exit 0'
+        language: system
+        pass_filenames: false

mlops-multi-account-cdk/mlops-sm-project-template/ADVANCED_TOPICS.md

@@ -0,0 +1,94 @@
+# Advanced topics
+The topics defined here assume you have already deployed the solution once following the steps in the main [README](README.md)
+
+- [Advanced topics](#advanced-topics)
+  - [Setup CodeCommit with this repository](#setup-codecommit-with-this-repository)
+  - [Test the created sagemaker templates](#test-the-created-sagemaker-templates)
+
+
+## Setup CodeCommit with this repository
+You would wonder after you have cloned this repository and deployed the solution how would you then start to interact with your deployed CodeCommit repository and start using it as a main repository and push changes to it. You have 2 options for this:
+1. Clone the created CodeCommit repository and start treating it seperately from this repository
+2. Just use this folder as a repository
+
+For the second option, you can do the following (while you are in the folder `mlops-sm-project-template`):
+```
+git init
+```
+this will create a local git for this folder which would be separate from the main so you can treat it as any git repo and it would not impact the main repository git. So, add the CodeCommit Repository as a remote source:
+```
+git remote add origin https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/mlops-sm-project-template
+```
+Ensure you have configured your machine to connect to CodeCommit and make `git push` or `git pull` commands to it; follow [Step 3 from the AWS documentation](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html).
+
+Now you can interact with the CodeCommit repository as normal. You will need to do the following for the first commit:
+```
+git add -A
+git commit -m "first commit"
+export AWS_PROFILE=mlops-governance
+git push origin main
+make init   # this will enable precommit which will now block any further pushes to the main branch
+```
+
+Ensure that your git uses the branch name **main** by default, otherwise the push command might fail and you will need to create a main branch then push changes through it.
+
+If you want to push the changes you made back to the main repository this folder belongs to you can just run this command:
+```
+rm -fr .git
+```
+This will remove the git settings from this folder so it would go back to the main repository settings. You can then raise a PR to include your changes to the main repository in GitHub.
+
+
+## Test the created sagemaker templates
+***NOTE:** make sure to run `cdk synth` before running any of the commands defined below.*
+
+You will need to deploy the `service catalog stack` as that would setup your account with the required resources and ssm parameters before you can start testing your templates directly. If you don't have the service catalog stack already deployed in your account, you can achieve this by running the following command:
+```
+cdk --app ./cdk.out/assembly-Personal deploy —all --profile mlops-dev
+```
+
+otherwise make sure you have these ssm parameters defined:
+- in the dev account:
+  - /mlops/dev/account_id
+  - /mlops/code/seed_bucket
+  - /mlops/code/build
+  - /mlops/code/build/byoc
+  - /mlops/code/deploy
+- in the preprod account:
+  - /mlops/preprod/account_id
+  - /mlops/preprod/region
+- in the prod account:
+  - /mlops/prod/account_id
+  - /mlops/prod/region
+
+**OPTION 1** For quick testing of the sagemaker templates, you could deploy the json generated by CDK directly in your account by running the following command:
+```
+aws cloudformation deploy \
+	--template-file ./cdk.out/byoc-project-stack-dev.template.json \
+	--stack-name byoc-project-stack-dev \
+	--region eu-west-1 \
+	--capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
+	--disable-rollback \
+  --s3-bucket <any s3 bucket in the account, preferred cdk bootstrap bucket> \
+  --profile mlops-dev \
+	--parameter-overrides \
+	  SageMakerProjectName=mlops-test-0 \
+	  SageMakerProjectId=sm12340
+```
+This command will deploy the byoc project stack if you want to deploy other templates just change the `--template-file`, if you want to create a new stack you can change the other fields as well. 
+
+**OPTION 2** It is also possible to use CDK command for this exact purpose but this would require you to add the following to `app.py` file:
+```
+from mlops_sm_project_template.templates.byoc_project_stack import MLOpsStack
+
+MLOpsStack(
+    app,
+    "test",
+     env=deployment_env,
+)
+```
+The run `cdk synth` and then run the following to deploy:
+```
+cdk deploy test --parameters SageMakerProjectName=mlops-test \
+    --parameters SageMakerProjectId=sm1234 --profile mlops-dev
+```

mlops-multi-account-cdk/mlops-sm-project-template-rt/README.md → mlops-multi-account-cdk/mlops-sm-project-template/README.md

@@ -118,20 +118,20 @@ There are 2 way to trigger the deployment CI/CD Pipeline:
 - **Model Events** - These are events which get triggered through a status change to the model package group in SageMaker Model Registry.
 - **Code Events** - The pipeline is triggered on git update events over a specific branch, in this solution it is linked to the **main** branch.
 
-**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template_rt/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file.
+**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file.
 
 ### CodeCommit Stack
 *This stack is only needed if you want to handle deployments of this folder of the repository to be managed through a CICD pipeline.*
 
-This stack handles setting up an AWS CodeCommit repository for this folder of the repository. This repository will be used as the source for the CI/CD pipeline defined in [Pipeline Stack](#pipeline-stack). The repository will be named based on the value defined in `mlops_sm_project_template_rt/config/constants.py` with this variable `CODE_COMMIT_REPO_NAME`. The repository will be intialised with a default branch as defined in the `constants.py` file under `PIPELINE_BRANCH` variable.
+This stack handles setting up an AWS CodeCommit repository for this folder of the repository. This repository will be used as the source for the CI/CD pipeline defined in [Pipeline Stack](#pipeline-stack). The repository will be named based on the value defined in `mlops_sm_project_template/config/constants.py` with this variable `CODE_COMMIT_REPO_NAME`. The repository will be intialised with a default branch as defined in the `constants.py` file under `PIPELINE_BRANCH` variable.
 
 ### Pipeline Stack
 
 *This stack is only needed if you want to handle deployments of this folder of repository to be managed through a CICD pipeline. The pipeline is configured to deploy to 1 account: DEV and will deploy the service catalog stack to the target account*
 
 The CICD pipeline in this repository is setup to monitor an AWS CodeCommit repository as defined in [CodeCommit Stack](#codecommit-stack).
 
-If you are using other sources like github or bitbucket for your repository, you will need to modify the connection to the appropriate repository as defined in `mlops_sm_project_template_rt/pipeline_stack.py`. This can be done using AWS CodeStar but must be setup on the account.
+If you are using other sources like github or bitbucket for your repository, you will need to modify the connection to the appropriate repository as defined in `mlops_sm_project_template/pipeline_stack.py`. This can be done using AWS CodeStar but must be setup on the account.
 
 Make sure the pipelines also point to your targeted branch; by default the pipeline is linked to `main` branch events, this is defined in the `constants.py` file under `PIPELINE_BRANCH` variable.
 
@@ -162,7 +162,7 @@ This is an AWS CDK project written in Python 3.8. Here's what you need to have o
 ├── app.py
 ├── cdk.json
 ├── diagrams
-├── mlops_sm_project_template_rt
+├── mlops_sm_project_template
 │   ├── README.md
 │   ├── __init__.py
 │   ├── cdk_helper_scripts
@@ -219,7 +219,7 @@ aws_session_token = YOUR_SESSION_TOKEN  # this token is generated if you are usi
 ...
 ```
 
-Before you start with the deployment of the solution make sure to bootstrap your accounts. Ensure you add the account details in `mlops_sm_project_template_rt/config/constants.py` mainly the target deployment accounts: **DEV**, **PREPROD** and **PROD**.
+Before you start with the deployment of the solution make sure to bootstrap your accounts. Ensure you add the account details in `mlops_sm_project_template/config/constants.py` mainly the target deployment accounts: **DEV**, **PREPROD** and **PROD**.
 ```
 PIPELINE_ACCOUNT = ""     # account to host the pipeline handling updates of this repository
 

mlops-multi-account-cdk/mlops-sm-project-template-rt/app.py → mlops-multi-account-cdk/mlops-sm-project-template/app.py

@@ -18,9 +18,9 @@
 
 import aws_cdk as cdk
 import os
-from mlops_sm_project_template_rt.pipeline_stack import PipelineStack, CoreStage
-from mlops_sm_project_template_rt.codecommit_stack import CodeCommitStack
-from mlops_sm_project_template_rt.config.constants import DEFAULT_DEPLOYMENT_REGION, PIPELINE_ACCOUNT, DEV_ACCOUNT
+from mlops_sm_project_template.pipeline_stack import PipelineStack, CoreStage
+from mlops_sm_project_template.codecommit_stack import CodeCommitStack
+from mlops_sm_project_template.config.constants import DEFAULT_DEPLOYMENT_REGION, PIPELINE_ACCOUNT, DEV_ACCOUNT
 
 app = cdk.App()
 

mlops-multi-account-cdk/mlops-sm-project-template-rt/mlops_sm_project_template_rt/codecommit_stack.py → mlops-multi-account-cdk/mlops-sm-project-template/mlops_sm_project_template/codecommit_stack.py

@@ -26,10 +26,7 @@
 
 from constructs import Construct
 
-from mlops_sm_project_template_rt.config.constants import (
-    CODE_COMMIT_REPO_NAME,
-    PIPELINE_BRANCH
-)
+from mlops_sm_project_template.config.constants import CODE_COMMIT_REPO_NAME, PIPELINE_BRANCH
 
 
 class CodeCommitStack(Stack):
@@ -52,7 +49,7 @@ def __init__(
             "DeployAsset",
             path="",
             bundling=BundlingOptions(
-                image=DockerImage.from_build("mlops_sm_project_template_rt/cdk_helper_scripts/zip-image"),
+                image=DockerImage.from_build("mlops_sm_project_template/cdk_helper_scripts/zip-image"),
                 command=[
                     "sh",
                     "-c",

mlops-multi-account-cdk/mlops-sm-project-template-rt/mlops_sm_project_template_rt/config/constants.py → mlops-multi-account-cdk/mlops-sm-project-template/mlops_sm_project_template/config/constants.py

@@ -15,7 +15,7 @@
 # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
 # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
-CODE_COMMIT_REPO_NAME = "mlops-sm-project-template-rt"
+CODE_COMMIT_REPO_NAME = "mlops-sm-project-template"
 PIPELINE_BRANCH = "main"
 
 PIPELINE_ACCOUNT = ""  # account used to host the pipeline handling updates of this repository

mlops-multi-account-cdk/mlops-sm-project-template-rt/mlops_sm_project_template_rt/pipeline_stack.py → mlops-multi-account-cdk/mlops-sm-project-template/mlops_sm_project_template/pipeline_stack.py

@@ -26,15 +26,15 @@
 
 from constructs import Construct
 
-from mlops_sm_project_template_rt.config.constants import (
+from mlops_sm_project_template.config.constants import (
     APP_PREFIX,
     CODE_COMMIT_REPO_NAME,
     DEV_ACCOUNT,
     DEFAULT_DEPLOYMENT_REGION,
     PIPELINE_BRANCH,
 )
 
-from mlops_sm_project_template_rt.service_catalog_stack import ServiceCatalogStack
+from mlops_sm_project_template.service_catalog_stack import ServiceCatalogStack
 
 
 class CoreStage(Stage):