feat(package/deploy): new option `--image-repositories` #2465

sriram-mv · 2020-12-08T22:12:00Z

Which issue(s) does this change fix?

#2447
It does not fix this issue, but allows for mapping of function to different ECR if required.

Why is this change necessary?

Allow for function to ecr mapping. each function can now be uploaded to a different ECR.

How does it address the issue?

Guided deploy prompts for an ECR per function and saves it.

What side effects does this change have?

if both --image-repository and --image-repositories are specifed. One has to be removed. Guided flow looks to fill --image-repositories but is not destructive to the configuration file and does not remove --image-repository if its already set.

Checklist

Write design document (Do I need to write a design document?)
Write unit tests
Write/update functional tests
Write/update integration tests
make pr passes
make update-reproducible-reqs if dependencies were changed
Write documentation

NOTE: WIP and needs more tests.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

samcli/commands/_utils/options.py

samcli/cli/types.py

samcli/lib/cli_validation/image_repository_validation.py

samcli/commands/deploy/utils.py

samcli/commands/deploy/guided_config.py

samcli/commands/_utils/options.py

mndeveci · 2020-12-10T23:48:58Z

samcli/commands/_utils/template.py

+    for resource_id, resource in template_dict.get("Resources", {}).items():
+        if resource.get("Properties", {}).get("PackageType", ZIP) == artifact and resource.get("Type") in [


There was an existing function collector, which is been used in build command, can we use that one instead?

does that process the template? I will take a look at it.

its very specific to build, we would need some refactoring to make it more generic so that we can share that.

I would still use SamFunctionProvider https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/providers/sam_function_provider.py#L16 which will handle parameter overrides etc.

There are some circular dependencies that we need to take care of here, I vote we do this in a separate PR.

samcli/lib/cli_validation/image_repository_validation.py

sriram-mv · 2020-12-11T05:34:44Z

@mndeveci : if we dont special case how we save to the config file. It saves as below, thats why we need it.

[default.deploy.parameters.image_repositories]
HelloWorldFunction = "123456789012.dkr.ecr.us-east-1.amazonaws.com/sam-cli"

mndeveci

Thanks for working on the feedbacks, left only one comment for collecting the functions from the template. Other than that it looks good.

moelasmar · 2020-12-14T23:50:46Z

samcli/commands/deploy/deploy_context.py

@@ -129,7 +131,7 @@ def run(self):
        print_deploy_args(
            self.stack_name,
            self.s3_bucket,
-            self.image_repository,
+            self.image_repositories if isinstance(self.image_repositories, dict) else self.image_repository,


[nit] I think if you check that self.image_repositories is not None will be more readable

metaskills · 2021-01-05T02:26:35Z

There is still no documentation on sam package (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-package.html) guides page. Trying to find the format to use here because I think this arg will allow me to deploy two functions with the same image.

* feat(package/deploy): new option `--image-repositories` * tests(package/deploy): unit and integration tests * fix(lint): make lint happy * fix(comments): address PR comments * fix(image-repositories): change callback to default to {} Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com>

JamesDougherty · 2021-08-13T15:43:23Z

Hopefully this will help someone and save them time. There is no documentation on this as @metaskills mentioned and no examples on how to use it. So after spending more time than needed on it and looking through the source code on this, I was able to figure out what they intended.

Let's say you have two functions in your template file that use the Image package type, like the following:

FooFunction:   # <------------------This is a functions logical ID, take note of this.
  Type: AWS::Serverless::Function
  Properties:
    PackageType: Image
    ...
  Metadata:
    DockerTag: latest
    DockerContext: ./opt/dist/functions/foo-function/        # Wherever your Dockerfile is
    Dockerfile: Dockerfile

BarFunction:   # <------------------This is a functions logical ID, take note of this.
  Type: AWS::Serverless::Function
  Properties:
    PackageType: Image
    ...
  Metadata:
    DockerTag: latest
    DockerContext: ./opt/dist/functions/bar-function/        # Wherever your Dockerfile is
    Dockerfile: Dockerfile

After you run the sam build command it will add the following ImageUri to the template it generates in the .aws-sam/build directory. The ImageUri consists of the image repository name (functions logical ID in all lower case) and DockerTag concatenated. Here is an example of what it will add for the functions defined above:

FooFunction:
  Type: AWS::Serverless::Function
  Properties:
    PackageType: Image
    ImageUri: foofunction:latest  # <------------- Added by SAM Build
    ...
  Metadata:
    DockerTag: latest
    DockerContext: ./opt/dist/functions/foo-function/
    Dockerfile: Dockerfile

BarFunction:
  Type: AWS::Serverless::Function
  Properties:
    PackageType: Image
    ImageUri: barfunction:latest  # <------------- Added by SAM Build
    ...
  Metadata:
    DockerTag: latest
    DockerContext: ./opt/dist/functions/bar-function/
    Dockerfile: Dockerfile

Now when you run the sam deploy command you need to add the --image-repositories in the following format:

--image-repositories <Function Logical ID>=<Function Image Repository>, <Function Logical ID>=<Function Image Repository>, ...

Where:
Function Logical ID = The functions logical ID as defined in the SAM template
Function Image Repository = The functions ECR image repository that SAM created in ECR URL format

So using the above example we would call sam deploy using the following:

sam deploy ... --image-repositories FooFunction=<account id>.dkr.ecr.<region>.amazonaws.com/foofunction, BarFunction=<account id>.dkr.ecr.<region>.amazonaws.com/barfunction

Where:
Account ID = Your AWS account ID
Region = Your AWS region (like us-east-1)

I have this automated using a PowerShell script so you will just need to figure out how to handle it using your tooling / use case.

Enjoy.

ygorth · 2021-08-20T20:00:44Z

The example from @JamesDougherty is wrong and will throw an error like this:

Error: Got unexpected extra argument (Function_Logical_ID=account_id.dkr.ecr.us-east-1.amazonaws.com/Function_Image_Repository)

Please, find the correct command below:
sam deploy ... --image-repositories <Function Logical ID>=<account id>.dkr.ecr.<region>.amazonaws.com/foofunction --image-repositories <Function Logical ID>=<account id>.dkr.ecr.<region>.amazonaws.com/barfunction

Option description: --image-repositories Specify mapping of Function Logical ID to ECR Repo uri, of the form Function_Logical_ID=ECR_Repo_Uri. This option can be specified multiple times.

My current version is 1.29.0.

JamesDougherty · 2021-08-21T15:12:02Z

Hello @ygorth - Thank you for the correction. I was only testing with one repository at the moment and made the assumption that they designed it how the parameter overrides work "sam deploy ... --parameter-overrides foo=bar, hello=world". Of course it would be different. This is why it would be nice to have some official documentation so we don't have to assume anything. Anyway, thanks again.

jmamin · 2023-02-13T17:06:33Z

Hello,

Any advice how to fix this issue? image name is compatible with format and regex but still errors out!:

MacBook-Pro:lambda-local jamin$ sam deploy --guided
Usage: sam deploy [OPTIONS]
Try 'sam deploy -h' for help.

Error: Invalid value for '--image-repositories': --image-repositories is not a valid format, it needs to be of the form function_logical_id=ECR_URI
MacBook-Pro:lambda-local jamin$

samconfig.toml
version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "everflowclicks"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-1785ltpwmjlwc"
s3_prefix = "everflowclicks"
region = "us-west-2"
confirm_changeset = true
capabilities = "CAPABILITY_IAM"
image_repositories = ["880954911558.dkr.ecr.us-west-2.amazonaws.com/everflow-app"]
disable_rollback = true

sriram-mv requested review from jfuss and mndeveci December 8, 2020 22:12

jfuss marked this pull request as draft December 9, 2020 17:26

mndeveci reviewed Dec 9, 2020

View reviewed changes

sriram-mv changed the title ~~[WIP] feat(package/deploy): new option --image-repositories~~ feat(package/deploy): new option --image-repositories Dec 10, 2020

sriram-mv marked this pull request as ready for review December 10, 2020 17:57

sriram-mv force-pushed the multi_image_repo branch from f147483 to 313181e Compare December 10, 2020 22:46

mndeveci reviewed Dec 10, 2020

View reviewed changes

sriram-mv added 5 commits December 10, 2020 21:50

feat(package/deploy): new option --image-repositories

cd0a425

tests(package/deploy): unit and integration tests

bda29de

fix(lint): make lint happy

b81695e

fix(comments): address PR comments

45a6fef

fix(image-repositories): change callback to default to {}

27ce6e2

sriram-mv force-pushed the multi_image_repo branch from 076945b to 27ce6e2 Compare December 11, 2020 05:50

mndeveci approved these changes Dec 11, 2020

View reviewed changes

Merge branch 'develop' into multi_image_repo

a08edfb

moelasmar approved these changes Dec 15, 2020

View reviewed changes

Merge branch 'develop' into multi_image_repo

0138307

mndeveci merged commit 07fcd61 into aws:develop Dec 16, 2020

metaskills mentioned this pull request Dec 19, 2020

Use New --image-repositories rails-lambda/lamby-cookiecutter#6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(package/deploy): new option `--image-repositories` #2465

feat(package/deploy): new option `--image-repositories` #2465

sriram-mv commented Dec 8, 2020 •

edited

mndeveci Dec 10, 2020

sriram-mv Dec 11, 2020

sriram-mv Dec 11, 2020

mndeveci Dec 11, 2020

sriram-mv Dec 12, 2020

sriram-mv commented Dec 11, 2020 •

edited

mndeveci left a comment

moelasmar Dec 14, 2020

metaskills commented Jan 5, 2021

JamesDougherty commented Aug 13, 2021

ygorth commented Aug 20, 2021 •

edited

JamesDougherty commented Aug 21, 2021

jmamin commented Feb 13, 2023

		for resource_id, resource in template_dict.get("Resources", {}).items():
		if resource.get("Properties", {}).get("PackageType", ZIP) == artifact and resource.get("Type") in [

feat(package/deploy): new option --image-repositories #2465

feat(package/deploy): new option --image-repositories #2465

Conversation

sriram-mv commented Dec 8, 2020 • edited

Which issue(s) does this change fix?

Why is this change necessary?

How does it address the issue?

What side effects does this change have?

Checklist

mndeveci Dec 10, 2020

Choose a reason for hiding this comment

sriram-mv Dec 11, 2020

Choose a reason for hiding this comment

sriram-mv Dec 11, 2020

Choose a reason for hiding this comment

mndeveci Dec 11, 2020

Choose a reason for hiding this comment

sriram-mv Dec 12, 2020

Choose a reason for hiding this comment

sriram-mv commented Dec 11, 2020 • edited

mndeveci left a comment

Choose a reason for hiding this comment

moelasmar Dec 14, 2020

Choose a reason for hiding this comment

metaskills commented Jan 5, 2021

JamesDougherty commented Aug 13, 2021

ygorth commented Aug 20, 2021 • edited

JamesDougherty commented Aug 21, 2021

jmamin commented Feb 13, 2023

feat(package/deploy): new option `--image-repositories` #2465

feat(package/deploy): new option `--image-repositories` #2465

sriram-mv commented Dec 8, 2020 •

edited

sriram-mv commented Dec 11, 2020 •

edited

ygorth commented Aug 20, 2021 •

edited