(aws cloudformation package) add support for URIs from 3rd party artifact repositories

[j5nb4l]

Describe the feature

The aws cloudformation package should optionally support the property values to be a 3rd party artifact repositories URI. When provided, the command will attempt to download the artifact from the provided URI, upload it to S3, and replace the property value with the new S3 URI.

Use Case

In our CI/CD pipeline, it is common to download artifacts from an artifact repository (like Artifactory) just to upload it to S3 so it can be used by CloudFormation.

Let's take the template snippet below as an example.

myFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: myFunction
    Runtime: python3.8
    Role: !GetAtt LambdaBasicExecutionRole.Arn
    Handler: lambda_function.handler
    Code: myArtifact.zip

At the moment, the CI/CD pipeline will have to download the artifact from the artifact repository, and store it in the same directory where the package command will be executed with the same name defined in the template. If there are any discrepancies, the artifact will not be found and the command will fail and ultimately so will the pipeline.

Proposed Solution

It would be better developer experience if package command could support downloading the artifact from the provided URI and upload it to S3. The template snippet would look something like the one below:

PythonFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: cfn-workshop-python-function
    Description: Python Function to return specific TimeZone time
    Runtime: python3.8
    Role: !GetAtt LambdaBasicExecutionRole.Arn
    Handler: lambda_function.handler
    Code: https://artifactory.domain.com:443/artifactory/repository/path/myArtifact.zip

Other Information

This could be supported by adding the logic below to the export method of the Resource class found here.

temp_dir = None
if is_repository_url(property_value):
     if self.support_artifact_repositories:
           LOG.debug("Property is an artifact repository URI but --support-artifact-repositories was not provided.")
           raise exceptions.ExportFailedError()
     if not is_approved_repository_domain(property_value):
           LOG.debug("Property is an artifact repository URI, but host is not found in the --approved-artifact-repository-domains list.")
           raise exceptions.ExportFailedError()
     temp_dir = download_artifact_to_temp_dir(property_value)
     set_value_from_jmespath(resource_dict, self.PROPERTY_NAME, temp_dir)

In the example above, I added a conditions to check whether the support should be enabled and whether the URI provided is part of a list of approved domains, as I don't think we want the package command to download artifacts by default or from anywhere. With that in mind, I think we could add two optional parameters here, maybe something like:

ARG_TABLE = [
        {
            'name': 'support-artifact-repositories',
            'action': "store_true",
            'help_text': (
                'Indicates whether to support downloading artifacts from 3rd party artifact repositories.'
                ' Defaults to False.'
            )
        },
        {
            'name': 'approved-artifact-repository-domains',
            'action': 'store',
            'schema': {
                'type': 'array',
                'items': { 
                    'type': 'string'
                }
            },
            'default': [],
            'help_text': (
                'A list of approved domains where artifacts can be downloaded'
                'to be uploaded to the Amazon S3 bucket. If not provided, all domains'
                'will be denied.'
                'Syntax: domain1.com domain2.com ...'
            )
        },
]

Please note: This is just an example of how it could be done. I'm sure there are other questions to be considered when actually implementing it, like how to handle authentication or options to perform any actions on the downloaded artifact before uploading it.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CLI version used

aws-cli/2.10.4

Environment details (OS name and version, etc.)

Python/3.11.2 Darwin/21.6.0 source/x86_64 prompt/off

[tim-finnigan]

Hi @j5nb4l thanks for creating this feature request. It sounds like your use case might overlap with this existing issue (although the approach is different): #4727. Can you confirm, or note any other major distinctions between the two?

[j5nb4l]

Hello @tim-finnigan, thank you for your response. I looked over the request on issue #4727, and I am confident it is not related.

#4727 is requesting support to configure the endpoint_url property of the s3 client used to perform the PutObject request made by the package command. In contrast, this request is about enhancing the package command with the ability to download an artifact from an artifact repository, like Artifactory, and then proceed to upload it as it would with any other local files.

I hope this clears up the confusion. Please let me know if you have any other questions.

[tim-finnigan]

Ok thanks for clarifying! I was thinking that passing the URL as a parameter might address your use case but understand the differences you pointed out. We are currently looking into feature gaps between the AWS CLI CloudFormation customization and AWS SAM CLI. It doesn't appear that this ability is available in the sam package command either, unless there's some workaround/workflow that I'm not familiar with. Regardless we can leave this feature request open for tracking and others can 👍 and share their use case here if they are also interested in this.