Support `--s3-endpoint-url` argument to `cloudformation package` #3309

baronswindle · 2018-05-05T05:34:55Z

Most other S3 commands accept an --endpoint-url argument. cloudformation package uses S3 under the hood, so it made sense to me that we could allow users to configure the S3 endpoint.

Additionally, it makes it somewhat easier to use to launch CF stacks for local development/testing. See localstack/localstack#632.

codecov-io · 2018-05-05T05:48:01Z

Codecov Report

Merging #3309 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3309   +/-   ##
========================================
  Coverage    93.96%   93.96%           
========================================
  Files          188      188           
  Lines        14525    14525           
========================================
  Hits         13649    13649           
  Misses         876      876

Impacted Files	Coverage Δ
awscli/paramfile.py	`100% <ø> (ø)`	⬆️
awscli/customizations/cloudformation/package.py	`80.85% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da79486...4085282. Read the comment docs.

rafenden · 2019-01-11T01:09:13Z

@baronswindle great work! However, there is some work required to support deploy.py.

miron4dev · 2019-12-03T12:08:15Z

Any news regarding this issue?

codygman · 2020-01-09T14:54:42Z

@miron4dev With renewed interest are there still plans to work on this or is localstack going to be deprecating aws cli v1 anytime soon?

whummer · 2020-02-29T14:05:04Z

Would love to see this feature added to aws cli v1. Can you please elaborate which changes are still required @rafenden @codygman ? (Apart from fixing the merge conflicts). I'd be happy to contribute, if needed, unless @baronswindle can update the PR to push this over the line..? Thanks

wh1tecat-nya · 2020-06-01T14:55:28Z

I just has this problem. is there any update?

codecov-commenter · 2020-06-03T22:41:08Z

Codecov Report

Merging #3309 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3309   +/-   ##
========================================
  Coverage    93.98%   93.98%           
========================================
  Files          188      188           
  Lines        14560    14560           
========================================
  Hits         13684    13684           
  Misses         876      876

Impacted Files	Coverage Δ
awscli/customizations/cloudformation/deploy.py	`100.00% <ø> (ø)`
awscli/customizations/cloudformation/package.py	`80.85% <ø> (ø)`
awscli/paramfile.py	`100.00% <ø> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6441511...d0bf26a. Read the comment docs.

whummer · 2020-06-04T00:16:08Z

awscli/customizations/cloudformation/deploy.py

@@ -276,7 +286,8 @@ def _run_main(self, parsed_args, parsed_globals):
                "s3",
                config=Config(signature_version='s3v4'),
                region_name=parsed_globals.region,
-                verify=parsed_globals.verify_ssl)
+                verify=parsed_globals.verify_ssl


@baronswindle Looks like we're missing a comma at the end of this line. Seems to be breaking the builds currently:

ERROR: Failure: SyntaxError (invalid syntax (deploy.py, line 290)) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/travis/build/aws/aws-cli/tests/unit/customizations/cloudformation/test_deploy.py", line 20, in <module> from awscli.customizations.cloudformation.deploy import DeployCommand File "/home/travis/build/aws/aws-cli/awscli/customizations/cloudformation/__init__.py", line 14, in <module> from awscli.customizations.cloudformation.deploy import DeployCommand File "/home/travis/build/aws/aws-cli/awscli/customizations/cloudformation/deploy.py", line 290 endpoint_url=parsed_args.s3_endpoint_url) ^ SyntaxError: invalid syntax

Thanks!

@whummer My apologies - that was careless. Just pushed up the fix.

whummer · 2020-06-04T07:36:56Z

Builds are green now, thanks @baronswindle . Are we good to get this merged from your side @rafenden @kyleknap @jamesls ? Please let us know if anything is missing - would be great to get this feature in. Thanks!

whummer · 2020-06-12T09:31:02Z

Awesome, thanks for approving the PR @eitens-mak ! Do you know what is the further process for getting this merged? Could one of the project maintainers please help us out here? Many thanks, really appreciated!!

whummer · 2020-06-19T11:09:41Z

Trying to bump this PR one more time - apologies for being persistent, but it would be great to get an official statement from the maintainers what is still required to get this merged and released.

@eitens-mak @stealthycoin @kyleknap @nateprewitt Can you please help out here? Many thanks in advance.

eitens-mak · 2020-06-19T15:46:44Z

@whummer Hey, I'm not a maintainer. Just a random internet person interested in merging this. I'm surprised Github let me approve it, but I was mostly hoping to get some traction because I too want this.

whummer · 2020-07-05T15:08:42Z

@jamesls @kyleknap @bisdavid @JordonPhillips @stealthycoin Can we please get some help or official statement for this PR?

(Pinging some of the top contributors, as I wasn't able to figure out the list of official maintainers of this repository - apologies for the broadcast.)

Would be great if we could get an official 👍 / 👎 on this pull request - are there any concerns about this new feature, anything still missing (as per the contribution guidelines)?

Many thanks for your help.

whummer · 2020-07-20T15:35:30Z

@baronswindle @eitens-mak @wh1tecat-nya @codygman @miron4dev @rafenden We took the liberty to incorporate this change directly into the awslocal command line (see here). If and when this PR gets merged, we can remove our patch again and use the official version contributed here.

You can try installing the latest version of the awslocal command line:

pip install --upgrade --no-cache 'awscli-local>=0.7'

The --s3-endpoint-url parameter will then become available for the package and deploy commands, for instance:

awslocal cloudformation package --template-file template.yml --s3-endpoint-url http://localhost:4566 --s3-bucket test --force-upload
Uploading to 76cdb2bad9582d23c1f6f4d868218d6c  22 / 22.0  (100.00%)
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs8.10
      CodeUri: s3://test/76cdb2bad9582d23c1f6f4d868218d6c

Hope that helps - any feedback welcome. Thanks for your help.

paul-uz · 2021-03-26T12:20:37Z

Can this be merged yet? I desperatley need to be able to deploy our cloudformation stack on Localstack and I need to use aws cloudformation package which is broken.

hammond756 · 2021-04-06T12:54:28Z

I'm also really hoping this can be merged soon.

kevin-woodward · 2021-08-11T19:02:55Z

Any progress on this?

paul-uz · 2021-09-01T08:54:49Z

3 years laters.... Can we get this merged please?

whummer

Changes LGTM (FWIW, as I'm not an official maintainer of this repo) - we've been using this approach successfully in a fork of this repo providing the awslocal command line.

Would also love to see this getting merged, so we don't have to depend on maintaining forks for customizing the S3 CloudFormation endpoint URL. Thanks in advance!

paul-uz · 2021-09-03T15:55:51Z

@kyleknap are you authorized to merge?

jonasdebeukelaer · 2022-04-22T13:42:57Z

bump. Any communication from aws team would be appreciated 👌

kdaily · 2022-04-22T16:33:16Z

Hi everyone,

Thank you for your patience, and I apologize that it's taken so long to get this reviewed. We're making some changes to our community contribution process to provide more transparency and make community contributions successful.

This PR linked should be reviewed within the next couple weeks as we work through all open pull requests to see where the fall in our newly proposed contribution process.

You can read more about the changes to the contribution process here.

stealthycoin · 2022-04-29T21:01:08Z

tests/unit/customizations/cloudformation/test_deploy.py

@@ -60,6 +60,7 @@ def setUp(self):
                                    fail_on_empty_changeset=True,
                                    s3_bucket=None,
                                    s3_prefix="some prefix",
+                                    s3_endpoint_url="http://localhost",


I would like to see tests explicit tests for this behavior.

At the least there needs to be :

Test where argument is not supplied showing that the old endpoint behavior used.

If we do supply an override argument, that is correctly passed through and used in the request.

tim-finnigan · 2023-04-24T21:23:26Z

Putting this PR in draft status as we haven't heard back since last year regarding the requested changes.

tim-finnigan · 2023-05-01T21:12:50Z

Hi @baronswindle just wanted to check in on this PR — is adding the tests requested here something that you are able to address? #3309 (comment)

tim-finnigan · 2023-05-08T22:13:25Z

Closing this PR as we have not heard back regarding the requested changes.

tim-finnigan · 2023-10-17T16:16:59Z

Just wanted to follow up and share this update on the linked issue:

Hi all - thanks again for your patience. Per this recent announcement you can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

To start using this feature, install the AWS CLI >=1.29.0 or >=2.13.0.

To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

baronswindle force-pushed the feature/cf-package-s3-endpoint branch from 434a276 to 7c532ba Compare May 5, 2018 05:40

miron4dev mentioned this pull request Dec 3, 2019

Support --s3-endpoint-url argument to cloudformation package #4727

Closed

baronswindle force-pushed the feature/cf-package-s3-endpoint branch from 7c532ba to 4085282 Compare March 6, 2020 21:51

baronswindle force-pushed the feature/cf-package-s3-endpoint branch from 4085282 to e8693eb Compare June 3, 2020 22:41

whummer reviewed Jun 4, 2020

View reviewed changes

add s3-endpoint-url argument to cloudformation package and deploy

d0bf26a

baronswindle force-pushed the feature/cf-package-s3-endpoint branch from e8693eb to d0bf26a Compare June 4, 2020 00:51

eitens-mak approved these changes Jun 9, 2020

View reviewed changes

whummer mentioned this pull request Aug 29, 2021

aws cloudformation package should put artifacts in a local bucket localstack/localstack#1783

Closed

whummer approved these changes Sep 3, 2021

View reviewed changes

kdaily added cloudformation package-deploy customization Issues related to CLI customizations (located in /awscli/customizations) labels Nov 17, 2021

kdaily added the needs-review This issue or pull request needs review from a core team member. label Apr 22, 2022

stealthycoin suggested changes Apr 29, 2022

View reviewed changes

kdaily added ready-for-review community response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-review This issue or pull request needs review from a core team member. labels May 10, 2022

tim-finnigan marked this pull request as draft April 24, 2023 21:23

tim-finnigan removed the ready-for-review label Apr 24, 2023

tim-finnigan closed this May 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support `--s3-endpoint-url` argument to `cloudformation package` #3309

Support `--s3-endpoint-url` argument to `cloudformation package` #3309

baronswindle commented May 5, 2018

codecov-io commented May 5, 2018 •

edited

rafenden commented Jan 11, 2019 •

edited

miron4dev commented Dec 3, 2019

codygman commented Jan 9, 2020

whummer commented Feb 29, 2020

wh1tecat-nya commented Jun 1, 2020

codecov-commenter commented Jun 3, 2020 •

edited

whummer Jun 4, 2020 •

edited

baronswindle Jun 4, 2020

whummer commented Jun 4, 2020

whummer commented Jun 12, 2020 •

edited

whummer commented Jun 19, 2020 •

edited

eitens-mak commented Jun 19, 2020

whummer commented Jul 5, 2020 •

edited

whummer commented Jul 20, 2020

paul-uz commented Mar 26, 2021

hammond756 commented Apr 6, 2021

kevin-woodward commented Aug 11, 2021

paul-uz commented Sep 1, 2021

whummer left a comment

paul-uz commented Sep 3, 2021

jonasdebeukelaer commented Apr 22, 2022

kdaily commented Apr 22, 2022

stealthycoin Apr 29, 2022

tim-finnigan commented Apr 24, 2023

tim-finnigan commented May 1, 2023

tim-finnigan commented May 8, 2023

tim-finnigan commented Oct 17, 2023

Support --s3-endpoint-url argument to cloudformation package #3309

Support --s3-endpoint-url argument to cloudformation package #3309

Conversation

baronswindle commented May 5, 2018

codecov-io commented May 5, 2018 • edited

Codecov Report

rafenden commented Jan 11, 2019 • edited

miron4dev commented Dec 3, 2019

codygman commented Jan 9, 2020

whummer commented Feb 29, 2020

wh1tecat-nya commented Jun 1, 2020

codecov-commenter commented Jun 3, 2020 • edited

Codecov Report

whummer Jun 4, 2020 • edited

Choose a reason for hiding this comment

baronswindle Jun 4, 2020

Choose a reason for hiding this comment

whummer commented Jun 4, 2020

whummer commented Jun 12, 2020 • edited

whummer commented Jun 19, 2020 • edited

eitens-mak commented Jun 19, 2020

whummer commented Jul 5, 2020 • edited

whummer commented Jul 20, 2020

paul-uz commented Mar 26, 2021

hammond756 commented Apr 6, 2021

kevin-woodward commented Aug 11, 2021

paul-uz commented Sep 1, 2021

whummer left a comment

Choose a reason for hiding this comment

paul-uz commented Sep 3, 2021

jonasdebeukelaer commented Apr 22, 2022

kdaily commented Apr 22, 2022

stealthycoin Apr 29, 2022

Choose a reason for hiding this comment

tim-finnigan commented Apr 24, 2023

tim-finnigan commented May 1, 2023

tim-finnigan commented May 8, 2023

tim-finnigan commented Oct 17, 2023

Support `--s3-endpoint-url` argument to `cloudformation package` #3309

Support `--s3-endpoint-url` argument to `cloudformation package` #3309

codecov-io commented May 5, 2018 •

edited

rafenden commented Jan 11, 2019 •

edited

codecov-commenter commented Jun 3, 2020 •

edited

whummer Jun 4, 2020 •

edited

whummer commented Jun 12, 2020 •

edited

whummer commented Jun 19, 2020 •

edited

whummer commented Jul 5, 2020 •

edited