New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use AWSCLI instead of s3cmd for publishing to AWS S3 bucket #1626

Closed
vlcinsky opened this Issue Feb 12, 2015 · 7 comments

Comments

Projects
None yet
5 participants
@vlcinsky

vlcinsky commented Feb 12, 2015

Current make command (see tools/templates/Makefile.in) file is for publishing to AWS S3 bucket using command s3cmd

Nowadays, there is much more widely used tool called AWSCLI which is able doing the same job very well.

My current Makefile is modified as follows:

s3_upload: publish
    #s3cmd sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl-public --delete-removed --guess-    mime-type
    aws s3 --profile ttr sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl public-read --delete
@justinmayer

This comment has been minimized.

Show comment
Hide comment
@justinmayer

justinmayer Mar 7, 2015

Member

Hi Jan. Would you like to submit a pull request implementing the change you are suggesting?

Member

justinmayer commented Mar 7, 2015

Hi Jan. Would you like to submit a pull request implementing the change you are suggesting?

@syndbg

This comment has been minimized.

Show comment
Hide comment
@syndbg

syndbg Mar 15, 2015

Contributor

I've got mixed feelings about this.

While aws-cli is modern and definitely the standard approach when using S3, it introduces a lot of new dependencies.

If there are ideas to add more s3 commands, I see a good reason to use aws-cli, otherwise is it really worth it for just 1 command?

Contributor

syndbg commented Mar 15, 2015

I've got mixed feelings about this.

While aws-cli is modern and definitely the standard approach when using S3, it introduces a lot of new dependencies.

If there are ideas to add more s3 commands, I see a good reason to use aws-cli, otherwise is it really worth it for just 1 command?

@cossatot

This comment has been minimized.

Show comment
Hide comment
@cossatot

cossatot Jul 22, 2016

I am reviving a blog neglected for a few years. In the interim, I've abandoned Python 2, and s3cmd doesn't support Python 3 (and seems to have no plan to). This seems like a reasonable reason to change the command used to upload to S3; it'd be unfortunate for some amount of Pelican to be tied to Python 2 because of one ancillary dependency.

cossatot commented Jul 22, 2016

I am reviving a blog neglected for a few years. In the interim, I've abandoned Python 2, and s3cmd doesn't support Python 3 (and seems to have no plan to). This seems like a reasonable reason to change the command used to upload to S3; it'd be unfortunate for some amount of Pelican to be tied to Python 2 because of one ancillary dependency.

@sevagh

This comment has been minimized.

Show comment
Hide comment
@sevagh

sevagh Aug 20, 2016

Similar experience to @cossatot - s3cmd in my case couldn't predict the correct MIME type (even with magic disabled). With the aws cli I use:

aws s3 sync --delete output/ s3://{BUCKET_URL}

It works just fine, correct MIME type and all.

sevagh commented Aug 20, 2016

Similar experience to @cossatot - s3cmd in my case couldn't predict the correct MIME type (even with magic disabled). With the aws cli I use:

aws s3 sync --delete output/ s3://{BUCKET_URL}

It works just fine, correct MIME type and all.

@justinmayer

This comment has been minimized.

Show comment
Hide comment
@justinmayer

justinmayer Sep 11, 2016

Member

Pull requests are welcome. Anyone want to implement this change?

Member

justinmayer commented Sep 11, 2016

Pull requests are welcome. Anyone want to implement this change?

@sevagh

This comment has been minimized.

Show comment
Hide comment
@sevagh

sevagh Sep 11, 2016

Just thinking out loud here, but what if the Makefile was reduced to only the commands which generate and serve $(OUTPUTDIR) - i.e. remove the gh-pages, s3, dropbox, all those vendor-specific hosting rules.

Does it make Pelican too difficult to use if it went like "OK, static files generated, now you take $(OUTPUTDIR) and do what you need to do to host it wherever you want"?

edit
Just to explain my reasoning a bit already in this thread we have several viewpoints on whether or not aws cli is better than s3cmd, and various different options used for the awscli s3 sync command. Instead of coming to a consensus we can just outsource the s3 task to the end user and allow them to do it however they want.

sevagh commented Sep 11, 2016

Just thinking out loud here, but what if the Makefile was reduced to only the commands which generate and serve $(OUTPUTDIR) - i.e. remove the gh-pages, s3, dropbox, all those vendor-specific hosting rules.

Does it make Pelican too difficult to use if it went like "OK, static files generated, now you take $(OUTPUTDIR) and do what you need to do to host it wherever you want"?

edit
Just to explain my reasoning a bit already in this thread we have several viewpoints on whether or not aws cli is better than s3cmd, and various different options used for the awscli s3 sync command. Instead of coming to a consensus we can just outsource the s3 task to the end user and allow them to do it however they want.

@vlcinsky

This comment has been minimized.

Show comment
Hide comment
@vlcinsky

vlcinsky Sep 11, 2016

Hi all
Still watching but too busy to find time for proper reaction (a pull request).

Meantime I have learned:

  • boto (and s3cmd) are now really obsolete - see reactions to issue boto3 is not a replacement of boto (yet?)
  • awscli is actively mantained by Amazon and we shall expect it to continue it this way
  • I use doit as great replacement of make to be less dependent on non-python tools.

vlcinsky commented Sep 11, 2016

Hi all
Still watching but too busy to find time for proper reaction (a pull request).

Meantime I have learned:

  • boto (and s3cmd) are now really obsolete - see reactions to issue boto3 is not a replacement of boto (yet?)
  • awscli is actively mantained by Amazon and we shall expect it to continue it this way
  • I use doit as great replacement of make to be less dependent on non-python tools.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment