role-based authentication fails in new AWS regions, including eu-central-1, ca-central-1 and others.

[tavisma]

I'm having trouble using this script in the eu-central-1 region

I've got a role/user that works with "aws ec2 create-snapshot". I've tested this in eu-central-1, us-west-2 and us-east-1 and it works

If i run ec2-consistent-snapshot in other regions using this user it works
When i switch the region to to Frankfurt with "--region 'eu-central-1'" i get an "ec2-consistent-snapshot: ERROR: create_snapshot: AuthFailure: AWS was not able to validate the provided access credentials at ./ec2-consistent-snapshot line 323.
" Error

[tavisma]

I'm using, Net::Amazon::EC2 is up to date (0.31) on CentOS 7
I've tried system PERL (perl5 5.16.3) and Activeperl (5.22.1)

Here is the command line i'm using:
./ec2-consistent-snapshot --aws-access-key-id 'REDACTED' --aws-secret-access-key 'REDACTED' --freeze-filesystem REDACTED --region 'eu-central-1' --description "Test_snapshot_1" --debug vol-REDACTED

Debug log:
ec2-consistent-snapshot: Using AWS access key: REDACTED
ec2-consistent-snapshot: Using description 'Test_snapshot_1' for all snapshot descriptions
ec2-consistent-snapshot: Tue Aug 2 23:53:52 2016: sync
ec2-consistent-snapshot: Tue Aug 2 23:53:52 2016: fsfreeze -f REDACTED
ec2-consistent-snapshot: Tue Aug 2 23:53:52 2016: create EC2 object
ec2-consistent-snapshot: Endpoint: https://ec2.eu-central-1.amazonaws.com
ec2-consistent-snapshot: volume_id: REDACTED; description: Test_snapshot_1
ec2-consistent-snapshot: Tue Aug 2 23:53:52 2016: aws ec2 create-snapshot vol-REDACTED
ec2-consistent-snapshot: ERROR: create_snapshot: AuthFailure: AWS was not able to validate the provided access credentials at ./ec2-consistent-snapshot line 323.
ec2-consistent-snapshot: Tue Aug 2 23:53:52 2016: fsfreeze -u REDACTED

running this command with the same user/key/secret works properly
aws --region 'eu-central-1' ec2 create-snapshot --volume-id vol-REDACTED

[tavisma]

Spinning up a server instance with an attached iam-role seems to have the same issue in eu-central-1
It works when i use the iam-role in us-west-1

[archeredu-circleci]

Thanks for taking the time to report this. If you (or anybody else) figures out what is causing the difference, please post here.

[tavisma]

have you also seen this behaviour on the eu-central-1 region?

[wfen]

We're using this script on CentOS7 with perl-Net-Amazon-EC2 0.31 successfully within eu-central-1. I believe that perhaps you might need to specify the option "--signature-version 4" in your command.

[tavisma]

No luck, same problem

I'm going to set aside some time next week to see if i can't debug this further

[tavisma]

Update:

Using "--use-iam-role" does NOT work in the eu-central-1 region with the latest version of Net::Amazon::EC2 (0.31)
Add "--signature-version 4" does NOT solve the problem

If i specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY AND add "--signature-version 4" to the command line, it works!

This may be a bug in Net::Amazon::EC2, specific to eu-central-1 region (maybe they're doing something different there as it is a new region)

I'll update this ticket with more information if i manage to find out why IAM doesn't work

[archeredu-circleci]

@tavispaquette Thanks for the additional details.

[m0n5t3r]

ran into this today, seems to affect ec2-expire-snapshots as well; in the end I just implemented the functionality I needed in a bash script with awscli

[effeks]

Is this related to #59 ?

[tavisma]

Not entirely.
Specifying "--signature-version 4" on the command line does resolve the issue unless i also specify "--use-iam-role"

So i have a workaround but i am currently not able to use iam-roles, i have to provision API keys along with the script on each server

[markstos]

New regions don't support "signature version 2", only "signature version 4". I confirmed that authentication fails when combining "signature version 4" with "--use-iam-role" in the new "ca-central-1" region. I'm investigating now.

There appears to be no related bug in ec2-consistent-snapshot, rather it appears that the bug is upstream in Net::Amazon::EC2.

Someone identified the bug in October of 2015, but the issue remains open there:

https://rt.cpan.org/Public/Bug/Display.html?id=107491

[markstos]

Considering that the underlying Net::Amazon::EC2 bug hasn't been fixed for over a year, I don't expect it to be fixed soon. In the meantime, AWS has endorsed PAWS as their unofficial Perl SDK. The Paws API explicitly documents that it support role-based authentication.

The recommend fix is to refactor the code to use the better-maintained PAWS library instead of the apparently-neglected Net::Amazon::EC2 library.

[markstos]

I made a related Pull Request to Paws to set the signature version appropriately for the newer AWS regions.

[markstos]

A challenge with using Paws instead of Net::Amazon::EC2 will be that there is no Ubuntu package for Paws. We could bundle a version of Paws in our own Ubuntu package, but that's also unattractive.

[pplu]

If you guys need help, I'm open to discussing how we can make Paws easier to distribute (currently you get all Paws with all services when you install it via CPAN)

[markstos]

Thanks @pplu I guess one option we have is to quit recommending that ec2-consistent-snapshot be installed exclusively via apt. Instead, we could provide instructions for a CPAN-based installation.

If have to pause my efforts on this now on the moment. I just have server in a problematic region at the moment, and I'm going to quickly setup an alternate backup method so I can return to other priorities at work.

I would enjoy fixing this in the future if I have time, though-- Almost all my work is in JavaScript now, and I appreciate the rare jobs that puts my 10+ years of Perl experience to use!

[markstos]

@pplu Users of ec2-consistent-snapshot won't necessarily be Perl developers. The installation instructions for Paws say to use cpanm and carton to install it, but those aren't installed by default on an OS and non-Perl-devs won't be familiar with them. So, it would be helpful to have instructions on getting those installed. (Or at least linking to the docs for those packages for their own installation instructions).

I found that those projects are already packaged for Ubuntu, so users could use familiar commands to install them:

 sudo apt-get install cpanminus carton

[markstos]

Net::Amazon::EC2 has resolved this issue upstream by making AWS Signature Version 4 the default.

A Pull request which would be welcome which made this project depend on latest Net::Amazon::EC2 version 0.32, after testing against and making any related refactors required to work with the new version.

Ref: https://metacpan.org/changes/distribution/Net-Amazon-EC2

[apptaro]

Hi @markstos thank you for your efforts to maintain this program. So, judging from your recent comments, it looks like you were waiting for Net::Amazon::EC2 0.32 to make AWS Signature Version 4 the default, and then you were waiting for Net::Amazon::EC2 to fix stateReason bug, which is now fixed in 0.33. But, I installed the latest master of ec2-consistent-snapshot and Net::Amazon::EC2 0.33, and the issue is still there.

$ sudo ec2-consistent-snapshot --region ap-northeast-1 --use-iam-role --signature-version=4 --no-freeze-filesystem / vol-0af8e6cbf54b0d41d
ec2-consistent-snapshot: ERROR: create_snapshot: AuthFailure: AWS was not able to validate the provided access credentials at /home/centos/jobs/ec2-consistent-snapshot line 323.
$ sudo ec2-consistent-snapshot --region ap-northeast-1 --use-iam-role --signature-version=2 --no-freeze-filesystem / vol-0af8e6cbf54b0d41d
snap-0d41e2c774eeb3717
$

For the region ap-northeast-1, to use --use-iam-role, I have to use --signature-version=2.
So, I think the root cause is still this: https://rt.cpan.org/Public/Bug/Display.html?id=107491

[apptaro]

I have logged the issue with Net::Amazon::EC2
jadeallenx/net-amazon-ec2#59

[jadeallenx]

I released 0.35 of Net-Amazon-EC2 to CPAN today which I believe fixes this issue.

[markstos]

Thanks everyone. At this point I'm waiting on Ubuntu to package a new version of Net::Amazon::EC2, as mentioned here: #94 (comment)

But perhaps I shouldn't hold my breath for that. The alternative is for @ehammond to publish Net::Amazon::EC2 .35 to his Ubuntu PPA, and then we can update our Ubuntu package for ec2-consistent-snapshot to depend on that newer package of Net::Amazon::EC2.

[ehammond]

I am not excited about the work that would probably be involved in publishing a Perl package to a PPA, and am uncertain if this would cause issues for anybody else, how we would know about those issues, and what would be involved with keeping it up to date as the months and years go by.

@markstos If you are interested in running a PPA, I'd be happy to point everything over that way :-)

Folks at my company have been moving to Paws for Perl AWS stuff. Not sure if that would be an option, if it is available as an Ubuntu package, or if it would solve the problem. https://metacpan.org/pod/Paws

[markstos]

I'm not keen to start maintaining a PPA, either.

I would welcome a refactor to use Paws instead of Net::Amazon::EC2 is the version currently published by Ubuntu would solve the problems with the published versions of Net::Amazon::EC2.

Another option is to switch the distribution method from Ubuntu packages to a CPAN-based approach, in which it would be easy to depend on whichever version of Net::Amazon::EC2 that we liked. Volunteers to explore that approach would be welcome as well.

My open source time is already somewhat booked helping with node-config and I also recently started to help maintain passport-saml for Node.js as well.

[nikolai-derzhak-distillery]

Sorry for commenting on closed ticket. But not sure if it the same issue or not (though I use recent EC2 lib). Every other (like 20%) of tries I get auth error (with both signature 2 and signature 4) in us-west-2:

ec2-consistent-snapshot.bin: Fri May 18 00:26:40 2018: Determining instance id
ec2-consistent-snapshot.bin: Fri May 18 00:26:40 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-2.amazonaws.com
ec2-consistent-snapshot.bin: Fri May 18 00:26:40 2018: Fetching instance description for i-060ba628ed66c0ff9
Amazon EC2 Errors [Request 2b88fdca-4099-4cd9-82c2-be65d887ea37]:
[AuthFailure] AWS was not able to validate the provided access credentials

It can fail with auth error on createSnapshot API call also , when volume discovery API calls went just fine.

Any suggestions ?

[nikolai-derzhak-distillery]

Corresponding comment on Net::Amazon:Ec2 issue: jadeallenx/net-amazon-ec2#59

[markstos]

It's probably the same issue. The new version should be packaged as a "snap":
#97 (comment)

Volunteers welcome.

[nikolai-derzhak-distillery]

@markstos I just curl master branch of the script, do not use package system. And then install recent dependencies.

RUN curl -L http://xrl.us/cpanm > /bin/cpanm && chmod +x /bin/cpanm
RUN cpanm install --notest --no-man-pages --without-recommend --without-suggests \
  MongoDB Any::Moose MongoDB::Admin Net::Amazon::EC2 File::Slurp PAR::Packer

# we patch it so have to pin exact version until PRs will be created and merged
# https://github.com/alestic/ec2-consistent-snapshot/issues/98
# https://github.com/alestic/ec2-consistent-snapshot/issues/72#issuecomment-391557982
#
#                 https://raw.githubusercontent.com/alestic/ec2-consistent-snapshot/master/ec2-consistent-snapshot
RUN curl -s -L -O https://raw.githubusercontent.com/alestic/ec2-consistent-snapshot/ecd9e9febc7cf1cfb6f9e48a49446ec8d1436849/ec2-consistent-snapshot

[markstos]

@nderzhak I have a "snap" package that I could use help testing, especially on 16.04. It packages it's on set of dependencies to avoid problems with getting versions that are too old or too new in manual installations. This is the recommended way to get the newer version of Net::Amazon::EC2 on 16.04.

 snap install ec2-consistent-snapshot --channel=edge --devmode

I built it on Bionic and haven't tested it on 16.04. It's my first Snap package, though. It would be nice to improve the "confinement" setting so we don't have to publish it in "devmode". You are welcome to work on that. However, the permissions it has in devmode are the same as it has in the .deb file.

So, it's no less secure, it's just that Snaps enable the possibility of being more secure by further limiting the filesystem access of your packages.

[markstos]

I ended building a bash alternative that avoids the Perl packaging problems: https://github.com/RideAmigosCorp/ec2-consistent-snapshot.sh

[ehammond]

Thanks for submitting this. Unfortunately, this project is no longer under development in this repo. Anybody is welcome to fork the project and continue development if there is interest.