New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws kms decrypt InvalidCiphertextException error #1043

Closed
mtougeron opened this Issue Dec 5, 2014 · 21 comments

Comments

Projects
None yet
@mtougeron
Copy link

mtougeron commented Dec 5, 2014

I upgraded to version 1.6.6 of the awscli today and aws kms decrypt started failing on decryption. It works in 1.6.5.

$> aws --version
aws-cli/1.6.6 Python/2.7.6 Darwin/13.4.0

$> python -V
Python 2.7.6

$> aws kms encrypt --key-id REDACTED --plaintext foo 
{
    "KeyId": "arn:aws:kms:us-east-1:REDACTED:key/REDACTED", 
    "CiphertextBlob": "CiDdiD7jljnCzXlfZUp27Y4LDY+QJa2Zqcw/7+ihfBDo7hKKAQEBAgB43Yg+45Y5ws15X2VKdu2OCw2PkCWtmanMP+/ooXwQ6O4AAABhMF8GCSqGSIb3DQEHBqBSMFACAQAwSwYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwVt2CzbGDR2nUwszQCARCAHgdJ4aHQ4i7TMzBN6XlKcC73oilECgep+basamtnXQ=="
}

$> aws kms decrypt --ciphertext-blob CiDdiD7jljnCzXlfZUp27Y4LDY+QJa2Zqcw/7+ihfBDo7hKKAQEBAgB43Yg+45Y5ws15X2VKdu2OCw2PkCWtmanMP+/ooXwQ6O4AAABhMF8GCSqGSIb3DQEHBqBSMFACAQAwSwYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAwVt2CzbGDR2nUwszQCARCAHgdJ4aHQ4i7TMzBN6XlKcC73oilECgep+basamtnXQ==

A client error (InvalidCiphertextException) occurred when calling the Decrypt operation: None

$> pip freeze                                       
Babel==1.3
Fabric==1.10.0
Jinja2==2.7.3
MarkupSafe==0.23
Pillow==2.6.1
PyChef==0.2.3
PyYAML==3.11
Pygments==2.0.1
Sphinx==1.2.3
argparse==1.2.1
astroid==1.2.1
awscli==1.6.6
bcdoc==0.12.2
binaryornot==0.3.0
boto==2.34.0
botocore==0.77.0
box.py==1.2.8
cffi==0.8.6
cliff==1.8.0
cmd2==0.6.7
colorama==0.2.5
cookiecutter==0.8.0
coverage==3.7.1
cryptography==0.6.1
decorator==3.4.0
docutils==0.12
dogapi==1.8.5
ecdsa==0.11
futures==2.2.0
httplib2==0.9
httpretty==0.8.3
iso8601==0.1.10
jmespath==0.5.0
jsonpatch==1.9
jsonpointer==1.5
jsonschema==2.4.0
kazoo==2.0
keyring==4.0
logilab-common==0.63.0
lxml==3.4.0
mock==1.0.1
mockito==0.5.2
netaddr==0.7.12
nose==1.3.4
numpy==1.9.1
oath==1.2
oslo.config==1.4.0
oslo.i18n==1.0.0
oslo.serialization==1.0.0
oslo.utils==1.0.0
paramiko==1.15.1
pbr==0.10.0
prettytable==0.7.2
pyOpenSSL==0.14
pyasn1==0.1.7
pycparser==2.10
pycrypto==2.6.1
pylint==1.3.1
pyparsing==2.0.3
python-ceilometerclient==1.0.12
python-cinderclient==1.1.1
python-dateutil==2.3
python-glanceclient==0.14.2
python-heatclient==0.2.12
python-keystoneclient==0.11.2
python-neutronclient==2.3.9
python-novaclient==2.20.0
python-openstackclient==0.4.1
python-swiftclient==2.3.1
python-troveclient==1.0.7
pytz==2014.9
qrcode==5.1
requests==2.4.3
rsa==3.1.2
scipy==0.14.0
simplejson==3.6.5
six==1.8.0
stevedore==1.1.0
urllib3==1.9.1
virtualenv==1.11.6
warlock==1.1.0
wsgiref==0.1.2

@jamesls

This comment has been minimized.

Copy link
Member

jamesls commented Dec 6, 2014

Just a little background info on what's going on:

In 1.6.6, we fixed a regression in which we were not base64 encoding "blob" types that we had previously been encoding. As a result, you now need to specify the raw binary bytes for any parameter marked as a "blob" type, and internally we will automatically base64 encode it for you. This means that to round trip key encryption you'll need to decode the base64 first:

$ aws kms encrypt --key-id <key-id> --plaintext "abcd" --query CiphertextBlob --output text | base64 -D > /tmp/encrypted-file
$ echo "Decrypted: $(aws kms decrypt --ciphertext-blob fileb:///tmp/encrypted-file --query Plaintext --output text | base64 -D)"
Decrypted: abcd

In this example, I'm also using the fileb:// prefix because it's a file with binary content.

@jamesls

This comment has been minimized.

Copy link
Member

jamesls commented Dec 8, 2014

I believe the behavior should remain as is. Given that the behavior in this initial issue relied on a regression in 1.6.5 that was fixed in 1.6.6, we really can't change the pre-existing behavior, as that would be a breaking change for customers. The example snippet above is the expected way to handle binary input/output in the AWS CLI, and except for this recent regression, is the way this behavior has always been.

@jamesls jamesls closed this Dec 8, 2014

@mtougeron

This comment has been minimized.

Copy link

mtougeron commented Dec 8, 2014

Thanks for the update!

@mtougeron

This comment has been minimized.

Copy link

mtougeron commented Dec 9, 2014

Actually, now I'm getting an error similar to issue #1001

$> aws kms encrypt --key-id $KMS_KEY_ID --plaintext "abcd" --query CiphertextBlob --output text | base64 -D > /tmp/encrypted-file

$> echo "Decrypted: $(aws kms decrypt --ciphertext-blob fileb:///tmp/encrypted-file --query Plaintext --output text | base64 -D)"

'ascii' codec can't decode byte 0xdd in position 2: ordinal not in range(128)
Decrypted: 
@jamesls

This comment has been minimized.

Copy link
Member

jamesls commented Dec 10, 2014

@mtougeron What version of the AWS CLI are you using? I just tried again on the latest version of the AWS CLI (1.6.8) and I'm not seeing this issue:

~ $ aws kms encrypt --key-id $AWS_KEY_ID --plaintext "abcd" --query CiphertextBlob --output text | base64 -D > /tmp/encrypted-file
~ $ hexdump -C /tmp/encrypted-file
00000000  0a 20 e1 68 92 dc 42 40  fe 07 80 ca f6 54 1c 68  |. .h..B@.....T.h|
00000010  e2 45 80 bb c3 e0 2a 2f  91 50 7c ac c3 02 9b c9  |.E....*/.P|.....|
00000020  a8 b3 12 8b 01 01 01 02  00 78 e1 68 92 dc 42 40  |.........x.h..B@|
00000030  fe 07 80 ca f6 54 1c 68  e2 45 80 bb c3 e0 2a 2f  |.....T.h.E....*/|
00000040  91 50 7c ac c3 02 9b c9  a8 b3 00 00 00 62 30 60  |.P|..........b0`|
00000050  06 09 2a 86 48 86 f7 0d  01 07 06 a0 53 30 51 02  |..*.H.......S0Q.|
00000060  01 00 30 4c 06 09 2a 86  48 86 f7 0d 01 07 01 30  |..0L..*.H......0|
00000070  1e 06 09 60 86 48 01 65  03 04 01 2e 30 11 04 0c  |...`.H.e....0...|
00000080  41 de f2 2a a6 c5 38 ef  8a 52 54 92 02 01 10 80  |A..*..8..RT.....|
00000090  1f 2e 01 90 65 7a 21 8c  dd 05 e4 4d 09 64 85 c4  |....ez!....M.d..|
000000a0  33 e3 3d e9 ce 33 6b e9  00 93 ec e5 54 33 8b 3b  |3.=..3k.....T3.;|
000000b0
~ $ echo "Decrypted: $(aws kms decrypt --ciphertext-blob fileb:///tmp/encrypted-file --query Plaintext --output text | base64 -D)"
Decrypted: abcd
~ $ aws --version
aws-cli/1.6.8 Python/2.7.7 Darwin/13.4.0
@mtougeron

This comment has been minimized.

Copy link

mtougeron commented Dec 10, 2014

@jamesls it works now in 1.6.8. thanks

@thegranddesign

This comment has been minimized.

Copy link

thegranddesign commented Apr 29, 2016

@jamesls personally I think this is incredibly bad UI. Not being able to pass the same ciphertext into decrypt that you got from encrypt? If you can't change it because you don't want to break existing customers, at least give them a better error message.

A client error (InvalidCiphertextException) occurred when calling the Decrypt operation: None

Doesn't even remotely tell me what to do to fix the problem. You could detect that the cipher is probably Base64 and output a better error and point them to a URL or something.

@thegranddesign

This comment has been minimized.

Copy link

thegranddesign commented Apr 29, 2016

Or heck why not pass an additional flag to decrypt to say "decode this from base64 first"?

@thegranddesign

This comment has been minimized.

Copy link

thegranddesign commented Apr 29, 2016

For those who come to this later, in order to do this without saving the ciphertext to a file, you can do:

aws kms decrypt --ciphertext-blob fileb://<(echo 'ciphertext' | base64 -d)

Note: as pointed out by @hauntingEcho below, the <(cmd) is not POSIX-compliant so if you're using sh, it won't work, but bash and zsh work fine.

@msabramo

This comment has been minimized.

Copy link

msabramo commented Apr 9, 2017

Re: @thegranddesign's comment, on OS X at least I had to use a capital 'D' for base64. E.g.:

$ echo "Decrypted: $(aws kms decrypt --ciphertext-blob fileb://<(echo $ENCRYPTED_DATA | base64 -D) --query Plaintext --output text | base64 -D)"

Decrypted: Hello world!

(Not sure why I get a newline before the Decrypted line, but not a big deal).

@ojitha

This comment has been minimized.

Copy link

ojitha commented May 9, 2017

This worked for me

#encrypt the password: TestReadWrite text in the test.txt file
aws kms encrypt --key-id cfc7acf7-4f20-49c3-aa11-8be4cdc3291d --plaintext fileb://test.txt --output text | base64 --decode > out.txt

#decrypt the password: TestReadWrite
aws kms decrypt  --ciphertext-blob fileb://out.txt --output text --query Plaintext | base64 --decode
@webjay

This comment has been minimized.

Copy link

webjay commented Nov 20, 2017

I had a similar issue, but Node helped:

'use strict';

const KMS = require('aws-sdk').KMS;
const fs = require('fs');

const kms = new KMS({
  apiVersion: '2014-11-01',
  // region: 'eu-west-1'
});

function encrypt(params) {
  return kms.encrypt(params).promise();
}

const arn = 'arn:aws:kms:xxx';

encrypt({
  KeyId: arn,
  Plaintext: fs.readFileSync('/path/to/key.pem')
})
.then(data => fs.writeFileSync('/path/to/key.json', JSON.stringify(data)));
@hauntingEcho

This comment has been minimized.

Copy link

hauntingEcho commented Jan 18, 2018

another note on @thegranddesign 's solution - <(cmd) is a bashism, not POSIX compliant, so you'll need to use another solution if bash isn't an option.

@sgendler-stem

This comment has been minimized.

Copy link

sgendler-stem commented Sep 6, 2018

This is a mindbogglingly dumb bug that really ought to still be open until it is actually fixed. It is a giant anti-pattern to build a tool that performs a symmetric operation that uses one format for output and a completely different (and incompatible) format for input. A symmetric tool that cannot consume its own output as input is BROKEN. I don't care if it is actually working as intended, it is still broken. The tool should either generate raw binary output by default if it is dependent on raw binary input, or else it damn well ought to be able to consume base64 encoded input THAT IT GENERATED without specifying any explicit base64 decode steps or special params that weren't also required in order to generate that output (or an anti-param if it doesn't use the same exact param). It is beyond absurd that I can use aws kms encrypt to generate base64 encoded output of a base64 encoded encryption payload without specifying EITHER of those base64 encodings, but then I have to explicitly base64 decode both the encrypted data AND the decrypted data in order to get it back to the form I provided in the first place. The fact that doing so also requires the use of command line params that aren't even documented at the level of the encrypt and decrypt subcommands makes it that much more difficult to figure out. Users are all but guaranteed to waste hours trying to figure out how to do the single most obvious thing someone might want to do with an encryption/decryption client - encrypt and then decrypt a string specified in a command line param - and entirely because the tool has a broken operating mode where outputs aren't compatible with inputs despite the fact that the two sub-commands are supposed to be symmetric. Closing this without fixing it states that you have absolutely no concern for usability or developer efficiency.

@hauntingEcho

This comment has been minimized.

Copy link

hauntingEcho commented Sep 6, 2018

another path to resolve this (as opposed to new flags, as @thegranddesign mentioned years ago) could be that / is not a valid base64 character, and there is no valid way to specify binary data in json. Therefore:

  • if the ciphertext-blob parameter is valid base-64, pass it as base 64
  • if the ciphertext-blob parameter contains /, parse as a path. fileb:// doing its current behavior, file:// could be b64-encoded.

Agreed with @sgendler-stem though - this ticket is linked in every project I've worked on which uses KMS, because it's always a problem for someone trying to get on board.

@RLThomaz

This comment has been minimized.

Copy link

RLThomaz commented Oct 2, 2018

This is a mindbogglingly dumb bug that really ought to still be open until it is actually fixed. It is a giant anti-pattern to build a tool that performs a symmetric operation that uses one format for output and a completely different (and incompatible) format for input. A symmetric tool that cannot consume its own output as input is BROKEN. I don't care if it is actually working as intended, it is still broken. The tool should either generate raw binary output by default if it is dependent on raw binary input, or else it damn well ought to be able to consume base64 encoded input THAT IT GENERATED without specifying any explicit base64 decode steps or special params that weren't also required in order to generate that output (or an anti-param if it doesn't use the same exact param). It is beyond absurd that I can use aws kms encrypt to generate base64 encoded output of a base64 encoded encryption payload without specifying EITHER of those base64 encodings, but then I have to explicitly base64 decode both the encrypted data AND the decrypted data in order to get it back to the form I provided in the first place. The fact that doing so also requires the use of command line params that aren't even documented at the level of the encrypt and decrypt subcommands makes it that much more difficult to figure out. Users are all but guaranteed to waste hours trying to figure out how to do the single most obvious thing someone might want to do with an encryption/decryption client - encrypt and then decrypt a string specified in a command line param - and entirely because the tool has a broken operating mode where outputs aren't compatible with inputs despite the fact that the two sub-commands are supposed to be symmetric. Closing this without fixing it states that you have absolutely no concern for usability or developer efficiency.

I have to completely agree with @sgendler-stem - if the code generates an output it should be, at least, able to take that as input.

However, there is more to that. If the ecryption is done through the SDK - adding a weird arn at the end of the encrypted file, the CLI won't be able to decrypt it, even after removing the non-base64 arn url. This is just broken.

@julianeden

This comment has been minimized.

Copy link

julianeden commented Nov 19, 2018

Agreed again. This has made things unnecessarily frustrating.

@Isnor

This comment has been minimized.

Copy link

Isnor commented Nov 29, 2018

This is actually closed? Why? I can't believe it's been 4 years since this opened and the only feedback seems to be somebody saying "works as intended". No it fucking doesn't; @sgendler-stem is right on target, this is completely broken. Hope the team responsible feels bad about themselves.

@sunbarve

This comment has been minimized.

Copy link

sunbarve commented Dec 4, 2018

Face the same exceptions:
botocore.errorfactory.InvalidCiphertextException: An error occurred (InvalidCiphertextException) when calling the Decrypt operation:

Used:
key = b64decode(key)
response = client.decrypt(
CiphertextBlob=key
)

@orirawlings

This comment has been minimized.

Copy link

orirawlings commented Dec 19, 2018

I think this wouldn't be so frustrating if the output of encrypt, generate-data-key, etc didn't call the field CiphertextBlob. When the input to decrypt is also called --ciphertext-blob, it is thoroughly unintuitive that this ciphertext blob needs to be encoded differently.

If the flag to decrypt had a different name, it would be a hint that it has a different encoding.

@rdkls

This comment has been minimized.

Copy link

rdkls commented Dec 28, 2018

There's a feature proposal for base64-encoded input to decrypt over here #2063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment