New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS IAM auth 'Content-Length' header causes validation error when sent as number instead of string or []strings #3763

Closed
DaniGuardiola opened this Issue Jan 8, 2018 · 23 comments

Comments

Projects
None yet
3 participants
@DaniGuardiola

DaniGuardiola commented Jan 8, 2018

Environment:

  • Vault Version: v0.9.1
  • Operating System/Architecture: Ubuntu 16.04 LTS on EC2 instance

Vault Config File:

(dev server)

Expected Behavior:

Vault returns the IAM auth login response, as described in the docs

Actual Behavior:

Vault returned this error:
Error parsing iam_request_headers: header "Content-Length" value "43" has type json.Number, not string or []interface

Steps to Reproduce:

Create request and sign it with aws4 npm module:

Original signed request (aws4)

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { 'X-Vault-AWS-IAM-Server-ID': 'i-99999999999993',
     Host: 'sts.amazonaws.com',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 43,
     'X-Amz-Security-Token': 'FQoaaaaEML//////////wEaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaL7O0gU=',
     'X-Amz-Date': 20210401T171213Z',
     Authorization: 'AWS4-HMAC-SHA256 Credential=AAAAAAAAAAAAAA/20210401/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-vault-aws-iam-server-id, Signature=8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

Formatted for vault

{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'some-role',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHRaaaaaaaaaaaaaaaaaaaaaaavbS8=',
     iam_request_body: 'QWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaQ==',
     iam_request_headers: 'eyJYLAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAafQ==' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }

Important Factoids:

  • I faked the data up there for obvious reasons, but I can guarantee the base64 text contains the correct (stringified) version of the json (with 'Content-Length': 43 - as a number not a string)
  • Running the dev server
  • Vault aws backend initialized
  • Vault AWS role and user are setup and access key and secret are configured in aws auth: the role policy is copied from the aws auth documentation without the `assumeRole' permission (not needed)
  • No header enforcement setup (the header in the request serves no purpose for now, but I don't think it influences this issue)
  • Using the request-promise-native npm module for requests
  • Using the aws4 module for aws request signing. This module adds the Content-Length header before signing the request, but it does so correctly (as a number).
  • Programming this as part of the node-vault module (I'm adding some functionalities), here's the failing code

Please help!
This is blocking and urgent for my company :/ thanks in advance!

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

I removed the additional headers and now vault returns this error:

error making upstream request: error parsing STS response

This is the request object that aws4 returns now (includes aws4 settings that are later discarded when formatting the request to vault):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { 'X-Vault-AWS-IAM-Server-ID': 'i-0222222222223',
     Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=AAAAAAAAAAFA/20222222/eu-west-1/sts/aws4_request, SignedHeaders=host;x-vault-aws-iam-server-id, Signature=eAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAa8e' },
  region: 'eu-west-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

This is the actual code now

DaniGuardiola commented Jan 8, 2018

I removed the additional headers and now vault returns this error:

error making upstream request: error parsing STS response

This is the request object that aws4 returns now (includes aws4 settings that are later discarded when formatting the request to vault):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { 'X-Vault-AWS-IAM-Server-ID': 'i-0222222222223',
     Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=AAAAAAAAAAFA/20222222/eu-west-1/sts/aws4_request, SignedHeaders=host;x-vault-aws-iam-server-id, Signature=eAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAa8e' },
  region: 'eu-west-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

This is the actual code now

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

Tested without the X-Vault-AWS-IAM-Server-ID header with the same result error making upstream request: error parsing STS response

DaniGuardiola commented Jan 8, 2018

Tested without the X-Vault-AWS-IAM-Server-ID header with the same result error making upstream request: error parsing STS response

@DaniGuardiola DaniGuardiola changed the title from AWS IAM auth: Error parsing iam_request_headers: header "Content-Length" value "43" has type json.Number, not string or []interface to AWS IAM auth does not work for some reason Jan 8, 2018

@joelthompson

This comment has been minimized.

Show comment
Hide comment
@joelthompson

joelthompson Jan 8, 2018

Contributor

Hey @DaniGuardiola -- I think there are two issues here.

On the first issue you mentioned, the AWS NodeJS SDK is returning a content-length header as an int, when HTTP headers should either a string or array of strings. This is the same issue mentioned in #2810 (comment) but it's a different error message now due to changes to support a string or array of strings (as opposed to just the latter).

On the second, try setting the region to us-east-1. Some background here, but in short, your request's Authorization header has a credential scope with a region of eu-west-1 but the Host header is set to sts.amazonaws.com which, in my experiments, expects a region of us-east-1. If the Vault admin hasn't overridden the config/client sts_endpoint value, this is what you need to do. And if the Vault admin has, then you need to ensure that the AWS SDK generates the right host header for that region-specific endpoint.

Contributor

joelthompson commented Jan 8, 2018

Hey @DaniGuardiola -- I think there are two issues here.

On the first issue you mentioned, the AWS NodeJS SDK is returning a content-length header as an int, when HTTP headers should either a string or array of strings. This is the same issue mentioned in #2810 (comment) but it's a different error message now due to changes to support a string or array of strings (as opposed to just the latter).

On the second, try setting the region to us-east-1. Some background here, but in short, your request's Authorization header has a credential scope with a region of eu-west-1 but the Host header is set to sts.amazonaws.com which, in my experiments, expects a region of us-east-1. If the Vault admin hasn't overridden the config/client sts_endpoint value, this is what you need to do. And if the Vault admin has, then you need to ensure that the AWS SDK generates the right host header for that region-specific endpoint.

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

Hi @joelthompson! Thank you very much for your response!

I tried changing the region to us-east-1 and I get the same error message :(

This is the new request (aws4 input):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20051111/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=dasddddasdasdasdawdawdasdasdasdasdasdasdasdasdasdasd' },
  region: 'us-east-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

DaniGuardiola commented Jan 8, 2018

Hi @joelthompson! Thank you very much for your response!

I tried changing the region to us-east-1 and I get the same error message :(

This is the new request (aws4 input):

{ service: 'sts',
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20051111/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=dasddddasdasdasdawdawdasdasdasdasdasdasdasdasdasdasd' },
  region: 'us-east-1',
  doNotModifyHeaders: true,
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
@jefferai

This comment has been minimized.

Show comment
Hide comment
@jefferai

jefferai Jan 8, 2018

Member

@DaniGuardiola Any chance you're able to build and test from our repo?

Member

jefferai commented Jan 8, 2018

@DaniGuardiola Any chance you're able to build and test from our repo?

@jefferai

This comment has been minimized.

Show comment
Hide comment
@jefferai

jefferai Jan 8, 2018

Member

Please try the https://github.com/hashicorp/vault/tree/aws-jsonnumber branch and let me know if it fixes the json.Number error with your original data set.

Member

jefferai commented Jan 8, 2018

Please try the https://github.com/hashicorp/vault/tree/aws-jsonnumber branch and let me know if it fixes the json.Number error with your original data set.

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

@jefferai Done

This is the signed request options (ignore non-request stuff, it's aws4's)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: false,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 43,
     'X-Amz-Date': '20180108T220028Z',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=b09aa0e47e9980357ab95970b301d64eadb3f116a2f5ce58c3902133c4284f96' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

And this is the request passed to the request(-promise-native) module

{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQ7IGNoYXJzZXQ9dXRmLTgiLCJDb250ZW50LUxlbmd0aCI6NDMsIlgtQW16LURhdGUiOiIyMDE4MDEwOFQyMjAyMjFaIiwiQXV0aG9yaXphdGlvbiI6IkFXUzQtSE1BQy1TSEEyNTYgQ3JlZGVudGlhbD11bmRlZmluZWQvMjAxODAxMDgvdXMtZWFzdC0xL3N0cy9hd3M0X3JlcXVlc3QsIFNpZ25lZEhlYWRlcnM9Y29udGVudC1sZW5ndGg7Y29udGVudC10eXBlO2hvc3Q7eC1hbXotZGF0ZSwgU2lnbmF0dXJlPTY1OWE5NTk5NmQxNjc0YzIyMmZkZTk3Mjg3OWUxMmRlZDYzNDVjZTAzM2RkYTllZGE4NjM1MTJiY2RmY2NiYWIifQ==' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }

(The data is not really sensitive so I'm not gonna censor it anymore for now)

Now I get this error:
Error parsing iam_request_headers: header "Content-Length" value %!q(float64=43) has type float64, not string or []interface

DaniGuardiola commented Jan 8, 2018

@jefferai Done

This is the signed request options (ignore non-request stuff, it's aws4's)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: false,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 43,
     'X-Amz-Date': '20180108T220028Z',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=b09aa0e47e9980357ab95970b301d64eadb3f116a2f5ce58c3902133c4284f96' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }

And this is the request passed to the request(-promise-native) module

{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJDb250ZW50LVR5cGUiOiJhcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQ7IGNoYXJzZXQ9dXRmLTgiLCJDb250ZW50LUxlbmd0aCI6NDMsIlgtQW16LURhdGUiOiIyMDE4MDEwOFQyMjAyMjFaIiwiQXV0aG9yaXphdGlvbiI6IkFXUzQtSE1BQy1TSEEyNTYgQ3JlZGVudGlhbD11bmRlZmluZWQvMjAxODAxMDgvdXMtZWFzdC0xL3N0cy9hd3M0X3JlcXVlc3QsIFNpZ25lZEhlYWRlcnM9Y29udGVudC1sZW5ndGg7Y29udGVudC10eXBlO2hvc3Q7eC1hbXotZGF0ZSwgU2lnbmF0dXJlPTY1OWE5NTk5NmQxNjc0YzIyMmZkZTk3Mjg3OWUxMmRlZDYzNDVjZTAzM2RkYTllZGE4NjM1MTJiY2RmY2NiYWIifQ==' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }

(The data is not really sensitive so I'm not gonna censor it anymore for now)

Now I get this error:
Error parsing iam_request_headers: header "Content-Length" value %!q(float64=43) has type float64, not string or []interface

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

And when I remove the extra headers I still get the error parsing STS response message

DaniGuardiola commented Jan 8, 2018

And when I remove the extra headers I still get the error parsing STS response message

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

Full example without the Content-Length header

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=c558f10dbf80e34b1e167d99d84e2b2f51144dcf510e73918096d19c52ce9b75' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

DaniGuardiola commented Jan 8, 2018

Full example without the Content-Length header

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=undefined/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=c558f10dbf80e34b1e167d99d84e2b2f51144dcf510e73918096d19c52ce9b75' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

The vault cli also fails:

$ vault write auth/aws/login role=ssh-admin \
	iam_http_request_method=POST \
	iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
	iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
	iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9'
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://vault.vpc/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response

DaniGuardiola commented Jan 8, 2018

The vault cli also fails:

$ vault write auth/aws/login role=ssh-admin \
	iam_http_request_method=POST \
	iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
	iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
	iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPXVuZGVmaW5lZC8yMDE4MDEwOC91cy1lYXN0LTEvc3RzL2F3czRfcmVxdWVzdCwgU2lnbmVkSGVhZGVycz1ob3N0LCBTaWduYXR1cmU9YzU1OGYxMGRiZjgwZTM0YjFlMTY3ZDk5ZDg0ZTJiMmY1MTE0NGRjZjUxMGU3MzkxODA5NmQxOWM1MmNlOWI3NSJ9'
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://vault.vpc/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response
@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

The credentials were not set because of a small bug I just fixed, now it's not undefined but I get the same error. Halp! :P @joelthompson @jefferai

DaniGuardiola commented Jan 8, 2018

The credentials were not set because of a small bug I just fixed, now it's not undefined but I get the same error. Halp! :P @joelthompson @jefferai

@joelthompson

This comment has been minimized.

Show comment
Hide comment
@joelthompson

joelthompson Jan 8, 2018

Contributor

Was just going to comment on that (wasn't sure if undefined was your redaction or not earlier), can you repost the Vault login data?

The error you get should only happen when STS returns a 200 response but the XML is in a format that's not recognized by Vault, which shouldn't ever happen (though the error gets eaten). Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

Also, since you're able to test dev builds, I'll try to push a branch to my fork later tonight that should add more verbose logging messages.

Contributor

joelthompson commented Jan 8, 2018

Was just going to comment on that (wasn't sure if undefined was your redaction or not earlier), can you repost the Vault login data?

The error you get should only happen when STS returns a 200 response but the XML is in a format that's not recognized by Vault, which shouldn't ever happen (though the error gets eaten). Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

Also, since you're able to test dev builds, I'll try to push a branch to my fork later tonight that should add more verbose logging messages.

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

@joelthompson here :)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=ASIAIBLI3BFCJY7GUO4A/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=e7d3a23018c8ba39cc1f415a0308597670ec183bb9b42dbb424027cb393259f9' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFJQkxJM0JGQ0pZN0dVTzRBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1lN2QzYTIzMDE4YzhiYTM5Y2MxZjQxNWEwMzA4NTk3NjcwZWMxODNiYjliNDJkYmI0MjQwMjdjYjM5MzI1OWY5In0=' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)

DaniGuardiola commented Jan 8, 2018

@joelthompson here :)

{ service: 'sts',
  region: 'us-east-1',
  doNotModifyHeaders: true,
  body: 'Action=GetCallerIdentity&Version=2011-06-15',
  headers: 
   { Host: 'sts.amazonaws.com',
     Authorization: 'AWS4-HMAC-SHA256 Credential=ASIAIBLI3BFCJY7GUO4A/20180108/us-east-1/sts/aws4_request, SignedHeaders=host, Signature=e7d3a23018c8ba39cc1f415a0308597670ec183bb9b42dbb424027cb393259f9' },
  method: 'POST',
  hostname: 'sts.amazonaws.com',
  path: '/' }
{ method: 'POST',
  path: '/auth/aws/login',
  json: 
   { role: 'ssh-admin',
     iam_http_request_method: 'POST',
     iam_request_url: 'aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=',
     iam_request_body: 'QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==',
     iam_request_headers: 'eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFJQkxJM0JGQ0pZN0dVTzRBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1lN2QzYTIzMDE4YzhiYTM5Y2MxZjQxNWEwMzA4NTk3NjcwZWMxODNiYjliNDJkYmI0MjQwMjdjYjM5MzI1OWY5In0=' },
  uri: 'http://vault.vpc/v1/auth/aws/login',
  headers: {} }
Error: error making upstream request: error parsing STS response
    at VaultClient._handleVaultResponse (/home/ubuntu/node-vault/src/main.js:223:13)
    at VaultClient._request (/home/ubuntu/node-vault/src/main.js:201:17)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

I don't think so :/

Would it help if I insert some logs somewhere, rebuild and re-run the dev server? Can I extend that error message in the code to add more information? Don't wanna pressure you but I gotta get this done soon so if there's any way I can speed up the process just tell me :)

DaniGuardiola commented Jan 8, 2018

Are you doing anything non-standard on the Vault server, such as going through an HTTP web proxy or setting an alternate sts_endpoint value, that would change how the traffic is going through?

I don't think so :/

Would it help if I insert some logs somewhere, rebuild and re-run the dev server? Can I extend that error message in the code to add more information? Don't wanna pressure you but I gotta get this done soon so if there's any way I can speed up the process just tell me :)

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

Wait, I'm using nginx to publish the vault dev server, I proxy localhost:8200 through the 80 port to expose it to other instances in the VPC. Is that a problem? Every other request works just fine (like help or status).

DaniGuardiola commented Jan 8, 2018

Wait, I'm using nginx to publish the vault dev server, I proxy localhost:8200 through the 80 port to expose it to other instances in the VPC. Is that a problem? Every other request works just fine (like help or status).

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

:(

ubuntu@ip-redacted:~$ export VAULT_ADDR='http://127.0.0.1:8200'
ubuntu@ip-redacted:~$ vault status
Seal Type: shamir
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.1
Cluster Name: vault-cluster-5376bd99
Cluster ID: dcf4b559-0607-5b7a-2993-28caa1bc5011

High-Availability Enabled: false
ubuntu@ip-redacted:~$ vault write auth/aws/login role=ssh-admin \
> iam_http_request_method=POST \
> iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
> iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
> iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFKQ1NRNlg1RUhKTFZRTkJBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1mYjY0YjM2MWQyMDY1YTFkMzhjZjUyYjRhN2ZhYmFiMWZlZDM2ZDEyOTRkOWU4N2Q5OWE5OTU3NGZjMDVkNzc3In0='
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response

DaniGuardiola commented Jan 8, 2018

:(

ubuntu@ip-redacted:~$ export VAULT_ADDR='http://127.0.0.1:8200'
ubuntu@ip-redacted:~$ vault status
Seal Type: shamir
Sealed: false
Key Shares: 1
Key Threshold: 1
Unseal Progress: 0
Unseal Nonce: 
Version: 0.9.1
Cluster Name: vault-cluster-5376bd99
Cluster ID: dcf4b559-0607-5b7a-2993-28caa1bc5011

High-Availability Enabled: false
ubuntu@ip-redacted:~$ vault write auth/aws/login role=ssh-admin \
> iam_http_request_method=POST \
> iam_request_url='aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8=' \
> iam_request_body='QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNQ==' \
> iam_request_headers='eyJIb3N0Ijoic3RzLmFtYXpvbmF3cy5jb20iLCJBdXRob3JpemF0aW9uIjoiQVdTNC1ITUFDLVNIQTI1NiBDcmVkZW50aWFsPUFTSUFKQ1NRNlg1RUhKTFZRTkJBLzIwMTgwMTA4L3VzLWVhc3QtMS9zdHMvYXdzNF9yZXF1ZXN0LCBTaWduZWRIZWFkZXJzPWhvc3QsIFNpZ25hdHVyZT1mYjY0YjM2MWQyMDY1YTFkMzhjZjUyYjRhN2ZhYmFiMWZlZDM2ZDEyOTRkOWU4N2Q5OWE5OTU3NGZjMDVkNzc3In0='
Error writing data to auth/aws/login: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: error parsing STS response
@joelthompson

This comment has been minimized.

Show comment
Hide comment
@joelthompson

joelthompson Jan 8, 2018

Contributor

I'm starting to think the issue is your setting doNotModifyHeaders to true which then causes the Content-Length and Content-Type headers to be missing from iam_request_headers.

First, can you take your original code without doNotModifyHeaders but modify the Content-Length header to be a string before JSON-serializing it?

If that doesn't work, try changing this line:

return nil, fmt.Errorf("error parsing STS response")

To something like:

		return nil, fmt.Errorf("error parsing STS response; original response: %q; error: %q", string(responseBody), err)

and then rebuild and rerun the test; I think that should give more information.

Contributor

joelthompson commented Jan 8, 2018

I'm starting to think the issue is your setting doNotModifyHeaders to true which then causes the Content-Length and Content-Type headers to be missing from iam_request_headers.

First, can you take your original code without doNotModifyHeaders but modify the Content-Length header to be a string before JSON-serializing it?

If that doesn't work, try changing this line:

return nil, fmt.Errorf("error parsing STS response")

To something like:

		return nil, fmt.Errorf("error parsing STS response; original response: %q; error: %q", string(responseBody), err)

and then rebuild and rerun the test; I think that should give more information.

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

@joelthompson I can modify the aws4 version locally to do so, gimme a sec

DaniGuardiola commented Jan 8, 2018

@joelthompson I can modify the aws4 version locally to do so, gimme a sec

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

@joelthompson bingo!!

{ request_id: 'asdfa3af4-as43-dsf3-sdf234-asdf7667sd7f6',
  lease_id: '',
  renewable: false,
  lease_duration: 0,
  data: null,
  wrap_info: null,
  warnings: null,
  auth: 
   { client_token: '1a9ec23d-bcf0-f810-a193-ea6dfb8ec420',
     accessor: '96d9ba25-7aad-dfb1-1ef8-ad2ef4135a7e',
     policies: [ 'default', 'dev', 'prod' ],
     metadata: 
      { account_id: '81972398723',
        auth_type: 'iam',
        canonical_arn: 'arn:aws:iam::98787996897898:role/ssh-admin',
        client_arn: 'arn:aws:sts::192837981273:assumed-role/ssh-admin/i-78327237832782376',
        client_user_id: 'ASGDJKWAGAWDWAD',
        inferred_aws_region: '',
        inferred_entity_id: '',
        inferred_entity_type: '' },
     lease_duration: 2764800,
     renewable: true,
     entity_id: '21hj3ghdsfg-asda-kj2hy3-asdf-8a7s5df5sadf' } }

(data modified for privacy)

DaniGuardiola commented Jan 8, 2018

@joelthompson bingo!!

{ request_id: 'asdfa3af4-as43-dsf3-sdf234-asdf7667sd7f6',
  lease_id: '',
  renewable: false,
  lease_duration: 0,
  data: null,
  wrap_info: null,
  warnings: null,
  auth: 
   { client_token: '1a9ec23d-bcf0-f810-a193-ea6dfb8ec420',
     accessor: '96d9ba25-7aad-dfb1-1ef8-ad2ef4135a7e',
     policies: [ 'default', 'dev', 'prod' ],
     metadata: 
      { account_id: '81972398723',
        auth_type: 'iam',
        canonical_arn: 'arn:aws:iam::98787996897898:role/ssh-admin',
        client_arn: 'arn:aws:sts::192837981273:assumed-role/ssh-admin/i-78327237832782376',
        client_user_id: 'ASGDJKWAGAWDWAD',
        inferred_aws_region: '',
        inferred_entity_id: '',
        inferred_entity_type: '' },
     lease_duration: 2764800,
     renewable: true,
     entity_id: '21hj3ghdsfg-asda-kj2hy3-asdf-8a7s5df5sadf' } }

(data modified for privacy)

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 8, 2018

The Content-Length header should always contain a string or an array of strings for all AWS request to any service, right? Because then I'll send a pull request to the aws4 project with the fix. For now I'll use my own version as a workaround.

Thank you both for your fast response! If you need me for further testing you can hmu :)

DaniGuardiola commented Jan 8, 2018

The Content-Length header should always contain a string or an array of strings for all AWS request to any service, right? Because then I'll send a pull request to the aws4 project with the fix. For now I'll use my own version as a workaround.

Thank you both for your fast response! If you need me for further testing you can hmu :)

@jefferai

This comment has been minimized.

Show comment
Hide comment
@jefferai

jefferai Jan 9, 2018

Member

@DaniGuardiola I've updated (via force-push) the aws-jsonnumber branch. It should now accept the Content-Length properly. Can you check?

Member

jefferai commented Jan 9, 2018

@DaniGuardiola I've updated (via force-push) the aws-jsonnumber branch. It should now accept the Content-Length properly. Can you check?

@jefferai jefferai added this to the 0.9.2 milestone Jan 9, 2018

@DaniGuardiola

This comment has been minimized.

Show comment
Hide comment
@DaniGuardiola

DaniGuardiola Jan 10, 2018

Checked (9b13355e78dcb48a07049fc71f3ce64d8835403e) and it works just fine, thanks! Will this be ready in 0.9.2? I'm gonna add the workaround in node-vault to support older versions anyway, but good to know it's solved.

DaniGuardiola commented Jan 10, 2018

Checked (9b13355e78dcb48a07049fc71f3ce64d8835403e) and it works just fine, thanks! Will this be ready in 0.9.2? I'm gonna add the workaround in node-vault to support older versions anyway, but good to know it's solved.

@DaniGuardiola DaniGuardiola changed the title from AWS IAM auth does not work for some reason to AWS IAM auth 'Content-Length' header causes validation error when sent as number instead of string or []strings Jan 10, 2018

@jefferai

This comment has been minimized.

Show comment
Hide comment
@jefferai

jefferai Jan 10, 2018

Member

It'll be in 0.9.2, yep!

Member

jefferai commented Jan 10, 2018

It'll be in 0.9.2, yep!

jefferai added a commit that referenced this issue Jan 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment