Use with Cognito?

[paolavness]

Hi there,

I am migrating our app from using AWS SDK's multipart uploader (with transfer acceleration and v4 signing ) to Evaporate.js

Two questions - does Evaporate rely on the aws sdk in any way?

And we're using Cognito to auth all our users. Additionally, our bucket policy is setup to only allow access from all auth'd users.

I am wondering if there is a way to utilize this fact, rather than requiring the step of signUrl or customAuthMethod? On one hand, feels like a step backward to have to to include an aws_key and signage for an already authenticated user. On the other hand I am hoping Evaporate will be more robust and flexible that the sdk's multipart uploader.

Thank you, looking forward to hearing from you,
Paola

[jakubzitny]

Hi @paolavness. Evaporate does not rely on aws sdk, you can check the source code yourself. And you probably have to implement the authentication yourself. You can do that in customAuthMethod if there is some sort of "temporary" aws_key from Cognito. If not, you probably have to figure it out 😉

[bikeath1337]

All users of EvaporateJS are required to understanding AWS signing principle, be they V2 or V4. The URL must be signed per the AWS documentation. If you already have an authenticated user, then you probably need to use that information to determine if you return a signed URL or not, but you cannot avoid signing the URL. This is how AWS manages security and upload consistency.

[paolavness]

2nd EDIT: Solution

Based the below working solution for EvaporateJS 2.x on this this helpful post. Thank you Drew!

const { accessKeyId, secretAccessKey, sessionToken } = AWS.config.credentials;

let xAmzHeadersCommon = {'x-amz-security-token': sessionToken }
let xAmzHeadersAtInitiate = xAmzHeadersCommon

return Evaporate.create({
    aws_key: accessKeyId,
    bucket: `${your_bucket}`,
    awsRegion: `${your_region}`,
    computeContentMd5: true,
    cryptoMd5Method: function (data) { return AWS.util.crypto.md5(data, 'base64'); },
    cryptoHexEncodedHash256: function (data) { return AWS.util.crypto.sha256(data, 'hex'); },
    logging: true,
    s3Acceleration: true,
    signTimeout: 10,
    maxConcurrentParts:5,
    sendCanonicalRequestToSignerUrl: true,
    customAuthMethod: customAuth,  // defined below
    signParams:  {secretKey: secretAccessKey},
    partSize: 1024 * 1024 * 6,
    s3FileCacheHoursAgo: 1,  
    allowS3ExistenceOptimization: true, // Checks for reexistence of file; Requires <AllowedMethod>HEAD</AllowedMethod>
}
)
.then(function (evaporate) {
    var addConfig = {
        name: `${your_filename}`,
        file: `${your_file}`,
        contentType: `${your_filetype}`,
        xAmzHeadersCommon,
        xAmzHeadersAtInitiate,
        progress: function (progressVal) { console.log('Progress: ', progressVal); },
        complete: function (_xhr, awsKey) { console.log('Complete!'); },
    },
    overrides = {
         // Any overides
    };
    evaporate.add(addConfig, overrides)
        .then(function (awsObjectKey) {
            console.log('File successfully uploaded to:', awsObjectKey);
            },
            function (reason) {
            console.log('File did not upload sucessfully:', reason);
            });
});

/**
 * Custom authorization method specifically for Evaporate.js - ie, needs to retain this parameter signature
 * @param {Object} signParams Object containing secretKey for signing
 * @param {Object} signHeaders unused
 * @param {String} stringToSign encoded string to sign strong to sign
 * @param {String} signatureDateTime Amz formatted date time sting
 * @param {String} canonicalRequest unused
 */
export const customAuth = (signParams, signHeaders, stringToSign, signatureDateTime, canonicalRequest) => {
    const stringToSignDecoded = decodeURIComponent(stringToSign)
    const requestScope = stringToSignDecoded.split('\n')[2];
    const [date, region, service, signatureType] = requestScope.split('/');

    return new Promise(function (resolve, reject) {   
        const round1 = hmacSha256(`AWS4${signParams.secretKey}`, date);
        const round2 = hmacSha256(round1, region);
        const round3 = hmacSha256(round2, service);
        const round4 = hmacSha256(round3, signatureType);
        const final = hmacSha256(round4, stringToSignDecoded, 'hex');
        resolve( final )
    });
}

---

1st EDIT: Not to worry I've found something.

Mmmmmm I figured as much... that I'd need to figure it myself.

@jakubzitny could you possibly give a footprint out of what the final signURL from looks like or what it and the customAuthMethod return, when used via Evaporate? I'm trying to figure out which of my cognito and auth goodies I can map in.

@bikeath1337 currently using cognito, https, v4 and authorizations headers, uploads via the aws sdk's S3.upload() do not require signage - X-Amz-Content-Sha256:UNSIGNED-PAYLOAD

and perhaps I'm mixing things up here. the sdk does takes care of quite a bit of behind the scenes goodies.

[bikeath1337]

@paolavness if you open an issue on an open source project, and then edit your own reply reporting "i've found something"....

It's common courtesy to those volunteers involved in the project to report what you actually found; whether it is still an issue and, appropriately close the ticket, if the issue has been resolved.

@paolavness to reply to your mention about UNSIGNED-PAYLOAD, you need to refer to the AWS documentation. The requests are always signed, but the PAYLOAD of the request is not signed. If you look at Evaporate's code Line 1365, you'll see we do the same, but AWS still requires the request to be signed. The signing is your job. Evaporate's codebase has several examples in several languages, which you can refer to.

The volunteers for this project also provided an out of the box JavaScript implementation and sample HTML which you can use to experiment in a development sandbox.

[paolavness]

@bikeath1337 I was about to post my solution here for other cognito users... Only to see my previous comment deleted. I'm stunned in the censorship of what felt like positive + constructive feedback re your style. Not sure what else to say other than it feels like a good idea to talk about this in person.

[bikeath1337]

The purpose of this project is to improve it, not create issues about what you consider to be my personal style. If you have a pull request that can implement or improve for Cognito users, we're all for that contribution, but you should keep personal assessments personal and private.

[paolavness]

Yeah, posted my solution for this issue above.

[mordka]

@paolavness Have you experienced any time out issues after 1h? I'm getting the following error

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>ExpiredToken</Code>
<Message>The provided token has expired.</Message>
<Token-0>AgoGb3JpZ2luEIn//////////wEaDmFwLXNvdXRoZWFzdC0yIo.............xrbSDEai+M+ZvczUXcRhKOGvH0BhvPh5hNEcKjy10jf+hGzg4YYW+ph1CTfsaTE9ZM48XJtdDVsT4gc1dcxqFUo7sdwcDqmF8P/kQSCgqRiin1KHigMMntgdEF</Token-0>
<RequestId>61CB27C6E74C2F52</RequestId>
<HostId>1iJU52GeSyaX8aH6RfZAw9/HSB4gvme/4Asu5G...........SpjBa0Vm8o4oSOJGnfb187eRFpo20=</HostId>
</Error>

[paolavness]

@mordka Yes, its the sts tokens - they expire by default after an hour. I experienced the issue both with the S3 ManagedUpload and EvaporateJS.

You need to manually refresh the cognitoUser session using the session.refreshToken and an update/refresh for identity credentials, the latter because we're using Amazon federated identities.

My final solution worked for S3's managedUpload. I have not tried it yet with evaporateJs tho. Would love to hear if you get it working with evaporateJS.

[mordka]

@paolavness Thanks for quick response. The thing is, I'm familiar with this and I do refresh both cognito JWT tokens and STS temporary credentials every 50 minutes - this works well for regular IAM signed requests to API Gateway. This doesn't work for S3 uploads with EvaporateJS. It's critical problem and I'm looking into this for longer time. This could be the signer issue. I will answer here once I manage to fix it.

[paolavness]

@mordka Its a pleasure, i know how frustrating this can be. Are you using EvaporateJS requests with API Gateway?

You need to pass the updated AWS.config.credentials to EvaporateJS after they have been refreshed. If you dont, requests will continue with the outdated ones.

If you revert to S3's ManagedUpload, I uncovered a a bug in it in which, unlike other requests, the ManagedUpload.next() is unaware of changes to AWS.config.credentials. The result is that it to would fail after one (or two hours if the internal refresh call happened before STS expired) I patched it to re-initialize to the updated AWS.config.credentials. You may need to do something similar with EvaporateJS.

[mordka]

@paolavness thanks for help, I found that credentials (aws, session, secret) spread in code all over the place. So to update it everywhere we need to overwrite the following places:

• evaporate.config.aws_key
• evaporate.filesInProcess[].xAmzHeaders
• evaporate.filesInProcess[].s3Parts[].config.aws_key
• evaporate.filesInProcess[].s3Parts[].request.x_amz_headers

The code:

 /** We need update evaporateJS properties with updated credentials
   * This is called when credentials are refreshed every 50 minutes */
  public updateCredentials() {
    //Evaporate.handle obtained in the Evaporate.create() promise handler
    if (!Evaporate.handle) return;
    Evaporate.handle.config.aws_key = AWS.config.credentials.accessKeyId;
    Evaporate.handle.filesInProcess.forEach(fileInProcess => {
      this.buildXamzHeaders(fileInProcess)
    })
  }

  private buildXamzHeaders(fileInProcess: any) {
    let header = {'x-amz-security-token': AWS.config.credentials.sessionToken};
    fileInProcess.xAmzHeadersCommon = header;
    fileInProcess.xAmzHeadersAtInitiate = header;
    //amend request in every PutPart instance - all requests are prepared before processing the parts!
    if (fileInProcess.s3Parts && fileInProcess.s3Parts.length>0) {
      fileInProcess.s3Parts.forEach((part: any) => {
        part.awsRequest.con.aws_key = AWS.config.credentials.accessKeyId;
        part.awsRequest.request.x_amz_headers['x-amz-security-token'] = AWS.config.credentials.sessionToken;
      })
    }
  }
  

[paolavness]

@mordka thanks for adding this here. We're looking at re-implementing evaporatejs in our production environment and I am wondering the above solution continued to work for you - in a production environment?
Thank you,
Paola

Thanks,
Paola

[mordka]

Hi @paolavness, yes it still works great.

[wschopohl]

Hey many thanks to @paolavness and @mordka. You helped me save a lot of frustration implementing Token Authentication with evaporate. One small addition, with the version of evaporate that I'm using (2.1.4) it was necessary to adjust mordka's code slightly to check if the awsRequest object is defined. Seems like this object is only defined on still outstanding parts. My implementation looked like this:

if (fileInProcess.s3Parts && fileInProcess.s3Parts.length>0) { 
   fileInProcess.s3Parts.forEach((part: any) => {
        if(part.awsRequest !== undefined) {
            part.awsRequest.con.aws_key = AWS.config.credentials.accessKeyId;
            part.awsRequest.request.x_amz_headers['x-amz-security-token'] = AWS.config.credentials.sessionToken;
        }
    })
}

[deepak-agarwal]

@paolavness i am your code snippet with amplify , but i always get a signature mismatch. Instead of using aws-sdk i am using crypto.js for hashing, any help would be appreciated.

AWS Code: SignatureDoesNotMatch, Message:The request signature we calculated does not match the signature you provided. Check your key and signing method.status:403

import { Credentials } from "@aws-amplify/core";
import { HmacSHA256, MD5, SHA256 } from "crypto-js";

export const customAuth = (
  signParams,
  signHeaders,
  stringToSign,
  signatureDateTime,
  canonicalRequest
) => {
  const stringToSignDecoded = decodeURIComponent(stringToSign);
  const requestScope = stringToSignDecoded.split("\n")[2];
  const [date, region, service, signatureType] = requestScope.split("/");

  return new Promise(function (resolve, reject) {
    const round1 = HmacSHA256(`AWS4${signParams.secretKey}`, date);
    const round2 = HmacSHA256(round1, region);
    const round3 = HmacSHA256(round2, service);
    const round4 = HmacSHA256(round3, signatureType);
    const final = HmacSHA256(round4, stringToSignDecoded);
    resolve(final);
  });
};

const _get = (name) =>
  Credentials.get().then(function (credentials) {
    if (!credentials) return false;
    var cred = Credentials.shear(credentials);
    return cred[name];
  });

export const _create = async (filename, fileObject, contentType) =>
  Evaporate.create({
    aws_key: await _get("accessKeyId"),
    bucket: "bucketname",
    awsRegion: "us-east-2",
    computeContentMd5: true,
    cryptoMd5Method: function (data) {
      return MD5(data, "base64");
    },
    cryptoHexEncodedHash256: function (data) {
      return SHA256(data, "hex");
    },
    logging: true,
    signTimeout: 10,
    maxConcurrentParts: 5,
    customAuthMethod: await customAuth,
    signParams: { secretKey: await _get("secretAccessKey") },
    partSize: 1024 * 1024 * 6,
    s3FileCacheHoursAgo: 1,
  }).then(async function (evaporate) {
    const addConfig = {
        name: filename,
        file: fileObject,
        contentType: contentType,
        xAmzHeadersCommon: {
          "x-amz-security-token": await _get("sessionToken"),
        },
        xAmzHeadersAtInitiate: {
          "x-amz-security-token": await _get("sessionToken"),
        },
        progress: function (progressVal) {
          console.log("Progress: ", progressVal);
        },
        complete: function (_xhr, awsKey) {
          console.log("Complete!");
        },
      },
      overrides = {
        // Any overides
      };
    evaporate.add(addConfig, overrides).then(
      function (awsObjectKey) {
        console.log("File successfully uploaded to:", awsObjectKey);
      },
      function (reason) {
        console.log("File did not upload sucessfully:", reason);
      }
    );
  });

[cesco69]

@paolavness I also have a signature mismatch. i am using crypto-js for hashing, any help would be appreciated.

import * as Evaporate from 'evaporate';
import * as crypto from "crypto-js";

Evaporate.create({
  aws_key: <ACCESS_KEY>,
  bucket: 'my-bucket',
  awsRegion: 'eu-west',
  computeContentMd5: true,
  cryptoMd5Method: data => crypto.algo.MD5.create().update(String.fromCharCode.apply(null, new Uint32Array(data))).finalize().toString(crypto.enc.Base64),
  cryptoHexEncodedHash256: (data) => crypto.algo.SHA256.create().update(data as string).finalize().toString(crypto.enc.Hex),
  logging: true,
  maxConcurrentParts: 5,  
  customAuthMethod: (signParams: object, signHeaders: object, stringToSign: string, signatureDateTime: string, canonicalRequest: string): Promise<string> => {
      const stringToSignDecoded = decodeURIComponent(stringToSign)
      const requestScope = stringToSignDecoded.split("\n")[2];
      const [date, region, service, signatureType] = requestScope.split("/");
      const round1 = crypto.HmacSHA256(`AWS4${signParams['secret_key']}`, date);
      const round2 = crypto.HmacSHA256(round1, region);
      const round3 = crypto.HmacSHA256(round2, service);
      const round4 = crypto.HmacSHA256(round3, signatureType);
      const final = crypto.HmacSHA256(round4, stringToSignDecoded);
      return Promise.resolve(final.toString(crypto.enc.Hex));
  },
  signParams: { secretKey: <SECRET_KEY> },
  partSize: 1024 * 1024 * 6
  }).then((evaporate) => {
      evaporate.add({
          name: 'my-key',
          file: file, // file from <input type="file" />
          xAmzHeadersCommon: { 'x-amz-security-token': <SECURITY_TOKEN> },
          xAmzHeadersAtInitiate: { 'x-amz-security-token': <SECURITY_TOKEN> },
       }).then(() => console.log('complete'));
  },
     (error) => console.error(error)
  );

result

AWS Code: SignatureDoesNotMatch, Message:The request signature we calculated does not match the signature you provided. Check your key and signing method.status:403