Cannot upload buffer in Express route NodeJS

[thesiti92]

• Package Name: @azure/storage-blob
• Package Version: "^12.1.2"
• Operating system: windows 10 (WSL2)
• nodejs
  • version: 12
• browser
  • name/version:
• typescript
  • version: 3.8.3
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug

I want to upload an image posted in an express route using the blob storage sdk. I end up getting the HMAC auth errors when i try to do so. I instead turned to making a test case and it turns out that i can post a plain text string in the route but not a string converted to a buffer, indicating that my credentials are correct, etc... The same string as buffer upload works when running separately in a node script. I'm trying to upgrade from v2 to v12 so fixing this is a big blocker.

To Reproduce
Here's my express router:

import storageClient from './blobStorageClient'

export const CONTAINER_NAME = 'images'

const imageClient = storageClient.getContainerClient(CONTAINER_NAME)

async function testUpload(): Promise<void> {
  const filename = 'image' + Date.now().toFixed(0)
  const blobClient = imageClient.getBlockBlobClient(filename)
  const message = Buffer.from('hello world')
  await blobClient.upload(message, message.byteLength)
  console.log('Successfully uploaded as ' + filename)
}
const profileImgUpload = express.Router()
profileImgUpload.post('/', async (req, res) => {
  await testUpload()
})

My Client initialization script

import {
  BlobServiceClient,
  StorageSharedKeyCredential
} from '@azure/storage-blob'
import { setLogLevel } from '@azure/logger'
import '../../env'
setLogLevel('info')
// Enter your storage account name and shared key
const account = process.env.AZ_ACCOUNT_NAME
const accountKey = process.env.AZ_ACCOUNT_KEY

if (!accountKey || !account) {
  throw Error('Azure creds env variables not set')
}

const creds = new StorageSharedKeyCredential(account, accountKey)
export default new BlobServiceClient(
  `https://${account}.blob.core.windows.net`,
  creds
)

and the ts-node script that works fine:

import '../../env'
import client from './blobStorageClient'
const containerClient = client.getContainerClient('images')

async function main(): Promise<void> {
  const filename = 'image' + Date.now().toFixed(0)
  const blobClient = containerClient.getBlockBlobClient(filename)
  const message = Buffer.from('hello world')
  await blobClient.upload(message, message.byteLength)
  console.log('Successfully uploaded as ' + filename)
}

main()

Expected behavior
Image should upload fine and the console log should print

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I used azure logger as @ljian3377 suggested and the request from the node script and from the express server appear to be the same although it doesnt let me see the auth headers.

[jiacfan]

Thanks for reaching us.

Add link to issue 8903.

At same time, @thesiti92 would you help to provide the detailed error messages?

Thanks,
Jiachen

[thesiti92]

i get the generic <AuthenticationErrorDetail>The MAC signature found in the HTTP request '<sig>' is not the same as any computed signature. error message. is there any way i can get you more details?

[jiacfan]

@thesiti92

Would you help to provide the server request id?

e.g. The RequestId field in body:

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>AuthenticationFailed</Code>
    <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\n
RequestId:XXX00000-0000-0000-0000-000000000000\n
Time:2020-05-13T16:23:38.3682050Z</Message>
    <AuthenticationErrorDetail>The MAC signature found in the HTTP request 'QDv3EVRyiJiAgOT9GWtWC7VBITLvCEasKoTd3xai4+4=' is not the same as any computed signature. Server used following string to sign: 'PUT ...'.</AuthenticationErrorDetail>
</Error>

It would be nice if the repro is in 5 days.

Thanks,
Jiachen

[thesiti92]

Here's my entire error string, repro'd 2 mins before posting this

<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:66b8fd21-501e-008c-08a5-3de57c000000
Time:2020-06-08T14:59:09.5280190Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'AK8+bWG5qefZxrmZXzotJZU8ASipHfqOTbOrLih8N+8=' is not the same as any computed signature. Server used following string to sign: 'PUT

[jiacfan]

The request is put file with sharedKey authN, and it looks like when route with express content-length is not provided.
More details: As described here, the signature string to be sign should be in following format:

StringToSign = VERB + "\n" +  
               Content-Encoding + "\n" +  
               Content-Language + "\n" +  
               Content-Length + "\n" +  
               Content-MD5 + "\n" +  
               Content-Type + "\n" +  
               Date + "\n" +  
               If-Modified-Since + "\n" +  
               If-Match + "\n" +  
               If-None-Match + "\n" +  
               If-Unmodified-Since + "\n" +  
               Range + "\n" +  
               CanonicalizedHeaders +   
               CanonicalizedResource;

While in service side, we didn't see Content-Length field. Will check further and see if custom pipeline can help to serve this issue. Will update this thread further.

[jiacfan]

axios/axios#2107 @jeremymeng, as we changing to use node-fetch, it looks like the workaround is no more necessary, considering node-fetch brings override's semantic for content-length: https://github.com/node-fetch/node-fetch/blob/0936a9af0a049eb05897779f3f0df6b903a95d60/src/request.js#L237, what's your through?

[jeremymeng]

That's right we should remove the workaround for axios.

[jiacfan]

We'll involve it in next GA version release.

[thesiti92]

GA = "generally available" ?

[jiacfan]

@thesiti92 Yes, we'll involved it in July milestone.

Thanks,
Jiachen

[damianmiranda-ge]

What's the workaround?

[thesiti92]

Idt there is one?

[damianmiranda-ge]

For now I'm saving the buffer into a temporal file and then uploading it.

[ljian3377]

For now I'm saving the buffer into a temporal file and then uploading it.

@damianmiranda-ge
Wait, you mean by uploadFile which calls upload/stageBlock internally?
That shouldn't work as well if we have this content length problem, I think.

[ljian3377]

No easy workaround.
A custom pipeline which adds the content length header back would do. And it should be injected as the last pipeline. Refer to this InjectorPolicyFactory for an example.

azure-sdk-for-js/sdk/storage/storage-blob/test/retrypolicy.spec.ts

Line 36 in 9c9f64d

const injector = new InjectorPolicyFactory(() => {

[jiacfan]

@thesiti92 FYI on @ljian3377 mentioned workaround.

Thanks,
Jiachen

[thesiti92]

Sounds good. Will try it out! Currently I'm just on v2 which works fine it's just not ideal

[jiacfan]

Yes, v2 should work as well, feel free to let us know if there is any problem on the work around. And also we'll provide the fix in July release.

[jiacfan]

Close as fix released.