Example code for lib-storage doesn't work

[jagthedrummer]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

This example code doesn't work:

https://github.com/aws/aws-sdk-js-v3/blob/main/lib/lib-storage/example-code/file-upload.ts

When I run that an error is thrown saying:

Body Data is unsupported format, expected data to be one of: string | Uint8Array | Buffer | Readable | ReadableStream | Blob

SDK version number

@aws-sdk/lib-storage@3.577.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.15.0

Reproduction Steps

Run this example code:

aws-sdk-js-v3/lib/lib-storage/example-code/file-upload.ts

Lines 1 to 24 in 9f65b40

	import * as fs from "fs";
	import { Upload } from "@aws-sdk/lib-storage";
	import { S3 } from "@aws-sdk/client-s3";
	import { configuration } from "./config";
	const fileStream = fs.createReadStream(__dirname + "/big.file");
	(async () => {
	const upload = new Upload({
	params: {
	Bucket: configuration.Bucket,
	Key: configuration.Key,
	Body: fileStream,
	},
	client: new S3({}),
	queueSize: 3,
	});
	upload.on("httpUploadProgress", (progress) => {
	console.log(progress);
	});
	await upload.done();
	})();

Observed Behavior

And error is thrown that says:

Body Data is unsupported format, expected data to be one of: string | Uint8Array | Buffer | Readable | ReadableStream | Blob

Expected Behavior

The example code should work without throwing errors.

Possible Solution

After much searching and piecing together of various answers I found that the following two methods can be used to turn a node ReadStream into a ReadableStream. (In an electron app using node filesystem apis. I don't know if this would work in vanilla node.)

  /**
   * Taken from Next.js doc
   * https://nextjs.org/docs/app/building-your-application/routing/router-handlers#streaming
   * Itself taken from mozilla doc
   * https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream#convert_async_iterator_to_stream
   * @param {*} iterator
   * @returns {ReadableStream}
   */
  iteratorToStream(iterator) {
    return new ReadableStream({
      async pull(controller) {
        const { value, done } = await iterator.next()

        if (done) {
          controller.close()
        } else {
          controller.enqueue(new Uint8Array(value))
        }
      },
    })
  }

  /**
   * From https://github.com/MattMorgis/async-stream-generator
   */
  async *nodeStreamToIterator(stream) {
    for await (const chunk of stream) {
      yield chunk;
    }
  }

So then instead of this:

aws-sdk-js-v3/lib/lib-storage/example-code/file-upload.ts

Line 6 in 9f65b40

const fileStream = fs.createReadStream(__dirname + "/big.file");

You can do this:

const nodeStream = fs.createReadStream(__dirname + "/big.file")
const iterator = nodeStreamToIterator(nodeStream)
const fileStream = iteratorToStream(iterator)

Additional Information/Context

I'm using this in an electron app.

A few related issues, none of which seem to contain a full working resolution:

#2365

#2183

#2145

[aBurmeseDev]

Hi @jagthedrummer - thanks for reaching out, sharing feedback on our documentations and workarounds.

As we're always working on improving our docs to be more clear, concise and effective, feedback like this are crucial to our work.

As you'd probably already know, the reason behind the error is the Body Data type mismatch which was sent and rejected by S3. I haven't attempted to reproduce it with electron app but as soon as we can confirm, will update the code example accordingly.

Best,
John

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[jagthedrummer]

This is still an issue.

Also, the bot seems counterproductive and frankly kind of hostile to the idea that people should let you know about things that aren't working as expected. The message it seems to send is "we didn't bother to get around to this, so we're going to throw it in the trash".