Expect: 100-continue HTTP protocol that s3 putObject uses does not work properly for bun

[tomlarkworthy]

What version of Bun is running?

0.7.0

What platform is your computer?

Darwin 21.4.0 arm64 arm

What steps can reproduce the bug?

Using the AWS s3-client, bun calls to putObject with a non-zero Body decode and expose "httpStatusCode: 100", whereas the same call in nodejs resolves "httpStatusCode: 200". This is a problem, because additional headers expose important funcitonality like the versionId of the object written, which is no longer accessible when using bun.

The problem is s3 returns two status codes in a row. See the example in AWS documentation https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_Example_1

Note this happens for non-version enabled buckets and also for real s3 servers (I repro on minio but it affects normal usage too).

repro.ts

import { PutObjectCommandInput, S3 } from "@aws-sdk/client-s3";
const run = async () => {
  const s3 = new S3({
    endpoint: "http://127.0.0.1:9000",
    region: "eu-central-1",
    credentials: {
      accessKeyId: "mps3",
      secretAccessKey: "ZOAmumEzdsUUcVlQ",
    },
  });

  try {
    console.log("creating bucket");
    await s3.createBucket({
      Bucket: "test9",
    });
  } catch (e) {}

  try {
    console.log("enable version");
    await s3.putBucketVersioning({
      Bucket: "test9",
      VersioningConfiguration: {
        Status: "Enabled",
      },
    });
  } catch (e) {
    console.error(e);
  }

  const command: PutObjectCommandInput = {
    Bucket: "test9",
    Key: "key",
    ContentType: "application/json",
    Body: "{}", // Important
  };

  const fileUpdate = await s3.putObject(command);

  console.log(fileUpdate);

  // On node
  /*
  {
    '$metadata': {
      httpStatusCode: 200,
      requestId: '17747F9E88AB6C5B',
      extendedRequestId: 'dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8',
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    ETag: '"99914b932bd37a50b983c5e7c90ae93b"',
    VersionId: '6454e510-1240-401f-9618-74346da95e3e'
  }
    */
  /* on bun */
  /*
  {
    $metadata: {
      httpStatusCode: 100,
      requestId: undefined,
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    }
  }
*/
};

run();

To run the repro you need a s3 API running locally, here is a docker-compose for the backend

version: '3'
services:
  minio:
    image: 'minio/minio:latest'
    ports:
      - '${FORWARD_MINIO_PORT:-9000}:9000'
      - '${FORWARD_MINIO_CONSOLE_PORT:-9090}:9090'
    environment:
      MINIO_ROOT_USER: 'mps3'
      MINIO_ROOT_PASSWORD: 'ZOAmumEzdsUUcVlQ'
    command: minio server /data/minio --console-address ":9090"
volumes:
  minio:
    driver: local

What is the expected behavior?

s3.putObject calls $metadata.httpStatusCode should return 200 and include VersionIid if availible.

What do you see instead?

s3.putObject calls $metadata.httpStatusCode return 100 and do not include VersionIid. This contrasts to node executing the same code which handles to 100 continue header properly.

Additional information

AWS is using the 100-continue protocol https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/100 which seems not to work properly for bun

[tomlarkworthy]

Nodejs has specific handling for 100-continue with the 'continue' event

Sending an 'Expect' header will immediately send the request headers -- https://nodejs.org/dist/v5.2.0/docs/api/http.html#http_event_continue

bun source does not contain '100-continue' and the area where I would expect it to be handled does not consider it:

bun/src/http_client_async.zig

Line 2785 in b02f097

pub fn handleResponseMetadata(

AWS SDK has some explicit middleware for ExpectContinue, if I comment this out the SDK works as expected.

https://github.com/aws/aws-sdk-js-v3/blob/6108789c4328ef71eddc728d383f101998621096/packages/middleware-expect-continue/src/index.ts#L20

[Northernside]

Same thing happening for me, any news on this?

[dndred]

As a temporary workaround, uploading by pre-signed URLs can be used.

import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

import options from '@options'

const getS3SignedUrl = async (bucked: string, key: string) => {
  const client = new S3Client(options.s3configuration)
  const command = new PutObjectCommand({
    Key: key,
    Bucket: bucked,
  })

  return getSignedUrl(client, command, { expiresIn: 600 })
}

export const uploadToS3 = async (buffer: Buffer, key: string) => {
  const url = await getS3SignedUrl(options.s3configuration.bucket, key)
  
  return fetch(url, {
    method: 'PUT',
    body: buffer,
  })
}

[Northernside]

Thank you! 👍 I'll try it out once I'm back home on Friday :)

[koenbok]

😭 https://twitter.com/koenbok/status/1702653896697463283