Using SDK v3 in the browser still uses NodeJS `fs` method

[awaisabir]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I am using the AWS v3 SDK in the browser, and it seems like the SDK does not recognize the environment it is running in. Naturally, in the browser there is no access to Node internals like fs, path, etc etc. However, it still seems to be searching for a file on the file system for credentials, even though it is being executed in the browser and I am providing it credentials.

I am providing credentials through my own custom endpoint, but when this code executes in my React application, the Webpack bundle builds fine, but fails at run-time with the following error:

Uncaught (in promise) TypeError: Cannot destructure property 'readFile' of 'fs_1.promises' as it is undefined.
    at ./node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/getSSOTokenFromFile.js (getSSOTokenFromFile.js:6:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js (index.js:7:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/credential-provider-ini/dist-cjs/fromIni.js (fromIni.js:4:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/credential-provider-ini/dist-cjs/index.js (index.js:4:1)

Why is my SDK in the browser calling fs methods? This doesn't make sense to me. Am I misunderstanding or missing something in the configuration?

SDK version number

@aws-sdk/client-cloudwatch:^3.358.0

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

Chrome: 115.0.5790.102

Reproduction Steps

Webpack Config:

const path = require("path");
const HtmlWebpackPlugin = require("html-webpack-plugin");
const NodePolyfillPlugin = require("node-polyfill-webpack-plugin");

module.exports = {
  entry: path.resolve("src/index.tsx"),
  output: {
    path: path.resolve(__dirname, "../dist"),
  },
  module: {
    rules: [
      {
        test: /\.(png|woff|woff2|eot|ttf|svg)$/,
        loader: "url-loader",
        options: { limit: false },
      },
      {
        test: /\.tsx?$/,
        use: "ts-loader",
        exclude: /node_modules/,
      },
      {
        test: /\.css$/i,
        use: ["style-loader", "css-loader"],
      },
      {
        test: /\.js$/,
        enforce: "pre",
        use: ["source-map-loader"],
        exclude: /node_modules/
      },
    ],
  },
  plugins: [
    new HtmlWebpackPlugin({ template: "./public/index.html" }),
    new NodePolyfillPlugin(),
  ],
  resolve: {
    mainFields: ["browser", "main", "module"],
    extensions: [".tsx", ".ts", ".js"],
    fallback: {
      "fs": false,
      "child_process": false,
      "path": require.resolve("path-browserify"),
      "crypto": require.resolve("crypto-browserify"),
      "stream": require.resolve("stream-browserify"),
      "assert": require.resolve("assert"),
      "http": require.resolve("stream-http"),
      "http2": require.resolve("stream-http"),
      "https": require.resolve("https-browserify"),
      "os": require.resolve("os-browserify"),
      "url": require.resolve("url") 
    }
  },
};

TS File for Client:

import { CloudwatchClient } from "@aws-sdk/cloudwatch-client";

const getCredentials = async () => {
  const res = await fetch(`/credentials/endpoint`);
  const json = await res.json();

 return json();
}

export const initializeFooClient = async () => {
  const credentials = await getCredentials();

  const fooClient = new CloudwatchClient({
    region: "us-east-2",
    credentials: async () => {
      return {
        ...credentials
      };
    }
  });
  
  return { fooClient };
}};

React Component:

import { initializeFooClient } from "xyz";

const TestComponent = () => {
  useEffect(() => {
    const initalizeClient = async () => {
  
      // redacted the URL, but it does exist and works when I send a cURL request
      await initializeFooClient();
    };

    initalizeClient();
  }, []);

Observed Behavior

Get this error

Uncaught (in promise) TypeError: Cannot destructure property 'readFile' of 'fs_1.promises' as it is undefined.
    at ./node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/getSSOTokenFromFile.js (getSSOTokenFromFile.js:6:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js (index.js:7:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/credential-provider-ini/dist-cjs/fromIni.js (fromIni.js:4:1)
    at __webpack_require__ (bootstrap:24:1)
    at fn (hot module replacement:62:1)
    at ./node_modules/@aws-sdk/credential-provider-ini/dist-cjs/index.js (index.js:4:1)

Why is my SDK v3 in the browser calling fs methods? Should it not know at run-time that it is running in a browser environment? Is there something wrong with my webpack configuration? The target for my webpack config is also defaulted to web.

Expected Behavior

It should initialize the SDK with the credentials I am providing, and should also know that it should not look for a file on the file system, as it is running in the browser 😕

Possible Solution

No response

Additional Information/Context

I have tried adding the following in my package.json, and it did not work:

  "browser": {
    "crypto": false,
    "fs": false,
    "path": false,
    "os": false,
    "net": false,
    "stream": false,
    "tls": false
  },

I have tried adding fallbacks in my webpack config, and that also does not work.

[RanVaknin]

Hi @awaisabir ,

I'm not sure what kind of credentials you are loading from /credentials/endpoint, or where ...credentials is coming from.

This part is especially confusing to me:

  const client = await getCredentials();

Are you trying to assign credentials to client? or is this confusingly named client instead of credentials?

In the following code snippet:

credentials: async () => {
      return {
        ...credentials
      };
    }

it's not clear where the credentials object that you're spreading (...credentials) is coming from, as it's not defined in your code.
Additionally, your async function needs to return something that conforms to the Credentials interface. Perhaps this does not return what you expect it to return?

A valid credentials object should look like this:

credentials: {
  accessKeyId: credentials.accessKeyId,
  secretAccessKey: credentials.secretAccessKey,
  sessionToken: credentials.sessionToken
}

My guess is when this fails, the SDK will invoke the credentials chain and try to load creds from all the available providers in the chain. The first one being INI credentials (credentials stored on disc) so it will invoke fs since its supported in NodeJS. The solution to this is to properly setup the client with valid credentials.

Since its not clear how / where you are loading credentials from I don't know what type of credentials you are getting back (static or temporary?)

At any rate, the recommended way to get credentials in the browser is to use Cognito Identity pools.
Please see Loading Browser Credentials

Thanks,
Ran~

[awaisabir]

@RanVaknin My apologies. I have updated the code snippet to:

const credentials = await getCredentials();

---

My credentials are being returned in the correct format:

credentials: {
  accessKeyId: credentials.accessKeyId,
  secretAccessKey: credentials.secretAccessKey,
  sessionToken: credentials.sessionToken
}

---

Also updating here.. I was able to get this working, but I had to add the following snippets myself on my webpack configuration. You can find my webpack config below:

module.exports = {
  entry: path.resolve("path/to/src/index.tsx"),
  output: {
    path: path.resolve(__dirname, "/path/to/dist"),
  },
  module: {
    rules: [
      {
        test: /\.(png|woff|woff2|eot|ttf|svg)$/,
        loader: "url-loader",
        options: { limit: false },
      },
      {
        test: /\.tsx?$/,
        use: "ts-loader",
        exclude: /node_modules/,
      },
      {
        test: /\.css$/i,
        use: ["style-loader", "css-loader"],
      },
      {
        test: /\.js$/,
        enforce: "pre",
        use: ["source-map-loader"],
        exclude: /node_modules/
      },
      // needed to add this. But why??
      {
        test: /@aws-sdk\/client-.*\//,
        resolve: {
          alias: {
            "./runtimeConfig": "./runtimeConfig.browser",
          },
        },
      },
    ],
  },
  plugins: [
    new HtmlWebpackPlugin({ template: "./path/to/index.html" }),
    // needed this to polyfill the process call
    new ProvidePlugin({
      process: "process/browser",
    }),
    new ModuleFederationPlugin({
      name: "Some Name",
      shared: {
        react: {
          singleton: true,
          requiredVersion: "^18.2.0",
        },
        "react-dom": { singleton: true },
      },
    }),
  ],
  resolve: {
    mainFields: ["browser", "main", "module"],
    extensions: [".tsx", ".ts", ".js"],
    // needed to add this. But why? Shouldn't the SDK be doing this?
    fallback: {
      "fs": false,
      "child_process": false,
      "stream": require.resolve("stream-browserify"),
      "http": require.resolve("stream-http"),
      "http2": require.resolve("stream-http"),
      "https": require.resolve("https-browserify"),
    }
  },
};

The fallback here is needed because for some reason the AWS SDK is also calling NodeJS methods - fs, http that I need polyfill with browserify packages. This doesn't make sense to me. Shouldn't the AWS SDK package be doing this itself? Am I missing something here?

[kuhe]

mainFields: ["browser", "main", "module"]

I'm not sure if that array is a priority order for webpack, but you should resolve the module entry point first. Our module replacement directives are for the ESM distribution of the AWS SDK.

Example in @aws-sdk/client-s3 package.json:

  "main": "./dist-cjs/index.js",
  "module": "./dist-es/index.js",
  "browser": {
    "./dist-es/runtimeConfig": "./dist-es/runtimeConfig.browser"
  },

If you resolve the main entry point, the replacement directives in the browser field will not be in effect, causing the application to load the default Node.js implementations, instead of the desired browser-compatible implementations.

⚠️ Do not polyfill the fs, http etc. Node.js packages in a browser environment. They may be very large polyfills, or they may not work as intended in the browser.

[awaisabir]

This array is a priority order for webpack: https://webpack.js.org/configuration/resolve/#resolvemainfields

The browser property takes precedence because it's the first item in mainFields

Should browser not be the first item in my mainFields? Wouldn't this effect the resolution for all other packages?

⚠️ Do not polyfill the fs, http etc. Node.js packages in a browser environment. They may be very large polyfills, or they may not work as intended in the browser.

I understand, but right now this is the only thing that is working 😓

[kuhe]

Browser can be first, the main issue for the SDK is that module should go before main.

[awaisabir]

OK, so moving the order did fix it

mainFields: ["module", "browser", "main"],

That being said, I am not fully sure what kind of effect this will have when installing other packages. From my understanding, packages specify their module and main entries when directing their builds/bundles for Node. This feels like a mitigation rather than a "fix". I don't see why the consumer of this package has to make this change (or any) to get the SDK working in the browser 🤔 .

It is also specifically mentioned in the Webpack documentation to use 'browser' as the main field for when the target is set to web:

When the target property is set to webworker, web, or left unspecified:

module.exports = {
  //...
  resolve: {
    mainFields: ['browser', 'module', 'main'],
  },
};

From what I see, webpack does not allow a mainFields per package, per say; but it could potentially be possible by writing a custom resolver plugin for any @aws-sdk/client-xxxx; but again, should this not be handled by the SDK package itself?

Let me know if I am misunderstanding something here.