Requests to S3 are malformed / missing bucket name in WebServerHandler Lambda

[kevin-mitchell]

Taken at face value, it looks like this parameter is missing from a configuration value somewhere in the Next.js app, deployed Lambda, etc. I'm not sure where this comes from yet, or who / what is responsible for supplying it, but it seems like it should be required. Also, it seems at least possible this is just a red herring for something bigger / different, but I'm creating this issue as a starting point.

Here are the steps I took to reproduce this error:

0. Create new AWS Account
   To do this I just created a brand new account in my organization.

1. Start a brand new NextJS project
   npx create-next-app@latest

Note: here are the resulting dep versions:

{
  "name": "nextjs-cdk-test",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@types/node": "20.3.1",
    "@types/react": "18.2.13",
    "@types/react-dom": "18.2.6",
    "autoprefixer": "10.4.14",
    "eslint": "8.43.0",
    "eslint-config-next": "13.4.6",
    "next": "13.4.6",
    "postcss": "8.4.24",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "tailwindcss": "3.3.2",
    "typescript": "5.1.3"
  }
}

2. Start new CDK project
   npx cdk init sample-app —language typescript

Note: here are the resulting dep versions:

{
  "name": "cdk",
  "version": "0.1.0",
  "bin": {
    "cdk": "bin/cdk.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.1",
    "@types/node": "20.1.7",
    "aws-cdk": "2.84.0",
    "cdk-nextjs-standalone": "^3.2.0",
    "esbuild": "^0.17.16",
    "jest": "^29.5.0",
    "ts-jest": "^29.1.0",
    "ts-node": "^10.9.1",
    "typescript": "~5.0.4"
  },
  "dependencies": {
    "aws-cdk-lib": "2.84.0",
    "constructs": "^10.0.0"
  }
}

3. Move the Next.js app into a sub-directory into the CDK directory called “web”

4. In root directory, bootstrap CDK
   cdk bootstrap aws://XX4312/us-east-2 —profile nextjs-cdk-test-deployer

SUCCESS!!

 ~/S/c/t/**cdk**  …  cdk bootstrap aws://XX4312/us-east-2 —profile nextjs-cdk-test-deployer                                                                                                   1181ms  Wed Jun 21 12:33:46 2023
 ⏳  Bootstrapping environment aws://XX4312/us-east-2…
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of ‘arn:aws:iam::aws:policy/AdministratorAccess’. Pass ‘—cloudformation-execution-policies’ to customize.
**CDKToolkit**: creating CloudFormation changeset…
 ✅  Environment aws://XX4312/us-east-2 bootstrapped.

5. Follow directions on quickstart

https://github.com/jetbridge/cdk-nextjs

In short, I added the dependencies listed

npm install -D esbuild@0.17.16 cdk-nextjs-standalone

Then modified the sample-app CDK stack as such:

import { Stack, StackProps } from “aws-cdk-lib”;
import { Nextjs } from “cdk-nextjs-standalone”;
import { Construct } from “constructs”;

export class CdkStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new Nextjs(this, “Web”, {
      nextjsPath: “./web”, // relative path to nextjs project root
    });
  }
}

6. Running synth results in failure because no region specified

cdk synth CdkStack —profile nextjs-cdk-test-deployer

...
...
Error: stacks which use EdgeFunctions must have an explicitly set region

7. Update default cdk stack config with region fixes EdgeFunction issue

I just modified the default (otherwise unmodified) bin/cdk.ts file produced from the cdk init sample-app .. to include a region (and I also added my account number:

#!/usr/bin/env node
import * as cdk from “aws-cdk-lib”;
import { CdkStack } from “../lib/cdk-stack”;

const app = new cdk.App();
new CdkStack(app, “CdkStack”, {
  env: { account: “XX4312”, region: “us-east-2” },
});

8. Cdk synth worked now

cdk synth CdkStack —profile nextjs-cdk-test-deployer

┌ Building Next.js app ▼ ...
├ Running "npx --yes open-next build" in ./web
Next.js v13.4.6
OpenNext v2.0.0

┌─────────────────────────────────┐
│ OpenNext — Building Next.js app │
└─────────────────────────────────┘


> nextjs-cdk-test@0.1.0 build
> next build

- info Creating an optimized production build  
- info Compiled successfully
- info Linting and checking validity of types  
- info Collecting page data  
- info Generating static pages (4/4)
- info Finalizing page optimization  

Route (app)                                Size     First Load JS
┌ ○ /                                      0 B                0 B
└ ○ /favicon.ico                           0 B                0 B
+ First Load JS shared by all              77.6 kB
  ├ chunks/769-8ab68f842512379f.js         25.2 kB
  ├ chunks/bce60fc1-49ee79ad31766ac6.js    50.5 kB
  ├ chunks/main-app-c2f0e612a52199d9.js    216 B
  └ chunks/webpack-737972036df45a17.js     1.64 kB

Route (pages)                              Size     First Load JS
─ ○ /404                                   182 B          74.7 kB
+ First Load JS shared by all              74.5 kB
  ├ chunks/framework-8883d1e9be70c3da.js   45 kB
  ├ chunks/main-542a92a81d12d254.js        27.7 kB
  ├ chunks/pages/_app-998b8fceeadee23e.js  195 B
  └ chunks/webpack-737972036df45a17.js     1.64 kB

○  (Static)  automatically rendered as static HTML (uses no initial props)


┌──────────────────────────────┐
│ OpenNext — Generating bundle │
└──────────────────────────────┘

Bundling static assets...
Bundling cache assets...
Bundling server function...
Bundling revalidation function...
Bundling image optimization function...
Bundling warmer function...
+ cd /Users/kevin/Sites/cdk-nextjs-sandbox/test-2/cdk/web/.open-next/server-function
+ zip -ryq1 /var/folders/1b/0m12cbyn1td6lf9_25xr4v5w0000gn/T/nextjs-cdk-build-gUn9Sj/standalone/serverFn.zip .
+ cd /var/folders/1b/0m12cbyn1td6lf9_25xr4v5w0000gn/T/nextjs-cdk-build-gUn9Sj/static
+ zip -ryq1 /var/folders/1b/0m12cbyn1td6lf9_25xr4v5w0000gn/T/nextjs-cdk-build-gUn9Sj/static/assets/assets.zip .
└ Finished preparing NextJS app for deployment
Resources:
  WebBucketE51352C0:
    Type: AWS::S3::Bucket
    Properties:
      Tags:
        - Key: aws-cdk:auto-delete-objects
          Value: "true"
...
...

9. Trying to deploy resulted in bootstrap error (even though I already bootstrapped?)

I’m not sure why, but when I ran cdk deploy I got an error, even though I already ran cdk bootstrap — profile … - running cdk bootstrap again

cdk bootstrap —profile nextjs-cdk-test-deployer did show that one region was bootstrapped, but not another. So perhaps there is some sort of multi-region (something?) that is required by the Next.js cdk setup? Note below “no changes” for one region but not the other.

...
...
CDKToolkit: creating CloudFormation changeset...

 ✨ hotswap deployment skipped - no changes were detected (use --force to override)

 ✅  Environment aws://XX4312/us-east-2 bootstrapped (no changes).
 ✅  Environment aws://XX4312/us-east-1 bootstrapped.
...
...

10. Cdk deploy

cdk deploy CdkStack —profile nextjs-cdk-test-deployer

This “worked”, deploy kicked off (prompted to confirm security changes and such, but kicked off as expected).

Do you wish to deploy these changes (y/n)? y
**CdkStack**: deploying… [1/2]
**CdkStack**: creating CloudFormation changeset…

 ✅  CdkStack

✨  Deployment time: 570.86s

Stack ARN:
arn:aws:cloudformation:us-east-2:XX4312:stack/CdkStack/2ea9c7c0-1054-11ee-b947-06023923a9b7

✨  Total time: 581.29s

11. Login to management console to view domain for CloudFront

I’m not aware of any other way to get the cloud front distribution domain name, so I just logged into the AWS console to get the domain, in my case

12. Visiting domain in a browser results in Next.js’s 500 error page

Note that the favicon.ico request also resulted in a 500

13. Visiting the Lambda console, I see multiple Lambdas were created

15. CloudWatch logs for the “WebServerHandler” show repeated errors

E.g.

2023-06-21T17:04:38.390Z	75882801-8e7c-4100-9c33-6101a6aec3f9	ERROR	{
  clientName: 'S3Client',
  commandName: 'ListObjectsV2Command',
  input: { Bucket: undefined, Prefix: 'ggXh0qs_tsiubcOgG_DEu/index' },
  error: Error: No value provided for input HTTP label: Bucket.
      at _V (/var/task/cache.cjs:15:9782)
      at ase (/var/task/cache.cjs:31:99926)
      at async /var/task/cache.cjs:2:382,
  metadata: undefined
}

The other Lambda logs didn’t seem to have anything that looked like errors

[kevin-mitchell]

Just incase this is useful to anybody, https://github.com/kevin-mitchell/nextjs-cdk-test is exactly what I deployed (though you may need to enter your account number you're deploying to. I actually can probably remove that property but for now it's there due to copy / pasta).

[kevin-mitchell]

I was alerted by somebody in a Discord conversation (thanks Jack M!) that this issue popped up for him with the 3.x release of cdk-nextjs-standalone. Sure enough, when I changed

"cdk-nextjs-standalone": "^3.2.0",

to

"cdk-nextjs-standalone": "^2.0.0",

I was able to deploy the Next.js sample app and everything worked... so what changed? I guess that's the next thing to look at.

[kevin-mitchell]

OK I'm not sure if this is useful or not (it might very well be I'm pulling on the wrong thread here), but

cdk-next calls npx --yes open-next build

open-next then builds a bunch of stuff (where I need to go deeper next), but ultimately the place that the bucket name is used is here: https://github.com/serverless-stack/open-next/blob/45d67314748c0875ad9972203272a3e1328af808/packages/open-next/src/adapters/cache.ts#L81

From what I can tell, that env variable is set outside of open-next because I can't see where it'd be set within the open-next code... Somebody on discord pointed me to https://github.com/serverless-stack/sst/blob/master/packages/sst/src/constructs/NextjsSite.ts#L151. Unfortunately I don't know the intended interaction between NextjsSite and open-next. It's possible NextjsSite is being used somewhere.

[jlegreid]

@kevin-mitchell I've made progress! so the issue stems from the open-next v2 update that just went out on monday, not from cdk-nextjs-standalone. the reason why cdk-nextjs-standalone v2 works is because that was before they implemented open-next. So the fix for now is to just add this line to your Nextjs class in your CDK stack.

      buildCommand: 'npx --yes open-next@1.4.0 build',

In cdk-nextjs-standalone the default build command was npx --yes open-next@latest build so when open-next updated on monday, the next time we built it pulled in those changes, and thats when they implemented the S3Client caching for ISR. Im sure this is something that cdk-nextjs-standalone can support but hasn't implemented yet so for now that S3 bucket is undefined. By locking the open next version to 1.4 we avoid this issue until cdk-nextjs-standalone is ready to support it

[revmischa]

I think we should update CDK-nextjs for the new open-next, not sure what's involved. maybe @khuezy knows

PRs always welcome!

[kevin-mitchell]

@jlegreid this is awesome, thank you!

I'm new to all of this, but it certainly seems like the correct course should be to fix things so that the latest features of open-next and the latest stable release is supported. I'm actually out away from a computer the next week but will try to keep looking into this when I get back in town. Obviously others might get to it first, which is also great of course :)

Thank you!!

[kevin-mitchell]

@jlegreid did you ever make any progress on this? I stepped away (vacation, work work, etc) for a while and am just getting back to this.

I'm new to this entire stack (Next.js and SST most significantly, although I'm not a CDK expert either) so it's a sort of a lot to take in all at once, but putting the pieces together it seems like this issue is very much related to #119 - it's the update in open-next to support ISR that is causing this issue.

Your fix, which makes sense because it's pre-open-next-v2, doesn't seem to work currently because of this issue: sst/open-next#159

Honestly I don't understand how the dependencies play together here because it seems like by specifying an older version of open-next we wouldn't have this issue, but when I attempt to build with open-next@1.4.0 I get

2023-07-26T12:38:04.262Z	undefined	ERROR	Failed to override Next.js require hooks. Error: Cannot find module '/var/task/node_modules/react/jsx-runtime.js'
    at createEsmNotFoundErr (node:internal/modules/cjs/loader:1098:15)
    at finalizeEsmResolution (node:internal/modules/cjs/loader:1091:15)
    at resolveExports (node:internal/modules/cjs/loader:567:14)
    at Module._findPath (node:internal/modules/cjs/loader:636:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1063:27)
    at /var/task/node_modules/next/dist/server/require-hook.js:113:36
    at Function.resolve (node:internal/modules/cjs/helpers:116:19)
    at ie (file:///var/task/index.mjs:4:7059)
    at F (file:///var/task/index.mjs:4:6670)
    at file:///var/task/index.mjs:4:9932 {
  code: 'MODULE_NOT_FOUND',
  path: '/var/task/node_modules/react/package.json'
}

It really seems like it would be ideal to update this package to work with the current version of open-next, i.e. to support ISR and such. But for now, just fixing the above issue would be nice.

[jlegreid]

@kevin-mitchell To be honest I haven't spent any more time on the issue. After specifying the open-next version in my cdk-stack I haven't had any other issues. Unfortunately with our limited resources on our team we probably won't be able to spend more time digging into it as long as this keeps working for us.

[bestickley]

This was resolved in #119. Please re-open if it's still an issue!