Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next JS with Sharp in docker container #3877

Closed
3 tasks done
albinlang opened this issue Dec 3, 2023 · 17 comments
Closed
3 tasks done

Next JS with Sharp in docker container #3877

albinlang opened this issue Dec 3, 2023 · 17 comments

Comments

@albinlang
Copy link

Possible bug

Is this a possible bug in a feature of sharp, unrelated to installation?

  • Running npm install sharp completes without error.
  • Running node -e "require('sharp')" completes without error.

If you cannot confirm both of these, please open an installation issue instead.

Are you using the latest version of sharp?

  • I am using the latest version of sharp as reported by npm view sharp dist-tags.latest.

If you cannot confirm this, please upgrade to the latest version and try again before opening an issue.

If you are using another package which depends on a version of sharp that is not the latest, please open an issue against that package instead.

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

System:
OS: Windows 11 10.0.22631
CPU: (16) x64 AMD Ryzen 9 4900HS with Radeon Graphics
Memory: 10.61 GB / 31.42 GB
Binaries:
Node: 18.18.2 - C:\Program Files\nodejs\node.EXE
npm: 9.8.1 - C:\Program Files\nodejs\npm.CMD
npmPackages:
sharp: ^0.33.0 => 0.33.0

The issue

I'm trying to use sharp in my conatainer with Next JS:

My Dockerfile:

FROM node:18-alpine AS base

# Install dependencies only when needed
FROM base AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
WORKDIR /app

# Install dependencies based on the preferred package manager
COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* ./
RUN \
  if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
  elif [ -f package-lock.json ]; then npm ci; \
  elif [ -f pnpm-lock.yaml ]; then yarn global add pnpm && pnpm i --frozen-lockfile; \
  else echo "Lockfile not found." && exit 1; \
  fi

# Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules

COPY . .

# Next.js collects completely anonymous telemetry data about general usage.
# Learn more here: https://nextjs.org/telemetry
# Uncomment the following line in case you want to disable telemetry during the build.
ENV NEXT_TELEMETRY_DISABLED 1

RUN yarn build

# Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app

ENV NODE_ENV production
# Uncomment the following line in case you want to disable telemetry during runtime.
ENV NEXT_TELEMETRY_DISABLED 1


RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/public ./public

# Set the correct permission for prerender cache
RUN mkdir .next
RUN chown nextjs:nodejs .next

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

USER nextjs

EXPOSE 3000

ENV PORT 3000
# set hostname to localhost
ENV HOSTNAME "0.0.0.0"

# server.js is created by next build from the standalone output
# https://nextjs.org/docs/pages/api-reference/next-config-js/output
CMD ["node", "server.js"]

My Package.json

{
  "name": "historiska-vandringar",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "classnames": "^2.3.2",
    "clsx": "^2.0.0",
    "embla-carousel-react": "^8.0.0-rc15",
    "mapbox-gl": "^2.15.0",
    "next": "13.5.6",
    "react": "^18",
    "react-dom": "^18",
    "react-map-gl": "^7.1.6",
    "sharp": "^0.33.0",
    "tailwind-merge": "^2.0.0",
    "zustand": "^4.4.3"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "autoprefixer": "^10",
    "eslint": "^8",
    "eslint-config-next": "13.5.6",
    "postcss": "^8",
    "tailwindcss": "^3",
    "typescript": "^5"
  }
}

After running my container on this image I get the following error:

Error: 'sharp' is required to be installed in standalone mode for the image optimization to function correctly. Read more at: https://nextjs.org/docs/messages/sharp-missing-in-production

@lovell
Copy link
Owner

lovell commented Dec 3, 2023

It looks like you're using yarn v1:

$ docker run -it --rm node:18-alpine yarn --version
1.22.19

Please see #3871

@albinlang
Copy link
Author

@lovell Thank you for a quick response. However I'm not sure how to solve my problem. Would you mind explaining how I can solve this issue in my project? In my mind I use NPM and not yarn except in my Dockerfile on line 30: "RUN yarn build". I've tried replacing this with "RUN npm run build" but I still have the error.

@lovell
Copy link
Owner

lovell commented Dec 3, 2023

Make sure you're not copying node_modules into the container from outside. Perhaps you could simplify this Dockerfile so it can only use npm e.g. by removing all reference to yarn and pnpm lockfiles.

@albinlang
Copy link
Author

I replaced:

COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* ./
RUN \
  if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
  elif [ -f package-lock.json ]; then npm ci; \
  elif [ -f pnpm-lock.yaml ]; then yarn global add pnpm && pnpm i --frozen-lockfile; \
  else echo "Lockfile not found." && exit 1; \
  fi

with this:

COPY package.json package-lock.json* ./
RUN npm ci

"Make sure you're not copying node_modules into the container from outside."
What do I have to change and to what in order to do this?

@lovell
Copy link
Owner

lovell commented Dec 3, 2023

The COPY . . directive looks like it might be copying files into the container from outside.

These questions relate to Docker and Node.js rather than being specific to sharp, so might be more suitable for a more general site such as StackOverflow.

@albinlang
Copy link
Author

@lovell It's from Next JS example Dockerfile.

@albinlang
Copy link
Author

However... locking sharp to version 0.32.6 seems to have solved the problem.

@lovell
Copy link
Owner

lovell commented Dec 3, 2023

I tried running https://github.com/vercel/next.js/tree/canary/examples/with-docker locally and can reproduce - the problem is due to vercel/nft#371

@albinlang
Copy link
Author

I see... so it wasn't just me then. I'm somewhat pleased to hear that 😅 Would you say sticking to version 0.32.6 for now is a stable choice?

@lovell
Copy link
Owner

lovell commented Dec 4, 2023

The docs you need are here - https://nextjs.org/docs/pages/api-reference/next-config-js/output

Either switch off tracing or specify outputFileTracingIncludes - see #3870 (comment)

I'll close this as there's already one sharp issue tracking this and there's nothing else we can do.

@lovell lovell closed this as completed Dec 4, 2023
@99lalo
Copy link

99lalo commented Dec 11, 2023

However... locking sharp to version 0.32.6 seems to have solved the problem.

Where did you install sharp? I tried locking it on dependencies and i'm still getting the same issues on my container. I'm also using the nextjs official dockerfile example.

FROM node:20.10-alpine AS base

# Install dependencies only when needed
FROM base AS deps
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
WORKDIR /app

# Install dependencies based on the preferred package manager
COPY package.json package-lock.json* ./
RUN npm ci

# Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=deps /root/.npm /root/.npm
COPY . .

# Generate Prisma Files
RUN npx prisma generate

ENV NEXT_SHARP_PATH=/tmp/node_modules/sharp

RUN npm run build

# Production image, copy all the files and run next
FROM base AS runner
WORKDIR /app

ENV NODE_ENV production
ENV NEXT_SHARP_PATH=/tmp/node_modules/sharp

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/public ./public

# Set the correct permission for prerender cache
RUN mkdir .next
RUN chown nextjs:nodejs .next

# Automatically leverage output traces to reduce image size
# https://nextjs.org/docs/advanced-features/output-file-tracing
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/prisma ./prisma

USER nextjs

EXPOSE 3000

ENV PORT 3000
# set hostname to localhost
ENV HOSTNAME "0.0.0.0"

# server.js is created by next build from the standalone output
# https://nextjs.org/docs/pages/api-reference/next-config-js/output
CMD ["npm", "run", "start:migrate:prod"]

Package.json

{
  "name": "gamefinder",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "start:migrate:prod": "npx prisma migrate deploy && node server.js",
    "lint": "next lint",
    "migrate": "npx prisma migrate",
    "migrate:deploy": "npx prisma migrate",
    "prisma:seed": "npx prisma db seed",
    "docker:compose:up": "docker-compose up",
    "docker:compose:down": "docker-compose down",
    "docker:compose:restart": "npm run docker:compose:down && npm run docker:compose:up"
  },
  "prisma": {
    "seed": "ts-node --compiler-options {\"module\":\"CommonJS\"} prisma/seed.ts"
  },
  "engines": {
    "node": "20.10"
  },
  "dependencies": {
    "@auth/prisma-adapter": "^1.0.4",
    "@emotion/react": "^11.11.1",
    "@emotion/styled": "^11.11.0",
    "@mui/icons-material": "^5.14.14",
    "@mui/material": "^5.14.14",
    "@prisma/client": "^5.7.0",
    "axios": "^1.6.1",
    "bcrypt": "^5.1.1",
    "moment": "^2.29.4",
    "next": "^14.0.2",
    "next-auth": "^4.24.5",
    "react": "^18",
    "react-dom": "^18",
    "react-slick": "^0.29.0",
    "sharp": "0.32.6",
    "slick-carousel": "^1.8.1"
  },
  "devDependencies": {
    "@types/bcrypt": "^5.0.2",
    "@types/node": "^20.9.0",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "@types/react-slick": "^0.23.12",
    "eslint": "^8",
    "eslint-config-next": "13.5.6",
    "prisma": "^5.7.0",
    "sass": "^1.69.4",
    "ts-node": "^10.9.1",
    "typescript": "^5.2.2"
  }
}

@david7378

This comment was marked as off-topic.

@deadcoder0904
Copy link

@99lalo did you solve this problem?

i'm kinda doing the same thing & have already installed sharp in package.json but docker container can't find it. it is probably bcz my node_modules in production doesn't include sharp as it strips imports that are not used.

@b1rdex
Copy link

b1rdex commented Feb 28, 2024

@deadcoder0904

# using numeric uid/gid becuase of buildx bug (chmod+link can't use names for chmod): https://github.com/docker/buildx/issues/1408
COPY --from=builder --chown=1001:1001 --link /app/.next/standalone ./

# workaround for next.js standalone build bug
COPY --from=builder --chown=1001:1001 --link /app/node_modules/@img ./node_modules/@img
COPY --from=uploader --chown=1001:1001 --link /app/.next/static ./.next/static

So basically you just need to manually copy node_modules/@img to the docker image

@deadcoder0904

This comment was marked as off-topic.

@srigi
Copy link

srigi commented May 1, 2024

I found a root cause in my case: It was a discrepancy between dependencies of @img on my developer machine (macOS - darwin) and in the container (node:20-alpine - linux).

I'm kinda surprised that no one mentioned the binary compatibility of dependencies. By looking at Sharp's package.json the error starts becoming evident. My developer workflow persisted the darwin variants of @img into the lock file. What I need for container is a linux and linuxmusl.

The fix is super easy: do the proper dependencies installation inside the Docker with the target platform. Today's package managers are clever enough to install the correct binary variant, even if the lock file is followed.

@lovell
Copy link
Owner

lovell commented May 1, 2024

@srigi

My developer workflow persisted the darwin variants of @img into the lock file. What I need for container is...

If you're using npm this is probably due to npm/cli#4828

Please see https://sharp.pixelplumbing.com/install#cross-platform for more information if you are attempting to share lockfiles and node_modules directories across multiple platforms.

For anyone else reading this and still having problems, please note there were bugs in previous versions of Next.js relating to its "standalone" build feature that prevented some dependencies from being included. Please ensure you are using the latest version of all your dependencies, including both Next.js and sharp. Finally please avoid yarn v1 and its lockfiles.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants