fix: ProcessException when deploying server with puppeteer

[Wizzel1]

Description

I am trying to run puppeteer on my dart_frog server.

Steps To Reproduce

1. Create a new dart_frog project
2. Install puppeteer dependency
3. Paste this into your index.dart

import 'package:puppeteer/puppeteer.dart' as puppeteer;

Future<Response> onRequest(RequestContext context) async {
 puppeteer.Browser? browser;
 try {
   browser = await puppeteer.puppeteer.launch(
     headless: true,
   );
 } catch (e) {
   return Response(
     statusCode: HttpStatus.internalServerError,
     body: e.toString(),
   );
 } finally {
   await browser?.close();
 }

5. run dart_frog build
6. deploy to cloud run
7. call your endpoint

Expected Behavior
I expected puppeteer to run.

Screenshots

[felangel]

Hi @Wizzel1 👋
Thanks for opening an issue!

Since puppeteer requires chromium, you’ll need to use a custom Dockerfile which includes chrome.

You can check out https://github.com/buildkite/docker-puppeteer/blob/074354930ee18b76863a98ce636c2027d891c708/Dockerfile#L17 for an example.

Let me know if that helps! 👍

[Wizzel1]

@felangel thanks for another quick response!

I have successfully installed docker and created the Dockerfile. It currently looks like this:

# An example of using a custom Dockerfile with Dart Frog
# Official Dart image: https://hub.docker.com/_/dart
# Specify the Dart SDK base image version using dart:<version> (ex: dart:2.17)
FROM dart:stable AS build

WORKDIR /app

# Resolve app dependencies.
COPY pubspec.* ./
RUN dart pub get

# Copy app source code and AOT compile it.
COPY . .

# Generate a production build.
RUN dart pub global activate dart_frog_cli
RUN dart pub global run dart_frog_cli:dart_frog build
//added this line (copied from your example)
RUN apt-get update && apt-get install -y google-chrome-stable

# Ensure packages are still up-to-date if anything has changed.
RUN dart pub get --offline
RUN dart compile exe build/bin/server.dart -o build/bin/server

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage.
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/build/bin/server /app/bin/
# Uncomment the following line if you are serving static files.
# COPY --from=build /app/build/public /public/

# Start the server.
CMD ["/app/bin/server"]

but when I run docker build . -t dart-frog-app. It throws an error at the new lines:

 => ERROR [build  8/10] RUN apt-get install -y google-chrome-stable                                                                                                                     0.4s 
------
 > [build  8/10] RUN apt-get install -y google-chrome-stable:
#12 0.433 Reading package lists...
#12 0.436 Building dependency tree...
#12 0.436 Reading state information...
#12 0.437 E: Unable to locate package google-chrome-stable

[Wizzel1]

@felangel sorry, I had no idea what docker is up until 30 minutes ago.

If I understand correctly, I need to copy the whole example you sent me and add it to my dockerfile, right?
Like this:

FROM node:14.16.0-buster-slim@sha256:ffc15488e56d99dbc9b90d496aaf47901c6a940c077bc542f675ae351e769a12
RUN  apt-get update \
     && apt-get install -y wget gnupg ca-certificates procps libxss1 \
     && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
     && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
     && apt-get update \
     # We install Chrome to get all the OS level dependencies, but Chrome itself
     # is not actually used as it's packaged in the node puppeteer library.
     # Alternatively, we could could include the entire dep list ourselves
     # (https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix)
     # but that seems too easy to get out of date.
     && apt-get install -y google-chrome-stable \
     && rm -rf /var/lib/apt/lists/* \
     && wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
     && chmod +x /usr/sbin/wait-for-it.sh

# Install Puppeteer under /node_modules so it's available system-wide
ADD package.json package-lock.json /
RUN npm install

# An example of using a custom Dockerfile with Dart Frog
# Official Dart image: https://hub.docker.com/_/dart
# Specify the Dart SDK base image version using dart:<version> (ex: dart:2.17)
FROM dart:stable AS build

WORKDIR /app

# Resolve app dependencies.
COPY pubspec.* ./
RUN dart pub get

# Copy app source code and AOT compile it.
COPY . .

# Generate a production build.
RUN dart pub global activate dart_frog_cli
RUN dart pub global run dart_frog_cli:dart_frog build

# Ensure packages are still up-to-date if anything has changed.
RUN dart pub get --offline
RUN dart compile exe build/bin/server.dart -o build/bin/server

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage.
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/build/bin/server /app/bin/
# Uncomment the following line if you are serving static files.
# COPY --from=build /app/build/public /public/

# Start the server.
CMD ["/app/bin/server"]

[felangel]

@Wizzel1 no worries. Yeah you have the right idea but the Dockerfile you included won't work as it is. The easiest thing would probably be to use something like chrome-alpine with puppeteer:

# An example of using a custom Dockerfile with Dart Frog
# Official Dart image: https://hub.docker.com/_/dart
# Specify the Dart SDK base image version using dart:<version> (ex: dart:2.17)
FROM dart:stable AS build

WORKDIR /app

# Resolve app dependencies.
COPY pubspec.* ./
RUN dart pub get

# Copy app source code and AOT compile it.
COPY . .

# Generate a production build.
RUN dart pub global activate dart_frog_cli
RUN dart pub global run dart_frog_cli:dart_frog build

# Ensure packages are still up-to-date if anything has changed.
RUN dart pub get --offline
RUN dart compile exe build/bin/server.dart -o build/bin/server

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage.
FROM zenika/alpine-chrome:with-puppeteer
COPY --from=build /runtime/ /
COPY --from=build /app/build/bin/server /app/bin/
# Uncomment the following line if you are serving static files.
# COPY --from=build /app/build/public /public/

# Start the server.
CMD ["/app/bin/server"]

I haven't tested this but in theory it should work. Hope that helps 👍

[felangel]

Closing this for now since it's not directly related to Dart Frog but feel free to comment with any follow up questions and I'm happy to continue the conversation 👍

[Wizzel1]

@felangel thanks. I think the docker file creation worked and docker run -i -t -p 8080:8080 dart-frog-app created the output: Running on http://0.0.0.0:8080;

But when I call the endpoint with puppeteer I get the error Exception: Websocket url not found
Can I even use the puppeteer package from pub dev?

I don't quite understand why installing chrome this way is even necessary.

If I understand correctly

# Resolve app dependencies.
COPY pubspec.* ./
RUN dart pub get

installs all dependencies from my pubspec.yaml when "executing" the image. That means that puppeteer also should get installed, no?

When I do that locally, it installs a local chromium instance into the .local-chromium folder.
Why isn't it possible to use the chromium installation that comes with the dart package?

.local-chromium gets installed when first running puppeteer from pub.dev

my pubspec.yaml for reference

[felangel]

@Wizzel1 no problem! I'm not very familiar with the puppeteer package but I'm assuming you need chrome to run puppeteer and you most likely already have it installed on your local machine which is why it works locally but not in the Docker container.

But when I call the endpoint with puppeteer I get the error Exception: Websocket url not found

Can you share the full error and stacktrace?

[Wizzel1]

I'm assuming you need chrome to run puppeteer and you most likely already have it installed on your local machine which is why it works locally but not in the Docker container.

@felangel From my experience puppeteer installs the chromium version it needs automatically, so I just doublechecked that and took a look at the puppeteer repo. There actually is a dockerfile that installs the required chromium version (at least I think it does):
https://github.com/xvrh/puppeteer-dart/blob/master/Dockerfile

So maybe there are some extra steps required to make this package work by itself?

[felangel]

I'm assuming you need chrome to run puppeteer and you most likely already have it installed on your local machine which is why it works locally but not in the Docker container.

@felangel From my experience puppeteer installs the chromium version it needs automatically, so I just doublechecked that and took a look at the puppeteer repo. There actually is a dockerfile that installs the required chromium version (at least I think it does): https://github.com/xvrh/puppeteer-dart/blob/master/Dockerfile

So maybe there are some extra steps required to make this package work by itself?

I would suggest opening an issue on the puppeteer-dart repo so that the maintainers can take a look. This boils down to being able to use their package in a Docker container so if they can provide a vanilla example of that you should be able to adapt it to Dart Frog.

[Wizzel1]

@felangel I had a conversation with the maintainer and got it working. In case you want to try puppeteer on a dart_frog server yourself, I have created an example repo: https://github.com/Wizzel1/dart_frog_puppeteer_example

[felangel]

@felangel I had a conversation with the maintainer and got it working. In case you want to try puppeteer on a dart_frog server yourself, I have created an example repo: https://github.com/Wizzel1/dart_frog_puppeteer_example

That’s awesome! Thanks for sharing 🎉