renderJs: false appears to still use Lightpanda when Lightpanda is configured

[Antardas]

Summary

When a Lightpanda container is configured in Docker Compose, renderJs: false appears to produce the same result as renderJs: true.

Based on my understanding, I would expect renderJs: false to skip JavaScript rendering entirely and return the raw HTML result.

Environment

• Self-hosted FastCRW
• Docker Compose with a Lightpanda container

Request

curl -X POST http://localhost:4050/v2/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://ahbab.com.bd/product/creative-wooden-gun-shaped-mini-ballpoint-pen-05-solid-wood-metal-pen-body-color-pear-wood-rbo2pp",
    "renderJs": false,
    "formats":["markdown"],
    "onlyMainContent": true,
    "waitFor": 2500
  }'

I also tested the same request with:

"renderJs": true

Docker Compose

version: "3.9"

services:
  crw:
    image: ghcr.io/us/crw:0.26.0
    container_name: crw
    restart: unless-stopped

    ports:
      - "4050:3000"
    read_only: true

    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    mem_limit: 2g
    memswap_limit: 2g

    pids_limit: 512
    environment:
      - CRW_RENDERER__LIGHTPANDA__WS_URL=ws://lightpanda:9222

  lightpanda:
    image: lightpanda/browser:nightly
    restart: unless-stopped
    command:
      - lightpanda
      - serve
      - --host
      - 0.0.0.0
      - --port
      - "9222"

Observed behavior

When the Lightpanda container is running:

• renderJs: false returns the same output as renderJs: true.
• The response appears to contain JavaScript-rendered content.

When I stop/remove the Lightpanda container:

• renderJs: false returns the expected non-rendered result.

This makes it seem like requests may still be going through Lightpanda even when renderJs is set to false.

Expected behavior

I would expect:

• renderJs: false → Never use Lightpanda. Return the raw/non-JavaScript-rendered page.
• renderJs: true → Use Lightpanda and return the JavaScript-rendered page.

Question

Is this the intended behavior, or is renderJs: false expected to completely bypass the Lightpanda renderer when it is available?

[us]

thanks for the detailed report, the docker compose and both request variants made this quick to pin down.

you are right, and it is a bug on our side. renderJs is a real parameter on our native /v1/scrape, but the /v2/scrape request struct (the firecrawl-compatible surface) never declared it. that surface is intentionally lenient about unknown fields so a newer sdk build does not 400, which means your "renderJs": false was accepted and then silently dropped. the request reached the engine with no rendering preference at all, so it took the default auto path, and with a lightpanda container configured auto escalates into it. that also explains why true and false looked identical, and why removing the container "fixed" it.

to answer your question directly: renderJs: false is meant to do exactly what you expected, never touch a browser tier and return the raw html. it just was not wired on /v2.

two things you can do right now while we ship the fix:

• call /v1/scrape instead, same body, renderJs is honored there
• or leave the lightpanda container out of the compose file for the workloads you want raw

looking into it now, will link the pr here.

[us]

pr is up: #349

the fix threads renderJs through the v2 wire structs, so it now works on /v2/scrape, /firecrawl/v2/scrape, /v2/batch/scrape and scrapeOptions.renderJs on /v2/crawl.

verified it against a lightpanda container set up like yours, by counting how many requests actually reached the origin. renderJs:false on v2 went from 2 hits (http fetch + lightpanda) down to 1, while renderJs:true stays at 2 and v1 is unchanged. so it really does skip the browser now rather than just looking like it.

thanks again for the clean report, the compose file and testing both values is what made this quick to pin down. note the engine ships docker images on releases rather than on merge, so this lands in the next release rather than immediately.

[us]

merged in #349.

small correction to what i said above: i told you this would only ship in the next release, but that was wrong. a green build on main bumps the engine pin in the deploy repo automatically, so it goes out to the managed cloud without waiting for a tagged release. self-hosted docker images still follow releases, so if you are pinning ghcr.io/us/crw:0.26.0 you will want the next tag.

thanks again for the report.

[us]

fixed in #349, merged and live on the managed cloud (self-host follows the next release tag). verified on prod: v2 renderJs:false dropped from 2 origin hits to 1 while renderJs:true stays at 2.

[Antardas]

Thanks @us for the quick investigation and fix! The explanation of the root cause was very helpful, and I appreciate the fast turnaround. 🥰