[pull] main from activepieces:main

.agents/features/workers.md

@@ -1,16 +1,16 @@
 # Workers Module
 
 ## Summary
-Workers are separate Node processes that poll the app for jobs and execute flows/triggers. *Where* each job's engine step runs is behind a pluggable **Runtime** seam (`AP_RUNTIME`, default `LOCAL` — the original long-lived worker + sandbox model; `GCP_CLOUD_RUN` is a stub). They connect to the app over a Socket.IO channel: on connect a worker fetches its runtime settings (`WorkerSettingsResponse`) and the app registers an RPC server (`WorkerToApiContract`) for that socket. Jobs are pulled by the worker via `poll()` rather than pushed. A worker advertises liveness and config through `MachineInformation` (heartbeat), whose `workerProps` carry its identity including `version`. In the default Docker image both `activepieces-app` and `activepieces-worker` run under PM2 from `WORKDIR /usr/src/app`; `AP_CONTAINER_TYPE` (`APP` / `WORKER` / `WORKER_AND_APP`) selects which start.
+Workers are separate Node processes that poll the app for jobs and execute flows/triggers. **The worker *is* the sandbox**: each worker forks the engine **in-process** (`@activepieces/sandbox`, `createSandboxRuntime`) — there is no separate sandbox pool or Cloud Run runtime indirection. The destination model is one box per worker (`concurrency 1`), scaling out horizontally with replicas (each capped small, e.g. 0.5 vCPU / 1 GB) so an OOM kills one job, not a shared pool; a transitional compatibility mode still honours `AP_WORKER_CONCURRENCY` (N independent boxes). They connect to the app over a Socket.IO channel: on connect a worker fetches its runtime settings (`WorkerSettingsResponse`) and the app registers an RPC server (`WorkerToApiContract`) for that socket. Jobs are pulled by the worker via `poll()` rather than pushed. A worker advertises liveness and config through `MachineInformation` (heartbeat), whose `workerProps` carry its identity including `version`. In the default Docker image both `activepieces-app` and `activepieces-worker` run under PM2 from `WORKDIR /usr/src/app`; `AP_CONTAINER_TYPE` (`APP` / `WORKER` / `WORKER_AND_APP`) selects which start.
 
 ## Key Files
 - `packages/server/api/src/app/workers/machine/machine-controller.ts` — Socket.IO listeners (`FETCH_WORKER_SETTINGS`, `DISCONNECT`); registers the RPC server per connection
 - `packages/server/api/src/app/workers/machine/machine-service.ts` — `onConnection` / `onDisconnect`, `buildSettingsResponse` (emits `APP_VERSION`), worker listing
 - `packages/server/api/src/app/workers/rpc/worker-rpc-service.ts` — `createHandlers()`: `poll` (with version gate), `completeJob`, `extendLock`, progress/log RPCs, `getFlowBundle`, `prepareFlowBundleUpload`, `uploadFlowBundle`
-- `packages/server/worker/src/lib/worker.ts` — worker lifecycle (`worker.start/stop`), `pollAndExecute` loop (with version gate), `getWorkerProps`; holds the selected `Runtime` and drives the per-job lifecycle through it
-- `packages/server/worker/src/lib/runtime/` — the pluggable Runtime seam: `runtime-factory.ts` (picks the impl from `AP_RUNTIME`), `local-runtime.ts` (the `LOCAL` host), `cloud-run-runtime.ts` (the `GCP_CLOUD_RUN` stub), `sandbox-config.ts` (bridges worker settings → `SandboxPoolSettings` + cache base path)
-- `packages/server/sandbox-pool/` — standalone `@activepieces/sandbox-pool` library holding the pool logic, cache/piece installation, and the `Runtime` / `RuntimeExecution` / `ProvisionResult` type contracts (`src/lib/types.ts`)
-- `packages/server/worker/src/lib/config/configs.ts` — worker env vars incl. `AP_RUNTIME` (`WorkerSystemProp.RUNTIME`, default `LOCAL`)
+- `packages/server/worker/src/lib/worker.ts` — worker lifecycle (`worker.start/stop`), `pollAndExecute` loop (with version gate), `getWorkerProps`; builds the `Runtime` via `createSandboxRuntime` and a per-job `Resolver` via `createResolver`, and drives the per-job lifecycle through them
+- `packages/server/worker/src/lib/runtime/sandbox-config.ts` — bridges worker settings → `SandboxSettings` + cache base path (merges the env-only `AP_REUSE_SANDBOX` override into the fetched `WorkerSettings`)
+- `packages/server/sandbox/` — standalone `@activepieces/sandbox` library holding the in-process sandbox (`createSandboxRuntime`, `sandbox-manager`, isolate/fork process makers), cache/piece installation, the worker-side `createResolver`, and the `Runtime` / `Resolver` / `ProvisionInput` type contracts (`src/lib/types.ts`)
+- `packages/server/worker/src/lib/config/configs.ts` — worker env vars incl. `AP_REUSE_SANDBOX` (`WorkerSystemProp.REUSE_SANDBOX`, reuse the engine process between jobs) and `AP_WORKER_CONCURRENCY`
 - `packages/server/worker/src/lib/config/worker-settings.ts` — caches the `WorkerSettingsResponse` fetched on connect
 - `packages/server/utils/src/ap-version.ts` — `apVersionUtil.getCurrentRelease()`; both sides read the deploy-root `package.json` version
 - `packages/core/shared/src/lib/automation/workers/index.ts` — `WorkerProps`, `MachineInformation`, `WorkerSettingsResponse`, `WorkerToApiContract` contracts
@@ -19,8 +19,7 @@ Workers are separate Node processes that poll the app for jobs and execute flows
 - Community / Enterprise / Cloud: all editions run workers; topology differs (embedded `WORKER_AND_APP` for self-host single-container vs dedicated worker fleets on Cloud).
 
 ## Domain Terms
-- **`Runtime` / `RuntimeKind`** — the seam deciding *where* a job's engine step executes. `RuntimeKind` is `LOCAL` (default) or `GCP_CLOUD_RUN` (NOT_IMPLEMENTED stub), selected via `AP_RUNTIME` and resolved by `runtime-factory`. `Runtime` exposes `createExecution` / `getActiveExecutors` / `shutdown`.
-- **`RuntimeExecution`** — the per-job lifecycle the worker drives: **`provision → run → dispose`**. `provision(input)` resolves flow/piece dependencies, reserves the execution slot, and materializes pieces/code into the cache — in local-pool it acquires the sandbox lane and installs host-side into the bind-mounted cache, **atomically**: on unexpected error the lane is released before the promise rejects, so callers only ever guard `run`. It returns a discriminated `ProvisionResult` (`ready` — with the resolved `flowVersion` when a `flow` was passed; `flow-not-found`; `disabled`) rather than throwing for expected missing/disabled-flow cases, and only acquires the slot for `ready`. `run` executes one engine operation (local-pool spawns the sandbox child process, which mounts the now-populated cache). `dispose({ invalidate })` releases the lane for reuse (`false`) or discards it (`true`). Flow/piece resolution (and the missing-flow / disabled handling) now lives **inside `provision()`** in the `@activepieces/sandbox-pool` library, not in a separate worker-side step. `provision` can also return a **bundle hit** from the flow bundle store (`getFlowBundle` RPC): when a locked flow version's bundle is already cached, it supplies the frozen flow definition + piece manifest + compiled code directly, bypassing the per-piece `getPiece` RPC round-trips. Locked flows that change pieces must be re-locked to produce a new bundle; a freshly resolved locked version is published back (best-effort, after execution cache setup). **Bundle transport:** when the `FLOW_BUNDLE` file is S3-backed and `S3_USE_SIGNED_URLS` is on, the bundle bytes never cross the Socket.IO RPC — `getFlowBundle` returns a signed GET URL and `prepareFlowBundleUpload` returns a signed PUT URL, and the worker reads/writes S3 directly via the runtime's built-in `fetch` (`sandbox-pool/src/lib/utils/bundle-http.ts`, dependency-free). S3-backed bundles without signed URLs stream inline `Buffer` over the socket (`uploadFlowBundle`). On **DB-backed storage bundles are not persisted at all** — `prepareFlowBundleUpload` returns `{ kind: 'skip' }` and the worker always builds inline via the legacy resolve path (DB storage gains nothing from bundle caching and a null-data pre-save would throw). Any signed-link or fetch failure likewise degrades to the legacy `getVersion` + `getPiece` resolve path — a bundle never fails a run.
+- **`Resolver` / `Runtime`** — the two roles of the in-process `@activepieces/sandbox` library that the worker drives per job. The **`Resolver`** (worker-side, owns the only `apiClient`) turns a job into a fully-materialized `ProvisionInput`: it resolves the `flowVersion`, piece metadata, and a ready (compiled) flow bundle, disabling the flow on a missing piece. It returns a discriminated `ResolveResult` (`ready` — with the resolved `flowVersion` when a `flow` was passed; `flow-not-found`; `disabled`) rather than throwing for expected missing/disabled-flow cases. The **`Runtime`** (the in-process single sandbox box) exposes `execute` / `getActiveExecutors` / `shutdown` and never reaches the app; `execute` owns the box lifecycle internally — **acquire → provision (materialize pieces/code into the bind-mounted cache) → run one engine operation → release on success / invalidate on throw** — re-raising the sandbox `ActivepiecesError` codes (timeout / memory / log-size) that handlers already catch. The Resolver's bundle path can return a **bundle hit** from the flow bundle store (`getFlowBundle` RPC): when a locked flow version's bundle is already cached, it supplies the frozen flow definition + piece manifest + compiled code directly, bypassing the per-piece `getPiece` RPC round-trips. Locked flows that change pieces must be re-locked to produce a new bundle; a freshly resolved locked version is published back (best-effort, after execution cache setup). **Bundle transport:** when the `FLOW_BUNDLE` file is S3-backed and `S3_USE_SIGNED_URLS` is on, the bundle bytes never cross the Socket.IO RPC — `getFlowBundle` returns a signed GET URL and `prepareFlowBundleUpload` returns a signed PUT URL, and the worker reads/writes S3 directly via the built-in `fetch` (`sandbox/src/lib/utils/bundle-http.ts`, dependency-free). S3-backed bundles without signed URLs stream inline `Buffer` over the socket (`uploadFlowBundle`). On **DB-backed storage bundles are not persisted at all** — `prepareFlowBundleUpload` returns `{ kind: 'skip' }` and the worker always builds inline via the legacy resolve path (DB storage gains nothing from bundle caching and a null-data pre-save would throw). Any signed-link or fetch failure likewise degrades to the legacy `getVersion` + `getPiece` resolve path — a bundle never fails a run.
 - **`WorkerProps`** — typed worker identity sent in every heartbeat (`EXECUTION_MODE`, `WORKER_CONCURRENCY`, `SANDBOX_MEMORY_LIMIT`, `REUSE_SANDBOX`, `version`). Previously a free-form `Record<string,string>`.
 - **`WorkerSettingsResponse`** — runtime config the app hands a worker on connect; now includes `APP_VERSION` (the app's release).
 - **`connectionGeneration`** — worker-side counter bumped on every disconnect; in-flight poll loops exit when their captured generation goes stale, so a reconnect starts fresh loops.

.gcloudignore

@@ -0,0 +1,16 @@
+.git
+.github
+.angular
+.history
+.idea
+.vscode
+*.log
+**/node_modules
+node_modules
+dist
+**/dist
+builds
+deploy
+dev
+docs
+benchmark