You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v1.19.0 - Trace Your Workflows, Layer by Layer
#45541
Dify 1.19.0 brings nested workflow tracing to your observability tools. Follow a workflow into the Workflow Tools it calls, inspect the steps inside each invocation, and keep the LLM context alongside the execution tree.
This release also makes Workflow Tools portable with App packages, adds Skill Hub automation, and makes Agent files easier to upload and preview. The bundled plugin daemon is upgraded to 0.6.14-local.
See what happens inside a Workflow Tool, all the way down to its LLM calls.
Building on the Console drilldown introduced in 1.18.0, new external traces now include the persisted execution steps inside called Workflow Tools. Child steps appear beneath the Tool span that invoked them, preserving the hierarchy across nested and repeated calls. (#44673)
From the outer workflow to a nested LLM call: the execution tree and formatted messages, together in Langfuse. Screenshot from #44673.
Follow the hierarchy. Expand called Workflow Tools to see their internal execution steps under the invoking Tool span.
Understand each invocation. Repeated calls and failures retain their place in the trace.
Keep the LLM context. Inspect formatted prompts and responses, token usage, and project, trace, and observation links.
Note
Your saved provider settings continue to work. The richer hierarchy applies to new executions; historical traces are not backfilled. Independently nested workflow runs remain separate traces linked by parent identifiers. For rolling deployments, follow the tracing queue transition below.
Take Workflow Tools with your apps
Export an app and bring the workflows behind its Workflow Tools with it. .ifpkg packages include those workflows at their bound versions, including transitive dependencies. Import reuses tools already in the destination workspace and recreates, publishes, and registers missing ones, remapping their references. (#44592)
All Dify app exports now use package format v2, including apps without Workflow Tools. Existing v1 imports and explicit YAML export remain supported. See the upgrade guide before importing new packages into older installations. (#45353)
Upload together. Preview in place.
Select or drop multiple Agent configuration files and upload them in parallel. Successful uploads are bound in one atomic batch, concurrent Composer saves preserve the latest files, and Prompt uploads keep the cursor position while inserting multiple references. (#44947)
Read PDFs inline, inspect source files with syntax highlighting, and see when text has been truncated. Previews cover configured files, Skill members, and working-directory files, with a download action when a PDF cannot be loaded. (#45032, #45045)
See Agent file uploads and previews
Read a PDF directly in the Agent file preview and switch between project files.
Select several files and upload them together.
Inspect source files with syntax highlighting and line numbers.
Developer spotlight
Manage workspace Skills through OpenAPI and difyctl. Find and inspect Skills, export a draft or published version, import local files, and publish a new version from the command line. (#44941)
The six Skill commands
Command
Capability
difyctl get skill
List Skills, with keyword and tag filters.
difyctl describe skill
Read details, the draft hash, and latest published version.
difyctl get skill version
List published versions.
difyctl export skill
Export the draft or a selected published version as ZIP.
difyctl import skill --files <dir>
Create a Skill or replace an existing Skill's draft from local files.
difyctl publish skill
Publish the draft as a new version.
These operations require an account caller, the corresponding workspace permissions, and Skills to be enabled. Import does not publish. Importing over an existing Skill is classified as destructive; the operation catalog and CLI help distinguish read, write, and destructive operations. (#44941, #45314, #44587)
Upgrade guide
Important
Read the upgrade guide before deploying. Docker Compose now requires v2.35.0 or later. Preserve existing environment files, upgrade the Dify components together, and use the new tracing queue during rolling deployments. New App packages use format v2 and require a compatible destination importer. The bundled plugin daemon moves to 0.6.14-local.
Compatibility and rollout
Back up first. Retain the database, vector-store data, uploaded files, Agent snapshot/workspace storage, deployment secrets, and environment files. Use a maintenance window for a single-stack upgrade.
Upgrade the components together. Use matching 1.19.0 API, worker, web, Agent Backend, and Local sandbox images, with plugin daemon 0.6.14-local. Keep the existing Local sandbox authentication token and any configured PASSWORD_PEPPER unchanged. If upgrading from before 1.18.0, also follow the 1.18.0 upgrade guide.
Use Docker Compose 2.35.0 or later. This is the minimum supported version for root .env overrides to take precedence when Compose resolves service defaults. Check your installed version with docker compose version. (fix(docker): honor categorized service environment overrides #45359)
Keep tracked defaults separate from local overrides. Bundled docker/envs/**/*.defaults files load automatically. Customize a service through its matching local .env file, using the override path documented in the defaults file; root docker/.env holds deployment inputs and explicit overrides. Preserve existing .env files and edit them in place. Review accidentally empty assignments and values pinned by older template copies before upgrading. (fix(docker): honor categorized service environment overrides #45359, refactor(docker): name API and worker configuration explicitly #45376)
Existing shared.env files remain supported. Common API, worker, and Beat customizations now belong in docker/envs/core-services/api-worker.env. Compose still reads legacy docker/envs/core-services/shared.env first, then api-worker.env, so api-worker.env wins when both set conflicting values. Root docker/.env takes precedence over both. An unchanged copy of api-worker.defaults preserves existing shared.env customizations. See the Docker configuration guide. (refactor(docker): name API and worker configuration explicitly #45376)
Separate tracing queues during a rolling upgrade. New tasks default to OPS_TRACE_QUEUE=ops_trace_v2; API, workers, and Beat must agree. Custom CELERY_QUEUES, CELERY_WORKER_QUEUES, or --queues lists must include the new queue. Keep old tracing workers on ops_trace until old producers and all their queued, active, reserved, scheduled, and retry work have finished. Do not purge the old queue or ask the new retired handler to export legacy messages. See the tracing rollout guide. (refactor(tracing): one trace pipeline behind a minimal provider interface #44673)
Deployment instructions · Docker Compose, source, and difyctl
Docker Compose
Once the 1.19.0 release tag and images are published, stop a single-stack deployment gracefully, switch to the release, retain and review configuration, then pull and start the new images:
cd dify/docker
docker compose stop -t -1
git fetch --tags
git checkout 1.19.0
# Merge required configuration changes into existing env files here.
docker compose pull
docker compose up -d
Keep API startup migrations enabled (MIGRATION_ENABLED=true), or apply the full migration chain through your managed migration process before starting new workers. For rolling deployments, follow the separate tracing-queue sequence instead of mixing old and new consumers.
Configure shared plugin credentials, PLUGIN_DAEMON_KEY and PLUGIN_DIFY_INNER_API_KEY, in root docker/.env so the daemon and its clients receive matching values. Ports, volume paths, profiles, and health-check scheduling also remain root configuration inputs. See the Docker configuration guide. (#45359)
Source deployments
Use Python 3.13, Node.js 24.22.1 or later in the 24.x line, and pnpm 12.16.1. pnpm no longer provisions Node.js for the workspace, so install the required Node.js runtime separately. (#44976, #45094)
From the repository root:
uv sync --project api --frozen
uv run --directory api flask db upgrade
pnpm install --frozen-lockfile
Rebuild the web application and restart API, worker, scheduler, and Agent services from the same release. Refresh the standalone Agent Backend's locked server dependencies when deploying it separately.
difyctl
Install the matching 1.19.0 client and use difyctl help to discover the current server catalog. Refresh installed agent skills with difyctl install skills <dir>. Workflow and Chatflow automation should use the revised stop.run operation; message-based apps use stop.task.
Database migrations and existing data
Upgrading from 1.18.0 applies six migration revisions, ending at 53c10126d551. Apply the complete chain through the normal migration runner.
Operational changes to plan for include:
Storage-cleanup indexes (d7e9a1c3f502). Add reference indexes on message_files.upload_file_id, workflow_node_execution_offload.file_id, and workflow_draft_variable_files.upload_file_id. PostgreSQL creates them concurrently, but index construction still scans data and consumes I/O. Online upgrades reuse matching healthy indexes and reject conflicting or invalid definitions. Investigate an interrupted invalid build before retrying; do not drop healthy indexes as routine recovery. (feat: add miss index #44982)
Tracing configuration uniqueness (53c10126d551). Keep one configuration per app/provider and add a unique index. When duplicates exist, the most recently updated row wins, with the ID breaking timestamp ties. Superseded rows are deleted; downgrading the index does not restore them. Retain a database backup. (refactor(tracing): one trace pipeline behind a minimal provider interface #44673)
Package the workflows behind called Workflow Tools, including transitive dependencies, with a default maximum of 20 bundled tool workflows. Missing destination tools are recreated and published; existing tools are reused. Creating missing tools requires tool-management permission and sufficient app quota. Packages needing DSL-version confirmation are rejected in this flow, and apps created before an import failure are cleaned up. (feat(api): bundle called workflow tools in app packages #44592)
Export drafts or published Skill versions, import folders, and publish through account-authenticated workspace operations. An import with a Skill ID replaces that Skill's draft; the imported SKILL.md name must match. Optional draft_hash checks reject stale replacements, but use second-resolution timestamps and do not detect edits made within the same second. (feat(openapi): workspace skill list, export, import and publish ops #44941)
Workspace controls: enabling or disabling provider models requires an owner or administrator. Email-availability checks require account admission. (64fe95667b, 76979a1e24)
Request-scoped cookies: pooled SSRF clients retain response cookies only within the current outbound request and its redirects, preventing later requests from inheriting another request's cookies. (fix(api): keep SSRF client cookies within one request #44672)
Dataset token scope: resource access tokens can list only their explicitly bound active datasets; include_all=true cannot widen that scope. Legacy workspace Dataset API keys retain tenant-wide behavior without inheriting an injected owner's personal visibility filters. (fix(api): enforce dataset service API token scope #44840)
New; defaults to ops_trace_v2. Shared by trace delivery/retries, Agent trace work, payload cleanup, and sandbox metering. API, workers, and Beat must agree.
OPS_TRACE_EXPORT_MAX_ATTEMPTS
New; defaults to 6 total attempts for retryable trace exports.
ENABLE_TRACE_PAYLOAD_SWEEP_TASK
New; defaults to true, enabling daily cleanup of orphaned new-pipeline trace payloads.
APP_PACKAGE_MAX_WORKFLOW_TOOLS
New; defaults to 20 bundled Workflow Tool workflows per App package.
GUNICORN_KEEP_ALIVE
New entrypoint setting; defaults to 10 seconds.
The old OPS_TRACE_UNIFIED_ENABLED, OPS_TRACE_RETRYABLE_DISPATCH_MAX_RETRIES, OPS_TRACE_RETRYABLE_DISPATCH_DELAY_SECONDS, and OPS_TRACE_PARENT_CONTEXT_TTL_SECONDS settings are removed. Existing Agent application-tracing opt-in settings remain supported. (#44673)
For Docker, use envs/core-services/api-worker.defaults as the reference for common API/worker/Beat settings and api-worker.env for new overrides. Existing shared.env remains compatible. Tracked defaults load automatically; root .env and explicit service overrides still take precedence. Review the service-specific references rather than replacing local files with fresh templates. (#45359, #45376)
Endpoint compatibility
Replace deprecated Console profile-update POST calls to /account/name, /account/avatar, /account/interface-language, /account/interface-theme, and /account/timezone with PATCH /console/api/account/profile. Avatar URL reads remain available. (refactor(api): remove legacy account update endpoints #45015)
For Workflow and Chatflow OpenAPI calls, use POST /openapi/v1/apps/{app_id}/runs/{run_id}:stop (stop.run) and GET /openapi/v1/apps/{app_id}/runs/{run_id}/events (get.run.event). Chat, Agent Chat, and Completion retain the task-ID stop route under stop.task. (fix(openapi): stop runs by run id and publish each op's effect #44587)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Dify 1.19.0 brings nested workflow tracing to your observability tools. Follow a workflow into the Workflow Tools it calls, inspect the steps inside each invocation, and keep the LLM context alongside the execution tree.
This release also makes Workflow Tools portable with App packages, adds Skill Hub automation, and makes Agent files easier to upload and preview. The bundled plugin daemon is upgraded to 0.6.14-local.
Release highlights · Upgrade guide · Detailed changes
Release highlights
Nested workflow tracing
See what happens inside a Workflow Tool, all the way down to its LLM calls.
Building on the Console drilldown introduced in 1.18.0, new external traces now include the persisted execution steps inside called Workflow Tools. Child steps appear beneath the Tool span that invoked them, preserving the hierarchy across nested and repeated calls. (#44673)
From the outer workflow to a nested LLM call: the execution tree and formatted messages, together in Langfuse. Screenshot from #44673.
Note
Your saved provider settings continue to work. The richer hierarchy applies to new executions; historical traces are not backfilled. Independently nested workflow runs remain separate traces linked by parent identifiers. For rolling deployments, follow the tracing queue transition below.
Take Workflow Tools with your apps
Export an app and bring the workflows behind its Workflow Tools with it.
.ifpkgpackages include those workflows at their bound versions, including transitive dependencies. Import reuses tools already in the destination workspace and recreates, publishes, and registers missing ones, remapping their references. (#44592)All Dify app exports now use package format v2, including apps without Workflow Tools. Existing v1 imports and explicit YAML export remain supported. See the upgrade guide before importing new packages into older installations. (#45353)
Upload together. Preview in place.
Select or drop multiple Agent configuration files and upload them in parallel. Successful uploads are bound in one atomic batch, concurrent Composer saves preserve the latest files, and Prompt uploads keep the cursor position while inserting multiple references. (#44947)
Read PDFs inline, inspect source files with syntax highlighting, and see when text has been truncated. Previews cover configured files, Skill members, and working-directory files, with a download action when a PDF cannot be loaded. (#45032, #45045)
See Agent file uploads and previews
Read a PDF directly in the Agent file preview and switch between project files.
Select several files and upload them together.
Inspect source files with syntax highlighting and line numbers.
Developer spotlight
Manage workspace Skills through OpenAPI and difyctl. Find and inspect Skills, export a draft or published version, import local files, and publish a new version from the command line. (#44941)
The six Skill commands
difyctl get skilldifyctl describe skilldifyctl get skill versiondifyctl export skilldifyctl import skill --files <dir>difyctl publish skillThese operations require an account caller, the corresponding workspace permissions, and Skills to be enabled. Import does not publish. Importing over an existing Skill is classified as destructive; the operation catalog and CLI help distinguish read, write, and destructive operations. (#44941, #45314, #44587)
Upgrade guide
Important
Read the upgrade guide before deploying. Docker Compose now requires v2.35.0 or later. Preserve existing environment files, upgrade the Dify components together, and use the new tracing queue during rolling deployments. New App packages use format v2 and require a compatible destination importer. The bundled plugin daemon moves to 0.6.14-local.
Compatibility and rollout
1.19.0API, worker, web, Agent Backend, and Local sandbox images, with plugin daemon0.6.14-local. Keep the existing Local sandbox authentication token and any configuredPASSWORD_PEPPERunchanged. If upgrading from before 1.18.0, also follow the 1.18.0 upgrade guide..envoverrides to take precedence when Compose resolves service defaults. Check your installed version withdocker compose version. (fix(docker): honor categorized service environment overrides #45359)docker/envs/**/*.defaultsfiles load automatically. Customize a service through its matching local.envfile, using the override path documented in the defaults file; rootdocker/.envholds deployment inputs and explicit overrides. Preserve existing.envfiles and edit them in place. Review accidentally empty assignments and values pinned by older template copies before upgrading. (fix(docker): honor categorized service environment overrides #45359, refactor(docker): name API and worker configuration explicitly #45376)shared.envfiles remain supported. Common API, worker, and Beat customizations now belong indocker/envs/core-services/api-worker.env. Compose still reads legacydocker/envs/core-services/shared.envfirst, thenapi-worker.env, soapi-worker.envwins when both set conflicting values. Rootdocker/.envtakes precedence over both. An unchanged copy ofapi-worker.defaultspreserves existingshared.envcustomizations. See the Docker configuration guide. (refactor(docker): name API and worker configuration explicitly #45376)OPS_TRACE_QUEUE=ops_trace_v2; API, workers, and Beat must agree. CustomCELERY_QUEUES,CELERY_WORKER_QUEUES, or--queueslists must include the new queue. Keep old tracing workers onops_traceuntil old producers and all their queued, active, reserved, scheduled, and retry work have finished. Do not purge the old queue or ask the new retired handler to export legacy messages. See the tracing rollout guide. (refactor(tracing): one trace pipeline behind a minimal provider interface #44673)Deployment instructions · Docker Compose, source, and difyctl
Docker Compose
Once the 1.19.0 release tag and images are published, stop a single-stack deployment gracefully, switch to the release, retain and review configuration, then pull and start the new images:
Keep API startup migrations enabled (
MIGRATION_ENABLED=true), or apply the full migration chain through your managed migration process before starting new workers. For rolling deployments, follow the separate tracing-queue sequence instead of mixing old and new consumers.Configure shared plugin credentials,
PLUGIN_DAEMON_KEYandPLUGIN_DIFY_INNER_API_KEY, in rootdocker/.envso the daemon and its clients receive matching values. Ports, volume paths, profiles, and health-check scheduling also remain root configuration inputs. See the Docker configuration guide. (#45359)Source deployments
Use Python 3.13, Node.js 24.22.1 or later in the 24.x line, and pnpm 12.16.1. pnpm no longer provisions Node.js for the workspace, so install the required Node.js runtime separately. (#44976, #45094)
From the repository root:
Rebuild the web application and restart API, worker, scheduler, and Agent services from the same release. Refresh the standalone Agent Backend's locked
serverdependencies when deploying it separately.difyctl
Install the matching 1.19.0 client and use
difyctl helpto discover the current server catalog. Refresh installed agent skills withdifyctl install skills <dir>. Workflow and Chatflow automation should use the revisedstop.runoperation; message-based apps usestop.task.Database migrations and existing data
Upgrading from 1.18.0 applies six migration revisions, ending at
53c10126d551. Apply the complete chain through the normal migration runner.Operational changes to plan for include:
d7e9a1c3f502). Add reference indexes onmessage_files.upload_file_id,workflow_node_execution_offload.file_id, andworkflow_draft_variable_files.upload_file_id. PostgreSQL creates them concurrently, but index construction still scans data and consumes I/O. Online upgrades reuse matching healthy indexes and reject conflicting or invalid definitions. Investigate an interrupted invalid build before retrying; do not drop healthy indexes as routine recovery. (feat: add miss index #44982)53c10126d551). Keep one configuration per app/provider and add a unique index. When duplicates exist, the most recently updated row wins, with the ID breaking timestamp ties. Superseded rows are deleted; downgrading the index does not restore them. Retain a database backup. (refactor(tracing): one trace pipeline behind a minimal provider interface #44673)Detailed changes
New capabilities · tracing, packages, and Skills
Tracing and provider integrations
Portable App packages
langgenius/<uuid>plugin dependencies during export. (feat(api): bundle called workflow tools in app packages #44592)Skills and OpenAPI
SKILL.mdname must match. Optionaldraft_hashchecks reject stale replacements, but use second-resolution timestamps and do not detect edits made within the same second. (feat(openapi): workspace skill list, export, import and publish ops #44941)404, and finished or otherwise non-stoppable runs return409. Message-app stopping keeps its task-ID semantics. (fix(openapi): stop runs by run id and publish each op's effect #44587, fix(openapi): mark import.skill as destructive #45314)@dify/openapi-content-mapperpackage for typed API calls and query helpers inferred from OpenAPI sources, with reusable schema/input/output types and opt-in contract-compatible type subpaths. This is development groundwork; these changes do not migrate production Console or Marketplace calls away from their existing clients. (feat(web): set up @dify/openapi-content-mapper foundations #44627, feat(web): export OpenAPI mapper types and compatible contract subpaths #44915)Improvements · editing, accessibility, performance, and operations
.env, leaving optional overrides under operator control. (fix(docker): honor categorized service environment overrides #45359, refactor(docker): name API and worker configuration explicitly #45376)Security enhancements
include_all=truecannot widen that scope. Legacy workspace Dataset API keys retain tenant-wide behavior without inheriting an injected owner's personal visibility filters. (fix(api): enforce dataset service API token scope #44840)Bug fixes · workflows, Agents, knowledge, APIs, and the web app
Workflows and execution
415errors. (fix: honor file input defaults across app execution #44780, fix(web): prevent 415 when saving file input defaults #45042)sys.timestamp, and order weekly schedule previews by date. Webhook query/form numbers retain integer precision and reject non-finite values. (fix(workflow): preserve caller identity for scheduled Agent files #44796, fix(api): read the START node sys.timestamp from the UTC clock #44847, fix(web): order weekly schedule preview by date #44718, fix(trigger): keep webhook form/query integers exact and reject non-finite numbers #44706)Agents, tools, and Skills
Knowledge and document processing
top_k=0, normalize blank legacy retrieval settings on read, and use keyword retrieval for Economy datasets in retrieval previews. (fix: include documents without the metadata key in negative metadata filters #44753, fix(api): honor explicit zero top_k in dataset retrieval paths #42980, fix(api): normalize blank legacy dataset retrieval settings #45020, fix(api): use keyword search for economy datasets in hit testing #44895)APIs, files, and SDKs
blob, and reject incompatible multipart file shapes consistently. (fix: reject 4xx/5xx responses in download_with_size_limit instead of returning the error page #44333, fix(api): only treat Squid-generated 401/403 as SSRF blocks #44841, fix(import): preserve GitHub blob URL path components #44280, fix(api): reject conflicting multipart file shapes #44903)last_idfor conversation pagination while preserving its public method signature. (fix(php-sdk): encode empty app inputs as JSON objects #44675, fix(php-sdk): avoid buffering streaming responses #44271, fix(php-sdk): use last_id for conversation pagination #44659)Web app and workspace
Environment variables and integration changes
OPS_TRACE_QUEUEops_trace_v2. Shared by trace delivery/retries, Agent trace work, payload cleanup, and sandbox metering. API, workers, and Beat must agree.OPS_TRACE_EXPORT_MAX_ATTEMPTS6total attempts for retryable trace exports.ENABLE_TRACE_PAYLOAD_SWEEP_TASKtrue, enabling daily cleanup of orphaned new-pipeline trace payloads.APP_PACKAGE_MAX_WORKFLOW_TOOLS20bundled Workflow Tool workflows per App package.GUNICORN_KEEP_ALIVE10seconds.The old
OPS_TRACE_UNIFIED_ENABLED,OPS_TRACE_RETRYABLE_DISPATCH_MAX_RETRIES,OPS_TRACE_RETRYABLE_DISPATCH_DELAY_SECONDS, andOPS_TRACE_PARENT_CONTEXT_TTL_SECONDSsettings are removed. Existing Agent application-tracing opt-in settings remain supported. (#44673)For Docker, use
envs/core-services/api-worker.defaultsas the reference for common API/worker/Beat settings andapi-worker.envfor new overrides. Existingshared.envremains compatible. Tracked defaults load automatically; root.envand explicit service overrides still take precedence. Review the service-specific references rather than replacing local files with fresh templates. (#45359, #45376)Endpoint compatibility
/account/name,/account/avatar,/account/interface-language,/account/interface-theme, and/account/timezonewithPATCH /console/api/account/profile. Avatar URL reads remain available. (refactor(api): remove legacy account update endpoints #45015)GET /console/api/apps/{app_id}/trace-configmust consumeconfigured_secretsinstead of expecting saved secret values. (refactor(tracing): one trace pipeline behind a minimal provider interface #44673)POST /openapi/v1/apps/{app_id}/runs/{run_id}:stop(stop.run) andGET /openapi/v1/apps/{app_id}/runs/{run_id}/events(get.run.event). Chat, Agent Chat, and Completion retain the task-ID stop route understop.task. (fix(openapi): stop runs by run id and publish each op's effect #44587)status: eligible | not_eligibleresult; only eligible results contain a token. The corresponding internal billing service must support this response contract. (feat(api): return education verification as a status-discriminated result #45165)Full Changelog: 1.18.0 to the 1.19.0
This discussion was created from the release v1.19.0 - Trace Your Workflows, Layer by Layer.
All reactions