chore: easier and simpler deploy

[Stream29]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.68%. Comparing base (b43ebf5) to head (7af395b).
⚠️ Report is 28 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #35708      +/-   ##
==========================================
- Coverage   85.73%   85.68%   -0.05%     
==========================================
  Files        4461     4459       -2     
  Lines      209055   209095      +40     
  Branches    39075    39106      +31     
==========================================
- Hits       179224   179166      -58     
- Misses      26667    26766      +99     
+ Partials     3164     3163       -1     

Flag	Coverage Δ
api	85.08% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[QuantumGhost]

I would suggest:

1. Migrate all default values to the Pydantic configuration fields.
2. Add a Python script to initialize the default configuration file from configuration model definitions.
3. Automatically invoke the script introduced in step 2 on initial deployment to reduce manual works while deploying Dify.

This approach keeps all the information related to configuration (E.G. comment, default value, generation rules for default value) in one piece, and avoid manually synchronizing comments and default value between configuration definitions and default configuration file.

[Stream29]

I agree with @QuantumGhost . Another PR will be added for those changes.

[Stream29]

@QuantumGhost Shall we centralize the configuration definitions from .env.example to code, as the single source of truth? Then we can remove it.

[QuantumGhost]

@Stream29 I would suggest centralizing ALL information related to configuration to code. We may need to introduce special annotations for Pydantic fields such as generation_factory, since the default_factory method for Pydantic is evaluated every time the model is constructed and conflict with required property of setting items.

Currently, we have three pieces of default configuration:

1. The .env.example file docker compose deployment in docker directory.
2. The .env.example file for local development in api directory.
3. The default value in Pydantic configuration definitions.

Both the first one and the second one could be generated from Pydantic model configurations, though generation of the docker compose default configuration would need extra steps since it also includes configuration for other services, such as Postgres, Plugin Daemon and Dify Web.

If we could complete the migration above, then the .env.example is no longer needed and should be removed.

One caveat: the .env.example files in api and docker directories already diverge. Some configuration only exist in one of them and a comparison is need to merge them properly.

[Stream29]

@QuantumGhost
After diving into the codebase, I have new findings.

Migrate all default values to the Pydantic configuration fields.

Some default values have been already moved to Pydantic configuration fields. But there are still some default values meaningful for first-time deployment. For example, database and base url configurations. Keeping these in .env.default is better.

Add a Python script to initialize the default configuration file from configuration model definitions.

it's difficult to initialize the default configuration file from configuration model definitions with a Python script.
Maybe we still need to keep the .env.example and .env.default manually.

[laipz8200]

I think this PR fixes #34321.

[crazywoola]

Most of what the wrapper does can be replaced with native Compose features. The PR's 660 lines of bash/PS reimplement plumbing that Compose 2.20+ already provides.

services:
  api:
    env_file:
      - path: .env.default
        required: true
      - path: .env
        required: false

Ship defaults, user overrides

User just runs `docker compose up -d. .env.default is committed, .env is gitignored.

[QuantumGhost]

@crazywoola Good point.

However, the env_file only supporting exposing environment variables to compose services. For interpolation inside compose file, only .env or files provided by --env-file flag are used.

[QuantumGhost]

The variable substitution logic in this PR introduces maintenance burden. I would propose replace env merge logic with --env-file flag in docker compose command and let docker-compose handle env merging.

[Stream29]

The variable substitution logic in this PR introduces maintenance burden. I would propose replace env merge logic with --env-file flag in docker compose command and let docker-compose handle env merging.

This is will result in incorrect bahavior. For example:

User may cover this variable in .env:

DB_TYPE=postgresql

And this variable in .env.default will stay in a wrong value:

COMPOSE_PROFILES=${VECTOR_STORE:-weaviate},${DB_TYPE:-postgresql}

[crazywoola]

services:
  secret-init:
    image: alpine:3
    command: |
      sh -c '[ -s /secrets/secret_key ] || (head -c 42 /dev/urandom | base64 > /secrets/secret_key)'
    volumes: [secrets:/secrets]

  api:
    environment:
      SECRET_KEY_FILE: /run/secrets/secret_key
    volumes: [secrets:/run/secrets:ro]
    depends_on:
      secret-init: { condition: service_completed_successfully }

volumes: { secrets: }

Document: "set DB_TYPE=mysql and DB_HOST=db_mysql together." That's one extra line in the README vs. 100 lines of awk in the script.

The wrapper's metadata_db_host/metadata_db_port/metadata_db_user switch logic is essentially duplicating profile semantics in shell.

[QuantumGhost]

After some thought, I agree that sticking with Docker Compose is the better solution. For initial configuration generation (e.g., #34321), we could introduce a bootstrap script.

This reduces the maintenance burden and provides a better user experience, as there are far more resources available for Docker Compose than for a home-grown script. Sticking with Docker Compose also gives Dify deployers more flexibility, since they can leverage any mechanism Compose supports (e.g., compose file merging), rather than being forced to read and modify the dify-compose script.

[Stream29]

@QuantumGhost I agree with you. A new PR will be added to fix this experience.

[Stream29]

#35931