Failed to request plugin daemon (ConnectError) under concurrent browser load only — not reproducible via isolated requests — Dify 1.16.0

[fernanclok]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I upgraded a working self-hosted Dify install from 1.0.0 to 1.16.0 (full DB backup taken beforehand, official upgrade steps followed: git checkout 1.16.0, flask db upgrade ran successfully, Alembic head matches 7a1c2d9e4b60). Since then, the console shows a persistent toast:

"Failed to request plugin daemon, url: plugin/<tenant_id>/management/models"

Same error shape on management/tool, management/agent_strategies, and management/triggers — always the same tenant, and the underlying exception is always:

httpx.ConnectError: [Errno 111] Connection refused
raised from core/plugin/impl/base.py line 270 (_request_with_plugin_daemon_response)

It happens specifically on pages that fire a burst of 4-8 parallel XHR calls on load (e.g. Integrations -> Model Provider). The rest of the app (login, workflows, apps, knowledge base) works fine.

The most important thing I found: I could NOT reproduce this from a separate diagnostic process talking to the same running plugin_daemon container.

• 20 sequential HTTP requests (1/sec) from a fresh python3/httpx process inside the api container: 20/20 succeeded (200).
• 5 bursts of 8 fully parallel requests (ThreadPoolExecutor, same headers/X-Api-Key as the real client) from that same kind of process: 40/40 succeeded (200).
• Meanwhile, real browser traffic through the actual gunicorn+gevent worker(s) kept failing at a high rate (roughly 100-300 errors per 2-5 min of normal console usage) at the exact same time as those successful tests.

This points to something specific to the live gunicorn+gevent worker process handling real traffic (interaction between the persistent module-level pooled httpx.Client in core/plugin/impl/base.py — get_pooled_http_client("plugin_daemon", ...), max_keepalive_connections=50/max_connections=100 — and gevent's worker concurrency), not a general Docker networking/DNS/proxy issue.

2. Additional context or comments

Things I ruled out with evidence before posting:

• Network/DNS: plugin_daemon resolves and is reachable; both containers share docker_default and docker_ssrf_proxy_network.
• Proxy interference: no HTTP_PROXY/HTTPS_PROXY/ALL_PROXY anywhere (host, Docker daemon, containers) — only Dify's own SANDBOX_HTTP_PROXY/SSRF_PROXY_HTTP_URL, unrelated to this client. The pooled client already uses trust_env=False anyway.
• plugin_daemon crash-loop: RestartCount=0, never restarted; CPU/mem nominal (<1%/<300MB); nf_conntrack far below max.
• plugin_daemon cluster membership flap ("node has been removed from the cluster due to being disconnected", ~9-10s after every start): 100% reproducible, but confirmed NOT correlated with the failures (clean successful bursts captured both right after and long after this event).
• PLUGIN_DAEMON_URL/PORT/SERVER_PORT/PLUGIN_DAEMON_KEY: consistent and correct on both sides (http://plugin_daemon:5002, matching key).
• SANDBOX_ENABLE_NETWORK=true: confirmed.
• Outbound internet from plugin_daemon: confirmed working (GET https://github.com -> 200).
• New 1.16.0 env vars / DIFY_AGENT_* / agent_backend: all resolve correctly; agent_backend isn't even in this call's stack trace; api_websocket is intentionally not running (profiles: [collaboration], not used here).
• Gunicorn tuning: tried SERVER_WORKER_AMOUNT 1->4 and SERVER_WORKER_CONNECTIONS 10->100 (restarting nginx immediately after each, to rule out the known stale-upstream-IP gotcha). No change in error rate either way.

Environment:

• langgenius/dify-api:1.16.0
• langgenius/dify-plugin-daemon:0.6.3-local (pinned by the 1.16.0 docker-compose.yaml)
• Ubuntu 24.04.1 LTS, Docker 29.3.1, 8 vCPU / 8GB RAM
• SERVER_WORKER_CLASS=gevent (tried WORKER_AMOUNT both 1 and 4)
• Postgres 15, Redis 6, single-tenant install, one workspace

Related discussions with similar-looking error text but different confirmed root causes (proxy/VPN, crash-looping daemon from empty PLUGIN_S3_USE_AWS, SDK version mismatch) - none of which apply here:
#25822
#21330
#21396
#27666

Happy to provide more logs/traces or run further diagnostics.

[dosubot[bot]]

Hi @fernanclok! I'm Dosu and I'm helping the Dify team.

Your diagnosis is spot-on — this is a httpx.Client + gevent greenlet interaction problem, not a networking issue. Here's what's happening:

The core problem: Dify's plugin daemon client uses a synchronous httpx.Client (via get_pooled_http_client) that is shared across all greenlets in a Gunicorn gevent worker [1]. Under the hood, httpcore's synchronous connection pool does not cooperatively yield to gevent's event loop during I/O operations. When the browser fires 4–8 parallel XHR calls on page load, each greenlet blocks on its httpx call to the plugin daemon, and the connection pool's internal synchronization can starve other greenlets — producing the ConnectError: [Errno 111] Connection refused you're seeing.

Two compounding factors make this worse:

1. threading.Lock in the pool factory — get_pooled_http_client uses a threading.Lock() for synchronization [2]. Gevent monkey-patches threading.Lock to be greenlet-aware, but only if the module is imported after patching. The gunicorn.conf.py explicitly warns about this import-ordering sensitivity [3]. If the lock isn't properly patched, concurrent greenlets can deadlock when trying to acquire it.

2. No retry logic — _request_with_plugin_daemon_response makes a single attempt with no retry, so any transient connection pool contention immediately surfaces as a user-visible error [4].

This explains exactly why your standalone test process (fresh httpx client, no gevent, no shared pool) succeeds 40/40 while the live Gunicorn workers fail — the issue is specific to the interaction between the module-level pooled client and gevent's cooperative scheduling within the worker process.

Why increasing SERVER_WORKER_AMOUNT didn't help: More workers means more processes, each with its own connection pool. But the problem is greenlet contention within a single worker process's shared httpx.Client, so adding workers doesn't resolve it. The SERVER_WORKER_CONNECTIONS=10 default means up to 10 greenlets per worker can compete for the same pool simultaneously [5].

Possible workarounds to try:

• Raise SERVER_WORKER_CONNECTIONS to a higher value (e.g., 100 or 200) — this won't fix the root cause, but may reduce contention pressure by spreading greenlets across more workers if combined with higher SERVER_WORKER_AMOUNT.
• Lower PLUGIN_DAEMON_TIMEOUT from the 600s default to something like 30–60s [6] — this won't prevent the errors but will release stuck greenlets faster.
• Try DEBUG=true temporarily — this runs a single-process gevent.pywsgi server instead of Gunicorn [5], which has different initialization ordering and may behave differently with respect to monkey-patching. Not a production fix, but useful to confirm the diagnosis.

This is a known architectural limitation — the plugin daemon client was designed with synchronous httpx.Client calls that block the calling greenlet for the full duration of the plugin call [6]. A proper fix would likely require either migrating to httpx.AsyncClient, adding explicit gevent.sleep(0) yield points, or implementing retry logic with backoff in the plugin client. You might want to file this as a bug referencing the connection pool factory's threading.Lock usage and the lack of httpcore gevent compatibility.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[fernanclok]

@dosu Thanks for the detailed breakdown — I dug into the actual running code to verify these specific claims before trying anything, and wanted to share what I found, since some of it doesn't hold up and I ran a controlled test that changes the picture:

On the threading.Lock in get_pooled_http_client: confirmed the lock exists (core/helper/http_client_pooling.py), but it's only acquired inside a double-checked-locking block the first time a client is created for a given key. Once the "plugin_daemon" client exists, get_or_create returns early without ever touching the lock again. Since our failures are sustained (still ~100+ errors per 2-5 min, 30-40+ minutes after worker startup), this lock can't be the source of ongoing contention — there's nothing left to contend over after the first call.

On the gunicorn.conf.py import-ordering warning (referencing #26689): this one is real and I confirmed the warning text is exactly as you described, with the same reference to #26689. But I checked our actual startup logs for the MonkeyPatchWarning gevent prints by default when a module got imported before patching — there were none. And the current gunicorn.conf.py already defers the risky part (grpc_gevent.init_gevent()) to the post_patch hook specifically to avoid this, per the fix that came out of #26689. So this specific mechanism doesn't look active in our case.

Controlled test to isolate the variable: to test whether this is specific to gunicorn's fork-then-monkey-patch timing at all, I ran api with DEBUG=true (bypasses gunicorn entirely — exec python -m app, Flask-SocketIO's own gevent.pywsgi server, monkey-patched at the top of app.py before most other imports, no forking). If the gunicorn-timing theory were the cause, this should have looked very different.

It didn't: same error, same rate (~48 Failed to request plugin daemon / ConnectError: Connection refused in 2 minutes under real browser load, comparable to gunicorn with 1 or 4 workers). Reverted back to gunicorn/DEBUG=false afterward since this isn't something to run in production.

So the constant across every configuration we've tried (gunicorn 1 worker, gunicorn 4 workers + higher SERVER_WORKER_CONNECTIONS, and now no-gunicorn/DEBUG=true) is just: gevent + the shared module-level httpx.Client pool in core/plugin/impl/base.py, independent of how/when the process bootstraps or how many workers exist. That points more specifically at httpx/httpcore's synchronous connection pool not cooperating well with gevent's greenlet scheduling under concurrent load, rather than anything about Dify's own worker/startup configuration.

Given this, your suggestion of migrating to httpx.AsyncClient (or adding retry/backoff in _request_with_plugin_daemon_response) seems like the right direction rather than anything tunable via env vars. Happy to help verify a patch against this environment if that'd be useful — I have full logs/traces from all these test runs if a maintainer wants to look closer.

[dosubot[bot]]

Thanks for the thorough follow-up testing, @fernanclok — and you're right to push back on those two points:

1. threading.Lock contention — your observation is correct. The lock in HttpClientPoolFactory uses a double-checked locking pattern and is only acquired on the very first client creation per key [1]. Once the "plugin_daemon" client is cached, subsequent calls return it without touching the lock. This can't explain sustained failures minutes after startup.

2. Gunicorn import-ordering / monkey-patch timing — also fair. The DEBUG=true test is a clean control: app.py calls monkey.patch_all() at the top before most imports, no forking involved, and you still see the same error rate. That rules out Gunicorn's fork-then-patch timing as a contributing factor.

Your conclusion lines up with what the code shows. The constant across all your configurations is the module-level synchronous httpx.Client calling through httpcore's synchronous transport [2]. While gevent monkey-patches select() and socket, httpcore's synchronous connection pool doesn't insert cooperative yield points during I/O — so each greenlet monopolizes the event loop for the full duration of each plugin daemon call. Under a burst of 4–8 parallel browser XHRs, this causes greenlet starvation and the ConnectError you're seeing.

For context on the history: the plugin daemon client was migrated from requests to httpx in October 2025, then connection pooling was centralized into HttpClientPoolFactory in April 2026. Neither of those changes added retry logic or async support. As of now, there are no open PRs proposing a migration to httpx.AsyncClient or adding retry/backoff to _request_with_plugin_daemon_response.

Switching back to requests/urllib3 wouldn't help either — the fundamental issue is synchronous blocking during I/O regardless of HTTP library. The viable fixes would be:

• httpx.AsyncClient with async/await throughout the plugin client path
• Retry with backoff in the plugin client as a resilience layer (doesn't fix the root blocking but would recover from transient pool starvation)
• Manual yield points (e.g., gevent.sleep(0)) injected into the transport layer as a stopgap

Your DEBUG=true test data and the consistent reproduction across worker configurations would be valuable if a maintainer picks this up. This is a solid bug report — the httpx/httpcore gevent incompatibility under concurrent load is well-isolated at this point.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.