Recover from crashed worker

[andig]

2025-09-21T08:13:38Z app[908057d9a02328] fra [info][2025-09-21 08:13:38 +0000] [649] [CRITICAL] WORKER TIMEOUT (pid:7180)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info][2025-09-21 08:13:38 +0000] [7180] [ERROR] Error handling request /optimize/charge-schedule
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]Traceback (most recent call last):
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 135, in handle
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    self.handle_request(listener, req, client, addr)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/sync.py", line 178, in handle_request
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    respiter = self.wsgi(environ, resp.start_response)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/app.py", line 1498, in __call__
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    return self.wsgi_app(environ, start_response)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/app.py", line 1473, in wsgi_app
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    response = self.full_dispatch_request()
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/app.py", line 880, in full_dispatch_request
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    rv = self.dispatch_request()
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/app.py", line 865, in dispatch_request
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/api.py", line 402, in wrapper
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    resp = resource(*args, **kwargs)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/views.py", line 110, in view
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/resource.py", line 41, in dispatch_request
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    resp = meth(*args, **kwargs)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/marshalling.py", line 244, in wrapper
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    resp = f(*args, **kwargs)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/app/app.py", line 171, in post
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    result = optimizer.solve()
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/app/optimizer.py", line 276, in solve
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    self.problem.solve(solver)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    ~~~~~~~~~~~~~~~~~~^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/pulp.py", line 2092, in solve
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    status = solver.actualSolve(self, **kwargs)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/apis/coin_api.py", line 140, in actualSolve
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    return self.solve_CBC(lp, **kwargs)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/apis/coin_api.py", line 218, in solve_CBC
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    if cbc.wait() != 0:
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]       ~~~~~~~~^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 1280, in wait
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    return self._wait(timeout=timeout)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]           ~~~~~~~~~~^^^^^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 2066, in _wait
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    (pid, sts) = self._try_wait(0)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]                 ~~~~~~~~~~~~~~^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 2024, in _try_wait
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    (pid, sts) = os.waitpid(self.pid, wait_flags)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]                 ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    sys.exit(1)
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]    ~~~~~~~~^^^
2025-09-21T08:13:38Z app[908057d9a02328] fra [info]SystemExit: 1
2025-09-21T08:13:38Z app[908057d9a02328] fra [info][2025-09-21 08:13:38 +0000] [7180] [INFO] Worker exiting (pid: 7180)

leads to:

2025-09-21T08:19:25Z app[908057d9a02328] fra [info][2025-09-21 08:19:25 +0000] [649] [ERROR] Worker (pid:7339) was sent SIGKILL! Perhaps out of memory?
2025-09-21T08:19:25Z app[908057d9a02328] fra [info][2025-09-21 08:19:25 +0000] [7364] [INFO] Booting worker with pid: 7364
2025-09-21T08:19:26Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-09-21T08:19:28Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-09-21T08:19:29Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-09-21T08:19:32Z proxy[908057d9a02328] arn [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-09-21T08:20:02Z app[908057d9a02328] fra [info][2025-09-21 08:20:02 +0000] [649] [CRITICAL] WORKER TIMEOUT (pid:7336)
2025-09-21T08:20:04Z app[908057d9a02328] fra [info][2025-09-21 08:20:04 +0000] [649] [ERROR] Worker (pid:7336) was sent SIGKILL! Perhaps out of memory?
2025-09-21T08:20:04Z app[908057d9a02328] fra [info][2025-09-21 08:20:04 +0000] [7369] [INFO] Booting worker with pid: 7369

Something seems to eat all resources until the machine is exhausted.

[andig]

Speicehr erschöpft?

root@908057d9a02328:/bin# cat /proc/meminfo
MemTotal:         469760 kB
MemFree:           13628 kB
MemAvailable:      48540 kB
Buffers:            2428 kB
Cached:            58500 kB
SwapCached:            0 kB
Active:            28792 kB
Inactive:         334928 kB
Active(anon):        884 kB
Inactive(anon):   324276 kB
Active(file):      27908 kB
Inactive(file):    10652 kB
Unevictable:        3072 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               216 kB
Writeback:             0 kB
AnonPages:        305884 kB
Mapped:            30960 kB
Shmem:             22392 kB
KReclaimable:       8732 kB
Slab:              31288 kB
SReclaimable:       8732 kB
SUnreclaim:        22556 kB
KernelStack:        1792 kB
PageTables:         2504 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      234880 kB
Committed_AS:     687168 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        6044 kB
VmallocChunk:          0 kB
Percpu:              536 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:       18432 kB
DirectMap2M:      505856 kB
DirectMap1G:           0 kB

[lfloeer]

Hast du von der Instanz Langzeitstatistiken was freien Speicher angeht? Da könnte man sehen, ob das ein memory leak ist oder einfach ein Solver zu viel RAM braucht. Für Memleaks hat gunicorn so eine Funktion um Worker nach ein paar requests neu zu starten.

https://docs.gunicorn.org/en/stable/settings.html#max-requests

[andig]

Leider nein. Bei Go wüsste ich wir ich die in Prometheus bekomme, bei Python is mir das nicht klar? Aber max-requests klingt gut, der Neustart kostet ja nicht viel.

[andig]

Ich finde das hier: https://pypi.org/project/gunicorn-prometheus-exporter/, hab aber keine Ahnung, wie das zusammen zu knoten wäre?

[lfloeer]

Ich hatte mehr an Statistiken auf Host-Ebene gedacht. Wenn du Prometheus verwendest, sollte man da ja auch Metriken der VM bekommen, auf der Docker läuft.

[lfloeer]

Hostest du das auf fly.io? Ich hab da keine Erfahrung mit, hab aber das hier gefunden, was ganz sinnvoll klingt

https://fly.io/docs/monitoring/metrics/#instance-memory-fly_instance_memory_

[andig]

curl https://api.fly.io/prometheus/evcc/ \
--header "Authorization: FlyV1 ..." \
--header 'Authorization: Bearer ...'
404 page not found

[lfloeer]

Was ist mit der Grafana Instanz unter fly-metrics.net? Funktioniert die nicht?

https://fly-metrics.net/

[andig]

Alles flat:

[lfloeer]

Dann ist’s wohl eher kein schleichender Memory Leak. Man kann in den Logs glaube ich auch erkennen, dass in der Tat sofort neue worker gestartet werden, dann aber wieder ein worker gekillt wird (die PIDs sind immer andere).

Wie sieht es denn aus mit den Anzahl an Verbindungen und aktiven requests? Nicht, dass das nachher an einem Request liegt, der jede Instanz direkt crashed (warum auch immer) und der Client den immer wieder schickt und so nach und nach alle Instanzen abwechselnd killt… ist aber nur ne Vermutung.

[lfloeer]

Ich hatte die Tage immer mal wieder 502/503er auf dem Service. Sind das wieder die oben beschriebenen Fehler oder ist es was neues?

[andig]

I kanns nicht mit Sicherheit sagen. Maschine ist jetzt (15m Slots gibts mehrere) auf 4 CPU 1 GB upgraded.

[andig]

Just noticed machine at 100% CPU over all cores and load balancing failing. htop shows a number of running CBC processes. I couldn't figure out immediately if there were any old processes or all recently started. Wondering if we can limit CBC runtime somehow?

2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/api.py", line 402, in wrapper
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    resp = resource(*args, **kwargs)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask/views.py", line 110, in view
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/resource.py", line 41, in dispatch_request
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    resp = meth(*args, **kwargs)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/flask_restx/marshalling.py", line 244, in wrapper
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    resp = f(*args, **kwargs)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/app/app.py", line 174, in post
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    result = optimizer.solve()
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/app/optimizer.py", line 319, in solve
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    self.problem.solve(solver)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    ~~~~~~~~~~~~~~~~~~^^^^^^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/pulp.py", line 2092, in solve
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    status = solver.actualSolve(self, **kwargs)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/apis/coin_api.py", line 140, in actualSolve
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    return self.solve_CBC(lp, **kwargs)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/pulp/apis/coin_api.py", line 218, in solve_CBC
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    if cbc.wait() != 0:
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]       ~~~~~~~~^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 1280, in wait
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    return self._wait(timeout=timeout)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]           ~~~~~~~~~~^^^^^^^^^^^^^^^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 2066, in _wait
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    (pid, sts) = self._try_wait(0)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]                 ~~~~~~~~~~~~~~^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/subprocess.py", line 2024, in _try_wait
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    (pid, sts) = os.waitpid(self.pid, wait_flags)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]                 ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]  File "/usr/local/lib/python3.13/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    sys.exit(1)
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]    ~~~~~~~~^^^
2025-10-08T08:28:47Z app[908057d9a02328] fra [info]SystemExit: 1
2025-10-08T08:28:48Z app[908057d9a02328] fra [info][2025-10-08 08:28:47 +0000] [29629] [INFO] Worker exiting (pid: 29629)
2025-10-08T08:28:48Z app[908057d9a02328] fra [info]subject: czam70
2025-10-08T08:28:48Z app[908057d9a02328] fra [info]subject: bongoenze
2025-10-08T08:28:48Z app[908057d9a02328] fra [info][2025-10-08 08:28:48 +0000] [671] [ERROR] Worker (pid:29629) was sent SIGKILL! Perhaps out of memory?
2025-10-08T08:28:48Z app[908057d9a02328] fra [info][2025-10-08 08:28:48 +0000] [29651] [INFO] Booting worker with pid: 29651
2025-10-08T08:28:49Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:49Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:51Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:51Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:51Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:51Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:51Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy ams [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:52Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:28:53Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:53Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:53Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:55Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:55Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:55Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:28:56Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:56Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:56Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:56Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:28:56Z proxy cdg [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:28:59Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:28:59Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:28:59Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:29:00Z proxy cdg [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:01Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:29:02Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:02Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:03Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:05Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:05Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:05Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:06Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:07Z proxy[908057d9a02328] fra [error][PC05] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2025-10-08T08:29:08Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:11Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:13Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:14Z proxy fra [error][PR03] could not find a good candidate within 20 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the 'immediate' strategy? have your app's instances all reached their hard limit?)
2025-10-08T08:29:16Z proxy cdg [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:16Z proxy cdg [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:16Z proxy cdg [error][PR04] could not find a good candidate within 20 attempts at load balancing
2025-10-08T08:29:16Z proxy[908057d9a02328] fra [error][PR04] could not find a good candidate within 20 attempts at load balancing

[lfloeer]

The code uses the COIN_CBC_CMD, which has an option to set the number of threads. So I guess setting this to 1 or 2 should help here?

https://coin-or.github.io/pulp/technical/solvers.html#pulp.apis.PULP_CBC_CMD