🐛 Bug Report: Appwrite freezes randomly on function execution

[netapy]

👟 Reproduction steps

I have an Appwrite server on decent hardware (2 core CPU and 4 gigs of RAM).
There are many collections in my database and I use functions to retrieve and edit data before sending it to the client — functions are called very often.

I still haven’t figured out why/when exactly but at some point the server doesn’t serve functions, I can requests accumulating on the functions logs :
(this get_feed function only queries data, compacts it and send it back to the client)

At first the backend interface is still responsive but at some point other functions freeze and i’m unable to load pages or query anything.

The appwrite doctor doesn’t show any issues :

Thanks

👍 Expected behavior

It should not crash as the functions don’t require a lot of memory, and runs without any issues during my stress tests.

👎 Actual Behavior

Appwrite freezes for a while and then comes back after 15-20 minutes.

🎲 Appwrite version

Version 1.0.x

💻 Operating system

Linux

🧱 Your Environment

Only pure Appwrite is running on the server.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[stnguyen90]

@netapy thanks for raising this issue! 🙏🏼 Would you please provide the executor logs and check docker ps -a to see if the runtime container is still running? If it is, the logs from the runtime container can be helpful too.

[netapy]

Sure, as of writing this message the issue is ongoing and this is the log :

TAINER ID   IMAGE                        COMMAND                  CREATED          STATUS                  PORTS                                                                      NAMES
c7140174e711   openruntimes/python:v2-3.9   "/usr/local/src/star…"   17 minutes ago   Up 17 minutes           3000/tcp                                                                   62e0fcfca3b3c5124f16-633d37fa36cb75b79045
af36e0d71f8e   openruntimes/python:v2-3.9   "/usr/local/src/star…"   39 minutes ago   Up 39 minutes           3000/tcp                                                                   62e0fcfca3b3c5124f16-634d2fe7204a7c470294
8150a559904b   openruntimes/python:v2-3.9   "/usr/local/src/star…"   4 hours ago      Up 4 hours              3000/tcp                                                                   62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
f513055af40d   appwrite/appwrite:1.0.3      "worker-functions"       4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-functions
1a3d67b1302a   traefik:2.7                  "/entrypoint.sh --pr…"   4 hours ago      Up 4 hours              0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   appwrite-traefik
49db3fb1a079   appwrite/appwrite:1.0.3      "executor"               4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-executor
ada529598a25   appwrite/appwrite:1.0.3      "realtime"               4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-realtime
cf3ba23161dc   appwrite/appwrite:1.0.3      "maintenance"            4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-maintenance
f2efb28da48b   appwrite/appwrite:1.0.3      "worker-webhooks"        4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-webhooks
1a5f0181a33a   appwrite/appwrite:1.0.3      "worker-databases"       4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-databases
d2ec773125d8   appwrite/appwrite:1.0.3      "worker-mails"           4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-mails
2f219a79b78b   appwrite/appwrite:1.0.3      "worker-builds"          4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-builds
c1c41ef43feb   appwrite/appwrite:1.0.3      "schedule"               4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-schedule
bdd9ea078789   appwrite/appwrite:1.0.3      "worker-certificates"    4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-certificates
7629d292ce88   appwrite/appwrite:1.0.3      "worker-deletes"         4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-deletes
b1e21b3fa110   appwrite/appwrite:1.0.3      "worker-audits"          4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-audits
b2c5a6652e70   appwrite/appwrite:1.0.3      "usage --type=timese…"   4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-usage-timeseries
254e19f06e4b   appwrite/appwrite:1.0.3      "usage --type=databa…"   4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-usage-database
43dbf896b882   appwrite/appwrite:1.0.3      "docker-php-entrypoi…"   4 hours ago      Up 4 hours              80/tcp                                                                     appwrite
a8709093f11a   appwrite/appwrite:1.0.3      "worker-messaging"       4 hours ago      Up 4 hours              80/tcp                                                                     appwrite-worker-messaging
5e8a0ec85c12   redis:7.0.4-alpine           "docker-entrypoint.s…"   4 hours ago      Up 4 hours              6379/tcp                                                                   appwrite-redis
c1ff63fd3336   mariadb:10.7                 "docker-entrypoint.s…"   4 hours ago      Up 4 hours              3306/tcp                                                                   appwrite-mariadb
16bb528d243a   appwrite/influxdb:1.5.0      "/entrypoint.sh infl…"   4 hours ago      Up 4 hours              8086/tcp                                                                   appwrite-influxdb
96701ceb4b35   appwrite/telegraf:1.4.0      "/entrypoint.sh tele…"   4 hours ago      Up 4 hours              8092/udp, 8125/udp, 8094/tcp                                               appwrite-telegraf
dc8df12ede3b   hello-world                  "/hello"                 5 days ago       Exited (0) 5 days ago                                                                              cranky_kepler

[netapy]

And this is the executor log :

Function executed in 0.11850380897522 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.044380903244019 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.11296200752258 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.035964012145996 seconds, status: completed
[Error] Type: Exception
[Error] Message: Runtime not found. Please create the runtime.
[Error] File: /usr/src/code/app/executor.php
[Error] Line: 469
Building container : 62e0fcfca3b3c5124f16-633d37fa36cb75b79045
Build Stage completed in 1 seconds
Executing Runtime: 62e0fcfca3b3c5124f16-633d37fa36cb75b79045
[Error] Type: Exception
[Error] Message: An internal curl error has occurred within the executor! Error Msg: Connection refused
[Error] File: /usr/src/code/app/executor.php
[Error] Line: 541
Executing Runtime: 62e0fcfca3b3c5124f16-633d37fa36cb75b79045
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.12101793289185 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.04660701751709 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.11787605285645 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.040542840957642 seconds, status: completed
[Error] Type: Exception
[Error] Message: An internal curl error has occurred within the executor! Error Msg: Operation timed out
[Error] File: /usr/src/code/app/executor.php
[Error] Line: 544
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.18129801750183 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.22170782089233 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.088666915893555 seconds, status: completed
Function executed in 0.26568603515625 seconds, status: completed
Executing Runtime: 62e0fcfca3b3c5124f16-6347cbfd4d6a53ef838e
Function executed in 0.18561196327209 seconds, status: complet

edit : how can I get the log from the runtime container ?

[stnguyen90]

@netapy, docker logs <container name or id>. The container should be named <project id>-<deployment id>. Are these for when the function was frozen? What was the deployment ID for the frozen function?

[netapy]

Yes these logs were taken when the function was frozen.

It’s weird my containers are able to display logs, except for the two python openruntimes containers.

[stnguyen90]

@netapy, which deployment ID is the frozen one?

[netapy]

@netapy, which deployment ID is the frozen one?

Usually the one ending with 838e freezes first, but then the freeze generalizes to the whole appwrite application.
Also it recovers by itself after 15-20 minutes.

[stnguyen90]

@netapy is this get_feed function being executing asynchronously or synchronously?

[netapy]

@netapy is this get_feed function being executing asynchronously or synchronously?

“get_feed” is launched synchronously, as are all my functions.

[Shiba-Kar]

Message: An internal curl error has occurred within the executor! Error Msg: Operation timed out
check your variables for the function APPWRITE_FUNCTION_ENDPOINT http://<domain>.com/v1 you must have forgotten v1

[netapy]

Message: An internal curl error has occurred within the executor! Error Msg: Operation timed out
check your variables for the function APPWRITE_FUNCTION_ENDPOINT http://<domain>.com/v1 you must have forgotten v1

Thanks for your answer. Unfortunately I re-checked my code and it includes the right variables — my functions run correctly until some point in time (which I haven’t determined yet).

Not sure if this helps but when I docker compose down it seems that my appwrite_runtimes container has trouble shutting down. Logs below :

$ sudo docker compose down 
[+] Running 23/24
 ⠿ Container appwrite-worker-certificates  Removed                                                                                                           11.8s
 ⠿ Container appwrite-worker-deletes       Removed                                                                                                           11.7s
 ⠿ Container appwrite-usage-timeseries     Removed                                                                                                           11.1s
 ⠿ Container appwrite-realtime             Removed                                                                                                           11.2s
 ⠿ Container appwrite-worker-builds        Removed                                                                                                           11.7s
 ⠿ Container appwrite-maintenance          Removed                                                                                                           11.7s
 ⠿ Container appwrite-worker-mails         Removed                                                                                                           11.8s
 ⠿ Container appwrite-schedule             Removed                                                                                                           10.5s
 ⠿ Container appwrite-worker-audits        Removed                                                                                                           11.5s
 ⠿ Container appwrite-worker-messaging     Removed                                                                                                           11.2s
 ⠿ Container appwrite-usage-database       Removed                                                                                                           11.8s
 ⠿ Container appwrite-worker-databases     Removed                                                                                                           11.2s
 ⠿ Container appwrite-traefik              Removed                                                                                                            0.7s
 ⠿ Container appwrite-worker-webhooks      Removed                                                                                                           11.4s
 ⠿ Container appwrite-telegraf             Removed                                                                                                            0.4s
 ⠿ Container appwrite-worker-functions     Removed                                                                                                           11.8s
 ⠿ Container appwrite-executor             Removed                                                                                                           10.3s
 ⠿ Container appwrite                      Removed                                                                                                            0.2s
 ⠿ Container appwrite-mariadb              Removed                                                                                                            0.5s
 ⠿ Container appwrite-redis                Removed                                                                                                            1.1s
 ⠿ Container appwrite-influxdb             Removed                                                                                                            0.2s
 ⠿ Network appwrite_appwrite               Removed                                                                                                            0.1s
 ⠿ Network appwrite_gateway                Removed                                                                                                            0.2s
 ⠿ Network appwrite_runtimes               Error                                                                                                              0.0s
failed to remove network appwrite_runtimes: Error response from daemon: error while removing network: network appwrite_runtimes id a066910e107af1054e6f805b6397f09791183321b302fda52005b78b3d22b565 has active endpoints

[Shiba-Kar]

docker kill appwrite_runtimes