Dashboard is non functional

[sria91]

I've tried setting-up open-balena-admin on my local machine using both docker-compose and kubernetes.
But I encounter the same error: "Non-functional dashboard".

It looks like some communication issue between open-balena-postgrest and open-balena-db from browser console error messages.

Following are the screenshots of the chrome devtools window and dashboard for your reference:

Logs from open-balena-postgrest:

07/Jun/2022:15:24:37 +0000: An error ocurred when trying to query database settings for the config parameters
07/Jun/2022:15:24:37 +0000: ConnectionError (Just "could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n")
07/Jun/2022:15:24:37 +0000: Attempting to connect to the database...
07/Jun/2022:15:24:37 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:24:37 +0000: Attempting to reconnect to the database in 0 seconds...
> postgrest-proxy@1.0.0 start
> node postgrest-proxy.js
[HPM] Proxy created: / -> http://127.0.0.1:3000
07/Jun/2022:15:24:38 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:24:38 +0000: Attempting to reconnect to the database in 1 seconds...
07/Jun/2022:15:24:40 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:24:40 +0000: Attempting to reconnect to the database in 2 seconds...
07/Jun/2022:15:24:44 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:24:44 +0000: Attempting to reconnect to the database in 4 seconds...
[HPM] Error occurred while proxying request postgrest.openbalena.abc/user?limit=10&offset=0&order=id.asc to http://127.0.0.1:3000/ [ECONNREFUSED] (https://nodejs.org/api/errors.html#errors_common_system_errors)
07/Jun/2022:15:24:52 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:24:52 +0000: Attempting to reconnect to the database in 8 seconds...
[HPM] Error occurred while proxying request postgrest.openbalena.abc/user?limit=10&offset=0&order=id.asc to http://127.0.0.1:3000/ [ECONNREFUSED] (https://nodejs.org/api/errors.html#errors_common_system_errors)
07/Jun/2022:15:25:08 +0000: {"details":"could not translate host name \"openbalena-db.openbalena\" to address: Name does not resolve\n","code":"","message":"Database connection error. Retrying the connection."}
07/Jun/2022:15:25:08 +0000: Attempting to reconnect to the database in 16 seconds...

Logs from open-balena-db:

2022-06-07 14:49:50.879 UTC [6442] LOG: invalid length of startup packet
...
2022-06-07 15:16:56.159 UTC [6776] LOG: invalid length of startup packet

@dcaputo-harmoni Can you please help me resolve this issue?

[dcaputo-harmoni]

This looks like a docker networking issue - can you ssh into the open-balena-postgrest container and run ping openbalena-db? I suspect it will not resolve, in which case you need to confirm that the open-balena-admin containers are running within the same docker network as open-balena.

You might want to take a look at the services.yaml file, which you'll see puts all of the open-balena-admin containers on the openbalena_default network, which is the default network used by open-balena. If you have changed this, or if you are running your open-balena on a different docker server than open-balena-admin, you'll need to somehow ensure that the open-balena-admin containers can communicate with the open-balena containers.

[sria91]

Thanks for the reply @dcaputo-harmoni. Sorry for the confusion. Your point might be helpful in case I'm running them using docker. But I'm running both open-balena and open-balena-admin using kubernetes, where both of them are running on the same default namespace.

FYI: I'm seeing the following when I do the ping in kubernetes:

root@openbalena-admin-postgrest-557b7fbcfb-c7wg9:/usr/src/app# ping openbalena-db
PING openbalena-db.default.svc.cluster.local (10.107.164.210) 56(84) bytes of data.
^C
--- openbalena-db.default.svc.cluster.local ping statistics ---
80 packets transmitted, 0 received, 100% packet loss, time 80879ms

Following is the output when I do minikube kubectl get services:

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                                                                                                                                                                                      AGE
kubernetes                   ClusterIP      10.96.0.1        <none>           443/TCP                                                                                                                                                                                      4d22h
openbalena-admin-haproxy     LoadBalancer   10.105.171.63    10.105.171.63    80:31659/TCP,443:30041/TCP,10000:32538/TCP,10001:30731/TCP,10002:30684/TCP,10003:32040/TCP,10004:31675/TCP,10005:30778/TCP,10006:32599/TCP,10007:31427/TCP,10008:31011/TCP,10009:31224/TCP   4d21h
openbalena-admin-postgrest   ClusterIP      10.104.204.79    <none>           80/TCP                                                                                                                                                                                       4d21h
openbalena-admin-remote      ClusterIP      10.110.96.146    <none>           80/TCP,10000/TCP,10001/TCP,10002/TCP,10003/TCP,10004/TCP,10005/TCP,10006/TCP,10007/TCP,10008/TCP,10009/TCP                                                                                   4d21h
openbalena-admin-ui          ClusterIP      10.108.227.99    <none>           80/TCP                                                                                                                                                                                       4d21h
openbalena-api               ClusterIP      10.96.16.121     <none>           80/TCP                                                                                                                                                                                       4d22h
openbalena-db                ClusterIP      10.107.164.210   <none>           5432/TCP                                                                                                                                                                                     4d22h
openbalena-haproxy           LoadBalancer   10.107.201.237   10.107.201.237   80:30564/TCP,443:31031/TCP                                                                                                                                                                   4d22h
openbalena-internal-tunnel   ClusterIP      10.103.153.81    <none>           3128/TCP                                                                                                                                                                                     4d22h
openbalena-redis             ClusterIP      10.104.156.196   <none>           6379/TCP                                                                                                                                                                                     4d22h
openbalena-registry          ClusterIP      10.109.61.80     <none>           80/TCP                                                                                                                                                                                       4d22h
openbalena-s3                ClusterIP      10.98.253.0      <none>           80/TCP                                                                                                                                                                                       4d22h
openbalena-vpn               ClusterIP      10.110.117.105   <none>           443/TCP,3128/TCP                                                                                                                                                                             4d22h

As you can see above openbalena-db doesn't have an external ip.

Am I missing any other configuration to get it to work?

[dcaputo-harmoni]

This looks like a namespace issue. Notice the error message is looking for openbalena-db.openbalena but you are using openbalena-db.default. You will either need to change the service config files to point to the default namespace (change references to openbalena-db.default) or put openbalena in the openbalena namespace.

By the way - nice work getting all this up and running in kubernetes.