mariadb-galera: wsrep_node_address not correctly set in dual-stack environment

[sptmhall]

mariadb-galera 4.4.5

By default the wsrep_node_address is obtained by running hostname -i. In dual-stack environments this returns 2 IP Addresses seperated by a space. Using this value results in a "Too many arguments" error.

I tried overriding the wsrep_node_address by setting the "DB_GALERA_NODE_ADDRESS" env variable using "staus.podIP" (which returns only the IPv4 Address). And verified the environment variable was properly set in the running container, however it appears as if the check for the override does not work.

I was only able to get the pods to start by copying /opt/bitnami/scripts/libmariadbgalera.sh out of the container, modifying line 631 from "hostname -i" to "hostname -i | sed 's/.* //'", creating a configmap of the modified file, and mounting it into the pod using extraVolumes/extraVolumeMounts. Note: this is probably not a good solution, it was a quick hack that works in my environment. I am not confident that the IPv4 address will always show up as the second address when running hostname -i.

/opt/bitnami/scripts/libmariadbgalera.sh

<snip>
get_node_address() {
    if [[ -n "$DB_GALERA_NODE_ADDRESS" ]]; then
        echo "$DB_GALERA_NODE_ADDRESS"
    else
        # In some environments, the network may not be fully set up when starting the initialization
        # So, to avoid issues, we retry the 'hostname' command until it succeeds (for a few minutes)
        local -r retries="60"
        local -r seconds="5"
        retry_while "hostname -i" "$retries" "$seconds" >/dev/null
        # hostname -i
        hostname -i | sed 's/.* //'
    fi
}
<snip>

These are the errors from the log

2020-10-21 21:34:15 0 [Note] InnoDB: 10.5.6 started; log sequence number 45091; transaction id 21
2020-10-21 21:34:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-10-21 21:34:15 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
/opt/bitnami/mariadb/sbin/mysqld: Too many arguments (first extra is '172.26.215.106').
2020-10-21 21:34:15 0 [ERROR] Aborting
2020-10-21 21:34:15 2 [ERROR] WSREP: Exception: State wait was interrupted
2020-10-21 21:34:15 2 [ERROR] WSREP: View callback failed. This is unrecoverable, restart required. (FATAL)
	 at galera/src/replicator_smm.cpp:submit_view_info():2431
201021 21:34:15 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

This is the running mysqld process, you can see both IP Addresses and the reason it was reported as a "too many arguments" error:

1333580 ?        S      0:00 /opt/bitnami/mariadb/sbin/mysqld --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid --wsrep_node_name=mariadb-mariadb-galera-0 --wsrep_node_address=fc00:ac1a::d764 172.16.215.100 --wsrep_cluster_name=galera --wsrep_cluster_address=gcomm:// --wsrep_sst_method=mariabackup --wsrep_sst_auth=mariabackup:IH7qrVsZxf

To Reproduce:
Create a mariadb-galera instance on a dual-stack Kubernetes 1.19 environment.
"helm install -n my_namespace mariadb bitnami/mariadb-galera"

To reproduce with the override, add "--set extraEnvVars[0].name=DB_GALERA_NODE_ADDRESS --set extraEnvVars[0].valueFrom.fieldRef.fieldPath=status.podIP"
The behavior will be the same. I verified the env variable was properly set by running

kubectl exec -it -n test mariadb-test-mariadb-galera-0 env
<snip>
MARIADB_ENABLE_LDAP=no
DB_GALERA_NODE_ADDRESS=172.16.215.107
MY_POD_NAME=mariadb-test-mariadb-galera-0
<snip>

Kubernetes Version:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Helm Version:
version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

[miguelaeh]

Hi @sptmhall ,
Thank you very much for the detailed explanation. I will create an internal task to check this and find the best way to solve it.
Regards.

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[kkosewski]

Hello,
any progress on solving this issue? We are facing exactly the same problem in our dual stack environment.
Btw. There was a similar issue in cassandra DB image - maybe it will be helpful: https://github.com/bitnami/bitnami-docker-cassandra/issues/69

Regards,
KK

[alemorcuq]

Hi, @kkosewski. Thanks for reaching out. There hasn't been any progress so far, but I will try to raise the priority of this.

[sylvainOL]

Hello @sptmhall and @kkosewski, if you look at https://github.com/bitnami/bitnami-docker-mariadb-galera/blob/master/10.5/debian-10/rootfs/opt/bitnami/scripts/mariadb-env.sh#L180, you'll see that DB_GALERA_NODE_ADDRESS is not the right env variable to set but MARIADB_GALERA_NODE_ADDRESS (I did the same mistake with another env variable ^_^).

it would allow to have a workaround but I believe a proper solution (finding IPv6 address and IPv4 and choosing the right one according to where we want to listen on) would be better...

[kprasad99]

Any reason why fieldRef is not used or do you foresee any issue if fieldRef is used?

            - name: MARIADB_GALERA_NODE_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

[miguelaeh]

Hi, I can see in the container logic the env var is DB_GALERA_NODE_ADDRESS: https://github.com/bitnami/bitnami-docker-mariadb-galera/blob/61999a3f0c3dc9c799e225b48929ced3f6f24722/10.5/debian-10/rootfs/opt/bitnami/scripts/libmariadbgalera.sh#L645

There is probably a typo in the README or in the logic but in the logic MARIADB_GALERA_NODE_ADDRESS is not used, and in the README DB_GALERA_NODE_ADDRESS does not appear.

[kprasad99]

@miguelaeh I see MARIADB_GALERA_NODE_ADDRESS being reassigned to DB_GALERA_NODE_ADDRESS here https://github.com/bitnami/bitnami-docker-mariadb-galera/blob/61999a3f0c3dc9c799e225b48929ced3f6f24722/10.5/debian-10/rootfs/opt/bitnami/scripts/mariadb-env.sh#L204

[miguelaeh]

ah ok, so it seems that the variable that should be set is MARIADB_GALERA_NODE_ADDRESS since DB_GALERA_NODE_ADDRESS is taking the value from it.

[keliansb]

Hello, any update on this issue? I am having the same error Too many arguments on my dual stack K8s cluster.

[miguelaeh]

I think this PR will bring some light here: https://github.com/bitnami/bitnami-docker-mariadb-galera/pull/69

[keliansb]

So it seems that the docker image now supports the dual stack mode, what about the Helm chart? I still see the value bind_address=0.0.0.0 in the values.yaml file.

[miguelaeh]

You can edit that in your own values.yaml configuration.

[carrodher]

Unfortunately, this issue was created a long time ago and although there is an internal task to fix it, it was not prioritized as something to address in the short/mid term. It's not a technical reason but something related to the capacity since we're a small team.

Being said that, contributions via PRs are more than welcome in both repositories (containers and charts). Just in case you would like to contribute.

During this time, there are several releases of this asset and it's possible the issue has gone as part of other changes. If that's not the case and you are still experiencing this issue, please feel free to reopen it and we will re-evaluate it.

[sandeepk8s]

Any reason why fieldRef is not used or do you foresee any issue if fieldRef is used?

            - name: MARIADB_GALERA_NODE_ADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

Thanks @kprasad99 😄 that worked