Error adding arbiter while converting PSS to PSA in operator 0.7.3 & Mongo version 5.0

[GayathriRameshK]

What did you do to encounter the bug?
Steps to reproduce the behavior:

1. I have a successfully running 3 node replicaset (PSS)
2. While converting PSS to PSA by adding arbiters: 1 in cr.yaml (mongo version 5.0.6 ), it is getting failed with error
   "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again."
3. Setting default Read concern to local and write concern to 1 using the below admin command and reapplying the cr file , resulting in error as below:

db.adminComamnd({
"setDefaultRWConcern" : 1,
"defaultWriteConcern" : { "w" :1},
"defaultREadConcern": {"level" : "local"}
})

Error:

NOT EQUAL :
V1 : 1
V2 : majority
REASON: Type() has different result : int32, string
NOT EQUAL :
V1 : map[w:1 wtimeout:0]
V2 : map[w:majority wtimeout:0]
REASON: Map field is different for key w : 1, majority

What did you expect?
The arbiter should get added to the replica set while converting from PSS to PSA without any issues.

What happened instead?
The arbiter pod is not getting added and the mongo pod is resulting in readiness probe failed.

$kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
arbiter-mongodb-0                              2/3     Running   0          29m

Operator Information

• Operator Version - 0.7.3
• MongoDB Image used - 5.0.6

If possible, please include:
cr.yaml

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: arbiter-mongodb
spec:
  members: 3
  type: ReplicaSet
  arbiters: 1
  version: "5.0.6"
  featureCompatibilityVersion: "5.0"
  security:
    authentication:
      modes: ["SCRAM"]


  users:
    - name: mongodba
      db: admin
      passwordSecretRef: # a reference to the secret that will be used to generate the user's password
        name: dbauser-scrt
      roles:
        - name: root
          db: admin
        - name: role-restore
          db: admin
      scramCredentialsSecretName: mongodba-scram
  

  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib


#use specific storage class, CPU & Memory - start
  # persistent: true
  statefulSet:
    spec:
      volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: mongo-stg-class 
            resources:
              requests:
                storage: 6Gi

        - metadata:
            name: logs-volume
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: mongo-stg-class 
            resources:
              requests:
                storage: 3Gi
      template:
        spec:
          initContainers:
    

          containers:
            - name: "mongodb-agent"
              resources:
                requests:
                  cpu: 200m
                  memory: 200Mi
                limits:
                  cpu: 250m
                  memory: 250Mi
            - name: "mongod"
              resources:
                requests:
                  cpu: 1 #500m
                  memory: 1Gi #500Mi
                limits:
                  cpu: 2
                  memory: 2Gi
          
#use specific storage class, CPU & Memory - end

Logs - arbiter-mongodb-0

[2022-03-24T06:31:57.868+0000] [.info] [src/director/director.go:planAndExecute:563] <arbiter-mongodb-0> [06:31:57.868] Step=WaitPrimary as part of Move=WaitPrimary in plan failed : <arbiter-mongodb-0> [06:31:57.868] Postcondition not yet met for step WaitPrimary because 
['currentState.IsPrimary' = false].
 Recomputing a plan...
[2022-03-24T06:31:58.204+0000] [.info] [src/config/config.go:ReadClusterConfig:437] [06:31:58.204] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2022-03-24T06:31:58.204+0000] [.info] [main/components/agent.go:LoadClusterConfig:270] [06:31:58.204] clusterConfig unchanged
[(local=false) using desired auth key
NOT EQUAL : 
V1 : 1
V2 : majority
REASON: Type() has different result : int32, string
NOT EQUAL : 
V1 : map[w:1 wtimeout:0]
V2 : map[w:majority wtimeout:0]
REASON: Map field is different for key w : 1, majority
[2022-03-24T06:31:59.160+0000] [.info] [src/mongoctl/processctl.go:Update:3307] <arbiter-mongodb-0> [06:31:59.160] <DB_WRITE> Updated with query map[] and update [{$set [{agentFeatures [StateCache]} {nextVersion 2}]}] and upsert=true on local.clustermanager
[2022-03-24T06:31:59.256+0000] [.info] [src/config/config.go:ReadClusterConfig:437] [06:31:59.256] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2022-03-24T06:31:59.257+0000] [.info] [main/components/agent.go:LoadClusterConfig:270] [06:31:59.256] clusterConfig unchanged
[2022-03-24T06:32:00.057+0000] [.info] [src/director/director.go:computePlan:279] <arbiter-mongodb-0> [06:32:00.057] ... process has a plan : WaitPrimary,UpdateDefaultRWConcern,WaitRsConfCommitted
[2022-03-24T06:32:00.057+0000] [.info] [src/director/director.go:tracef:793] <arbiter-mongodb-0> [06:32:00.057] Running step: 'WaitPrimary' of move 'WaitPrimary'
[2022-03-24T06:32:00.057+0000] [.info] [src/director/director.go:tracef:793] <arbiter-mongodb-0> [06:32:00.057] because 
[All the following are true: 
    ['currentState.NeedToStepDownCurrentPrimary' = false]
    ['currentState.Up' = true]
]

[siegenthalerroger]

The issue is that you need to up members to 4 to achieve 3 replicas + 1 arbiter. But this is broken in a different way, see #773

[GayathriRameshK]

Hi,

Having members:4 and arbiters:1 in cr.yaml, the separate statefulset for arbiter is getting created also it is adding one extra pod to the existing mongo stateful set . It is also noticed the existing mongo pods are exiting with readiness probe error as follows:

Though the arbiter pod is showing as running , In rs.status() , only the mongo nodes are available and the arbiter node is not getting added.

This behaviour mainly noticed when adding arbiter in mongo version 5.0.x.

$kubectl get pods   
NAME                                           READY   STATUS    RESTARTS   AGE
arbiter-mongodb-0                              2/3     Running   0          18m
arbiter-mongodb-1                              2/3     Running   0          20m
arbiter-mongodb-2                              2/3     Running   0          19m
arbiter-mongodb-3                              3/3     Running   0          11m
arbiter-mongodb-arb-0                          3/3     Running   0          10m

$kubectl get sts
NAME                  READY   AGE
arbiter-mongodb       1/4     26m
arbiter-mongodb-arb   1/1     17m

$kubectl describe pod arbiter-mongodb-0

 Normal   Created    24m                   kubelet            Created container mongodb-agent
  Normal   Started    24m                   kubelet            Started container mongodb-agent
  Warning  Unhealthy  4m49s (x79 over 24m)  kubelet            Readiness probe failed:

$kubectl logs arbiter-mongodb-0 -c mongodb-agent
[2022-03-28T06:54:32.738+0000] [.error] [src/director/director.go:mainLoop:398] <arbiter-mongodb-0> [06:54:32.738] Failed to planAndExecute : <arbiter-mongodb-0> [06:54:32.738] Plan execution failed on step RsReconfig as part of move RsReconfig : <arbiter-mongodb-0> [06:54:32.738] Failed to apply action. Result = <nil> : <arbiter-mongodb-0> [06:54:32.731] Error reconfiguring replica set : <arbiter-mongodb-0> [06:54:32.731] Error running command:
        dbName=admin

 Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.

[GayathriRameshK]

Also noticed the arbiter pod is bound to service arbiter-mongodb-svc instead of arbiter-mongodb-arb-svc.
Hence getting the below error:

Error connecting to <hostname:port> :: caused by :: Could not find address for <hostname:port> : SocketException: Host not found (authoritative)

On manually updating the arbiter pod labels to arbiter-mongodb-arb-svc, the arbiter node is getting added successfully.

Is this the correct way of doing?
Kindly note, this issue is common in all versions of mongo while adding the arbiter.

[tzorzan]

Also starting a PSA replicaset from scratch is broken, see #926. Also tried to create a PSS replicaset, then patch the CR to achieve a PSA replicaset but it doesn't work (different errors, same result).
Arbiters management seems totally screwed up in recent versions of the operator. It was working in the previous chart (mongodb-kubernetes-operator).

[GayathriRameshK]

Can anyone share your inputs on this issue?

[rodrigovalin]

Hello everyone, I have an update on this.

I've been investigating for the last couple of days, and I found the bug already. We have a semi working solution, but I will have to ask you for a little bit of patience, we will have a solution to this today or early next week at the most.

This is the PR that's in progress: #941

[tzorzan]

Wonderful! Could I help somehow?

[rodrigovalin]

@tzorzan thanks for your help! If you can take a look at #941 and try a few scenarios; Doing PSS -> PSA does not work, but doing PSA to any configuration seems to work. I'm investigating doing PSS -> PSA right now.

[rodrigovalin]

MongoDB version 5.x can't be configured with an Arbiter right now. I'm investigating a solution to this.

I've just merged #941 and 4.x MongoDB arbiters is fixed now.

[GayathriRameshK]

On testing with the latest code from master also. getting the same issue with arbiter addition. Arbiter is not able to add and giving host not found error.

Also , Scaling replica set is giving reconciliation issue and resulting in unhealthy cluster

[GayathriRameshK]

can anyone update on this issue?

[GayathriRameshK]

Team,

Scaling replicaset without arbiter configuration with minimal configuration is working but when tls is enabled and replica set hosrizons are set, we are facing issues with scaling replicaset from 3 to 1 and 1 to 3. This is resulting in a unhealthy cluster and giving reconciliation issue.This issue is noticed in operator version 0.7.2 and 0.7.3.

[github-actions]

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.