Mon pods have a hardcoded nodeselector. How do I get it to refresh?

[2fst4u]

I have updated my placement settings for my mons to prefer my more powerful nodes, but schedule on my arm nodes if they absolutely must in order to maintain uptime.

My placement spec in cluster.yaml looks like this:

  placement:
    mon:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
            - key: kubernetes.io/arch
              operator: In
              values:
              - amd64

I can see that the affinity applies to the pods, because they show the following while running:

spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        weight: 100

But when I delete the mon pods to force them to reschedule, they never schedule where I want them to in the affinity, they just schedule back where they were. When I drain the node one of the mons is on, the pod remains pending until it is uncordoned. When I describe the pod with kubectl, it shows the mons have a hardcoded node selector of the specific name of the respective pod they have been running on.

How do I get this node selector to go away, or get the operator to reschedule a completely new mon to get the affinity to apply?

[travisn]

With the cluster.yaml example, after scheduled, the mons will have affinity to a specific node because they use a host path to store data. If you want the mons to be schedulable to a different node, you will need to specify a volumeClaimTemplate for the mon as seen in the cluster-on-pvc example. Alternatively, you may want to taint the nodes and then failed over the mons.

[2fst4u]

I don't want it to have affinity to a specific node, I just want it to rebuild so it uses the affinity set in the schema now to prefer certain nodes. Not as a requirement but as a preference.

I just need the mon pod to be destroyed and the deployment for it rebuilt. But when I delete it, the operator creates it exactly the same as it was before and it runs on the same node it was.

[travisn]

If you're using the cluster.yaml example, the mons only use the preferred affinity during scheduling. After the scheduling is done, a mon will have the node affinity, which can't be changed. Otherwise, the storage on the dataDirHostPath would be lost if the mon is moved to a new node. Are you in a cloud environment where there are backing PVs that can be moved like in the cluster-on-pvc example?

[2fst4u]

Nah on-prem. So can I failover the mon and get the operator to reschedule a fresh one at all?

[travisn]

Correct, you can failover a mon to get a new mon created on a different node.

[2fst4u]

Awesome I'll give that a crack.

[2fst4u]

Correct, you can failover a mon to get a new mon created on a different node.

Ok so I'm having trouble getting the mon to fail over. If I delete the deployment or pod, the operator just won't rebuild it at all. What am I doing wrong?

[travisn]

To trigger failover, try scaling down the mon deployment. But if you deleted the mon deployment, that should work too. Other mons are still in quorum, right? Does ceph status still show healthy except a mon down? Any errors in the rook operator log?

[2fst4u]

Ah whoops I think I just wasn't waiting long enough. I didn't realise how long the default was.

Sorted now, thanks!