Issue facing in regards to bounding pulp file-storage storage

[harshad16]

Hello,

While trying to set up an instance of pulp on pulp-operator, we are facing an issue, as we don't have storage that supports the ReadWriteMany with volume mode file-system. We thought of using the alternative and switch to ReadwriteOnce mode [1]
and use node affinity to make the deployment stay on one single node , so all pods can interact with the storage(as the ReadWriteOnce on same node kinda behaves as ReadWriteMany)

However, when stating the node affinity, it is not bounding the pods to the same node.
Is this the right way of using the pulp node affinity?
https://github.com/operate-first/apps/blob/f67c2ae1556633acdad02f0771800ce488e62e83/pulp/overlays/moc/smaug/opf-pulp.yaml#L31

Related-to: operate-first/support#456

[fridex]

CC @mikedep333, could you please help us with this?

[mikedep333]

@fridex That looks correct.

We implemented it here:
#119

We would have to spend more time investigating why it's not working. I do not have a multi node test environment yet.

[harshad16]

@mikedep333
few more details, we are using the pulp-operator version: v0.5.0
the issue seems to be the affinity was not applied to any of the pods created by the pulp-operator.
I have checked the yaml of the instance of pulp-operator and it has the affinity that we have setup , however, the resulting pod yaml doesn't have any inclusion.

[chambridge]

So I deployed to minikube as a test with the specified affinity. I see the same issue as reported. However, if I update the main role for a component and add a debug var for affinity I can see that the value is set and present. For some reason the templating doesn't seem to be obtaining this value, which I don't understand. Will continue to investigate.

[chambridge]

Found the cause of the issue. This is related to the following operator-sdk problem:
operator-framework/operator-sdk#1770

The problem is when the data is passed into the template it has been converted to the lowercased and underscore separated variable:

  affinity:
    node_affinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: name
            operator: In
            values:
              - oct-03-31-compute

to:

affinity:
  nodeAffinity:
    required_during_scheduling_ignored_during_execution:
      node_selector_terms:
      - match_expressions:
        - key: name
          operator: In
          values:
          - oct-03-31-compute

This then gets discarded by the Kubernetes API as the format is invalid.

A similar issue had impacted the pulp_settings and was fixed with this PR: #35
I found it linked to the above issue.

This can be resolved with a similar approach. I'll make the necessary changes and open a PR.

[dkliban]

@harshad16 Looks like @chambridge fixed this issue. Please confirm and close this issue.

[harshad16]

@dkliban i m waiting on the operator release
I m not sure on how to test it without updating the operator to a newer version

[fao89]

@dkliban i m waiting on the operator release I m not sure on how to test it without updating the operator to a newer version

I'll be releasing soon

[harshad16]

Thanks for the release @fao89
when we upgrade the pulp instance and reapplied the changes
we are seeing the following issue:

 TASK [Set node affinity] ********************************
�[0;31mfatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: {{ vars['_pulpproject_org_pulp']['spec'] }}: 'dict object' has no attribute '_pulpproject_org_pulp'\n\nThe error appears to be in '/opt/ansible/roles/postgres/tasks/main.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Set node affinity\n  ^ here\n"}�[0m

let me know if this should be reported on a separate issue.

[fao89]

@harshad16 no need for reporting a new issue, I'll be looking into that

[fao89]

@harshad16 I've released 0.6.1 could you please test it?

[harshad16]

Thanks @fao89
I can see the v0.6.1 and it is propagating the node affinity to the pods.
Thanks all who helped with the issue.
closing it 💯