Don't schedule on Windows nodes by default #9165
Comments
|
Did you also deploy the |
|
@leseb I've not got as far as the cluster chart yet. I've just implemented Windows node functionality and was making sure I didn't break anything else. I'll take a look at the cluster chart stuff over the next few days and post a |
|
@leseb I've reset my cluster and re-run this time with the operator and cluster chart version 1.7.9. I used the values from the original post above for the operator and the following for the cluster: toolbox:
enabled: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
cephClusterSpec:
dashboard:
enabled: false
crashCollector:
disable: trueThere are a number of combinations possible here, however this time I see the The source for the chart script is here: https://github.com/RipcordSoftware/hvk8scluster/blob/feature/issue-63-rook-ceph/src/hyper-v/remote-commands/install-rook-ceph-chart.sh |
|
The |
|
@leseb I've reset and updated my configuration for 3 Linux and 3 Windows nodes. I added This time I'll review FileSystem and ObjectStore CRs placement YAML and clean install chart version 1.8.1. |
|
For the OSDs, did the prepare job run successfully? |
|
@leseb the OSD issue was likely due to memory pressure causing the pods to be evicted. This is probably due to the The good news is I can now run toolbox:
enabled: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
cephClusterSpec:
dashboard:
enabled: false
crashCollector:
disable: true
placement:
all:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
cephFileSystems:
- name: ceph-filesystem
spec:
metadataPool:
replicated:
size: 3
dataPools:
- failureDomain: host
replicated:
size: 3
metadataServer:
activeCount: 1
activeStandby: true
placement:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
storageClass:
enabled: true
isDefault: false
name: ceph-filesystem
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions: []
parameters:
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: ext4
cephObjectStores:
- name: ceph-objectstore
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
dataPool:
failureDomain: host
erasureCoded:
dataChunks: 2
codingChunks: 1
preservePoolsOnDelete: true
gateway:
port: 80
instances: 1
placement:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
healthCheck:
bucket:
interval: 60s
storageClass:
enabled: true
name: ceph-bucket
reclaimPolicy: Delete
parameters:
region: us-east-1In order to set placement on Changing the array to a object in the chart |
@craigminihan glad to see it is working now.
Do you think this is something you could send a PR for? Thanks |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions. |
|
This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation. |
|
This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation. |
Is this a bug report or feature request?
What should the feature do:
When deployed into a mixed Linux and Windows cluster don't attempt to schedule on the Windows nodes.
What is use case behind this feature:
The Rook/Ceph chart should schedule on the Linux nodes essentially ignoring the Windows nodes if present. There is support for
nodeSelectorand*NodeAffinitywhich is partiallty successful however it is verbose and the end result is some pods end up scheduled on Windows nodes (node11 in this case) which just fail. See:The operator chart was deployed with
values.yamlas follows:Environment:
A test cluster showing the problem above:
The text was updated successfully, but these errors were encountered: