Enable aliasing of isolation groups

[schallert]

We currently define isolation groups as the name of a zone and the number of desired instances in that zone. For example:

isolationGroups:
- name: us-east1-b
  numInstances: 1
- name: us-east1-c
  numInstances: 1
- name: us-east1-d
  numInstances: 1

The Name field maps directly to 1) the isolationGroup field in the M3DB placement, and 2) the value of the failure-domain.beta.kubernetes.io/zone label on a node. This means that one cannot create clusters with RF=3 within a single.

This has been suitable for us so far as our focus has been on creating highly available regional M3DB clusters with strict zonal isolation. However there may be use cases where users want to have RF=3 within a single zone, and we should accommodate that.

One possible solution would be to alias isolation groups, wherein multiple isolationGroups in the M3DB placement could map to separate zones in the Kubernetes cluster. For example:

isolationGroups:
- name: isoGroup1
  zoneValue: us-east1-b
  numInstances: 1
- name: isoGroup2
  zoneValue: us-east1-b
  numInstances: 1
- name: isoGroup3
  zoneValue: us-east1-b
  numInstances: 1

In this case we'd have an RF=3 cluster within us-east1-b, but 3 unique isolation groups for M3DB to replicate shards across.

[benjaminhuo]

hi @schallert, I'm deploying an test m3db cluster in my local kubernetes environments which means there're no zones or regions at all, so just wondering how to set the isolationGroups for this?

Currently I got below errors when I apply simple-cluster.yaml

Thanks
Ben

still the same error after I made below changes to the yaml to match the node label key and values:

[schallert]

Hey @benjaminhuo, this is something we fixed recently I believe. Your use case should be solved by using the master version of the operator (quay.io/m3db/m3db-operator:master) and setting isolationgroups with no affinity terms:

  isolationGroups:
  - name: rep0
    numInstances: 1
  - name: rep1
    numInstances: 1
  - name: rep2
    numInstances: 1

I happen to be prepping a release so this fix will be in an official version! We have some docs on "No Affinity" here: https://github.com/m3db/m3db-operator/pull/133/files#diff-0feacd3ccad0c9a5bfc84154f3d1bfec

Let me know if this fixes your issue!

[benjaminhuo]

The master version works! Thanks @schallert!

Regarding the node affinity settings:
The doc says it should be set using NodeAffinityTerm, while the code says it's set by nodeAffinityKey and nodeAffinityValues. So I'm a little bit confused here.

Another question is about the query engine , it will be included in this operator or there will be another query operator or helm/yaml to deploy it in k8s?

Thanks
Ben