Support shard allocation awareness

[munnerz]

Elasticsearch can have parameters set for shard allocation awareness, so that it can distribute shards according to various policies (e.g. across racks, AZs etc.)

We should support this through the ElasticsearchCluster resource. Ideally, we wouldn't introduce the concept of racks, but instead allow a user to be flexible through the use of labels.

https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html

[munnerz]

I've started mocking up how our API could look for this - so far I've come up with:

apiVersion: navigator.jetstack.io/v1alpha1
kind: ElasticsearchCluster
metadata:
  name: test
spec:
  minimumMasters: 1
  sysctl:
  - vm.max_map_count=262144

  pilot:
    repository:
    tag:
    pullPolicy:

  image:
    repository: docker.elastic.co/elasticsearch/elasticsearch
    tag: 5.6.2
    pullPolicy: IfNotPresent
    ## This sets the group of the persistent volume created for
    ## the data nodes. This must be the same as the user that elasticsearch
    ## runs as within the container.
    runAsUser: 1000

  # == new field ==
  allocationAwareness:
    attributes:
    - zone
    - region
  # == end new field ==
  nodePools:
  - name: mixed
    replicas: 3

    roles:
    - master
    - ingest
    - data

    resources:
      requests:
        cpu: '500m'
        memory: 2Gi

    persistence:
      enabled: false

    # == new field ==
    attributes:
    - name: "zone"
      valueFrom:
        nodeLabel: "kubernetes.io/zone"
    - name: "region"
      valueFrom:
        nodeLabels: "kubernetes.io/region"
    - name: "use"
      value: "mixed"
    # == end new field ==

/cc @simonswine @wallrj @cehoffman wdyt of this? I've purposefully namespaces allocationAwareness.attributes so that we can support the force.zone fields of the ES API in future.

I think the structure for nodeLabel (under attributes) might need re-jigging to be a struct in case we need to extend it in the future. This is a very similar model to how Pod.spec.containers.env is handled, so we can probably borrow some decisions from there. We probably also want to allow fieldRefs as well as label references here too.