Scheduler pool docs

[TomAugspurger]

@scottyhq I've added a pool dedicated for schedulers in 2cbef8b (autoscaling from 0 - 50 machines).

We'll need to update the configs to ensure that scheduler pods end up there. I'll do that in #536 once everyone is ready?

[TomAugspurger]

cc @tjcrone as well. Are you able to create a "scheduler" pool with the taint

  taints:
  - effect: NO_SCHEDULE
    key: k8s.dask.org/dedicated
    value: scheduler

[scottyhq]

Sounds good - feel free to merge and then we can test to make sure everything works as planned. I just created a new nodegroup on AWS. here is the eksctl config:

  - name: scheduler-spot
    minSize: 0
    maxSize: 20
    desiredCapacity: 0
    privateNetworking: true
    instancesDistribution:
      instanceTypes:
        - t3.large
        - t3a.large
      spotInstancePools: 2
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 0
    volumeSize: 100
    volumeType: gp2
    labels:
      node-role.kubernetes.io/scheduler: scheduler
      k8s.dask.org/node-purpose: scheduler
    taints:
      k8s.dask.org/dedicated: 'scheduler:NoSchedule'
    tags:
        k8s.io/cluster-autoscaler/node-template/label/k8s.dask.org/node-purpose: scheduler
        k8s.io/cluster-autoscaler/node-template/taint/k8s.dask.org/dedicated: "scheduler:NoSchedule"
    ami: auto
    amiFamily: AmazonLinux2
    iam:
      withAddonPolicies:
        autoScaler: true
    preBootstrapCommands: # see https://github.com/weaveworks/eksctl/issues/1310
      - yum install -y iptables-services
      - iptables --insert FORWARD 1 --in-interface eni+ --destination 169.254.169.254/32 --jump DROP
      - iptables-save | tee /etc/sysconfig/iptables
      - systemctl enable --now iptables

[TomAugspurger]

I'll merge this since we're just updating the docs.