hh: enforce hard antiaffinity on web nodes

[cirocosta]

Hey,

Having jobs that stream in and out a lot, we might end up having problems with the web pods if those end up being scheduled to run in the same machine as the whole network bandwidth of the machine might be consumed by them, while in reality, those two could be serving the full bandwidth of two VMs.

For instance, consider the following case:

There we have two web nodes in the same VM consuming the whole 1Gib of network bandwidth that the instance has (2* 250Mbit TX + 2*250MBit RX).

If those workloads were split in two instances, we could go even higher.

[cirocosta]

It's there!

hush-house/deployments/with-creds/hush-house/values.yaml

Lines 36 to 48 in dff9e7e

	web:
	replicas: 2
	nodeSelector: { cloud.google.com/gke-nodepool: generic-1 }
	affinity:
	podAntiAffinity:
	preferredDuringSchedulingIgnoredDuringExecution:
	- weight: 100
	podAffinityTerm:
	topologyKey: kubernetes.io/hostname
	labelSelector:
	matchLabels:
	app: hush-house-web
	release: hush-house

Funny thing: the throttling there was actually at the disk of the receiver that would slow down the whole streaming of the contents.