Horizontally scaling indexed-search

[keegancsmith]

This issue is for tracking scaling out zoekt. See RFC 30: Zoekt Horizontal Scaling for details.

[keegancsmith]

3.9 I spent far more time on src-expose than I expected, so didn't get to spend the time I needed get this in a state I am happy with. However, I am continuing to work on it so expect it to be merged in early in the 3.10 milestone. 3.9 does however include a migration for kubernetes which was one of my bigger concerns for the work needed.

[keegancsmith]

v0 is in master, came across an interesting issue while trying it out in production. Because the k8s service we have for the indexed-search pod isn't headless, it seems StatefulSet pod DNS entries don't exist.

eg see this debug output for gitserver (which has a headless svc) for indexed-searched

/ # nslookup gitserver-0.gitserver
Server:    10.55.240.10
Address 1: 10.55.240.10 kube-dns.kube-system.svc.cluster.local

Name:      gitserver-0.gitserver
Address 1: 10.52.3.147 gitserver-0.gitserver.prod.svc.cluster.local
/ # nslookup indexed-search-0.indexed-search
Server:    10.55.240.10
Address 1: 10.55.240.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'indexed-search-0.indexed-search'

As such we don't have a stable network identity with the way it is currently setup! This means we will probably need to introduce another svc for indexed search, which is just a headless service. I am hoping that works, since StatefulSets are special since they specify a service name in the StatefulSet resource :/ Will report back here.

[keegancsmith]

Kubernetes update: So it looks like the serviceName has to be pointing to a headless service. However, that field is immutable, and changing a service to a headless service is not possible. So for 3.10 we will have to introduce a migration (again), where we replace the indexed-search service with a headless one. We will set ZOEKT_HOST in k8s to indexed-search-0.indexed-search (rather than updating the default in Sourcegraph).

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields

[keegancsmith]

Horizontal scaling has been merged into master and is all ready to go. Important notes:

• We rely on service discovery, so only need to update replica count in k8s.
• Service discovery also simplified the implementation of reliability as detailed in v2 of the RFC. If no replicas are available we fail. We support partial availability. If a replica is down for a long time we start indexing its repos on other nodes.
• Sourcegraph.com is running with 3 replicas.
• If there are unexpected issues you can fallback to our old codepath with:

  DISABLE_HORIZONTAL_INDEXED_SEARCH=true
  ZOEKT_HOST=indexed-search-0.indexed-search:6070