Tilt up appears to delete previously created resources

[parak]

Given a set of resources that are only managed when tilt up runs (managed via a tilt_subcommand variable), when tilt down runs those resources remain as expected. For example, this is used when a PVC needs to be retained between tilt runs.

Subsequently after the tilt down, when tilt up runs again, these resources seem to get deleted then recreated. The aforementioned PVC gets an age value indicating that it was created on that tilt up run. Looking at the verbose (klog=8) logs, the very first thing logged in the uncategorized section is a list of all the resources that have been retained from a prior tilt up, and shortly thereafter kubectl logs that they are all being deleted. This also deletes other resources, such as secrets, configmaps, serviceaccounts, clusterroles, etc.

The main log message of deletions starts with 'kubectl stdout:' and gives a list of all the resources that have been deleted, mirroring the list logged to uncategorized.

Thanks!

[nicks]

Hmmm...I was not able to reproduce this issue. Here's the sandbox I'm playing around in:
https://github.com/tilt-dev/tilt-issue-3719/blob/master/Tiltfile

Does that capture what you're doing, or am i missing something? When I tilt up in that sandbox, it preserves both the configmap and the pvc

[landism]

Looking at the verbose (klog=8) logs, the very first thing logged in the uncategorized section is a list of all the resources that have been retained from a prior tilt up, and shortly thereafter kubectl logs that they are all being deleted.

My hypothesis for what is happening:

1. Tilt applies all mutable k8s entities as a single kubectl apply batch.
2. If even one of those entities fails to apply with an error necessitating a delete/create, the whole batch will be deleted/created. (source)
3. One such error is when the annotations are too long (source), which often happens with ConfigMaps, which is one of the types mentioned in the OP.

If your log shows an error along the lines of "metadata.annotations: Too long: must have at most X bytes" when applying the ConfigMap, that'd confirm this hypothesis. If you have a ConfigMap that's potentially bigger than 256k, that'd be evidence for this hypothesis.

Regardless of whether the scenario I described is what's actually happening in your case, I'm pretty sure it's possible and a thing we should fix.

[landism]

repro'd the above in this branch

The logs show this error even without using any (k)log verbosity flags:

Falling back to 'kubectl delete && create': The ConfigMap "my-config" is invalid: metadata.annotations: Too long: must have at most 262144 bytes (https://github.com/kubernetes/kubectl/issues/712)

[nicks]

@landism ya, that was my first instinct too. @parak and I were discussing this issue in slack, and he did not see the "Falling back to 'kubectl delete && create" message. we were more concerned about the PVC than the configmap

[landism]

he did not see the "Falling back to 'kubectl delete && create" message

shucks.

Two things that'd be nice to look at to rule things out:

1. the output of kubectl get -ojson pvc mypvc before and after the second tilt up
2. full logs from running the second tilt up with --klog 8
   @parak would you mind sharing those, here or privately?

[nicks]

OK, we think we know what's going on.

All the uncategorized resources are getting deployed together. But applying the ConfigMap fails with this error

Falling back to 'kubectl delete && create': I0831 14:00:29.961495 22339 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"ConfigMap "[...]" is invalid: metadata.annotations: Too long: must have at most 262144 bytes","reason":"Invalid","details":{"name":"[...]","kind":"ConfigMap","causes":[{"reason":"FieldValueTooLong","message":"Too long: must have at most 262144 bytes","field":"metadata.annotations"},{"reason":"FieldValueTooLong","message":"Too long: must have at most 262144 bytes","field":"metadata.annotations"},{"reason":"FieldValueTooLong","message":"Too long: must have at most 262144 bytes","field":"metadata.annotations"},{"reason":"FieldValueTooLong","message":"Too long: must have at most 262144 bytes","field":"metadata.annotations"}]},"code":422} (kubernetes/kubectl#712)

which means that Tilt will give up and do a complete delete and apply

I think what we should probably do in the short-term is apply these resources independently rather than batching them together. This would fix a bunch of issues in this space (e.g. #3686)

More medium-term, we should reimplement the kubectl apply like Helm did here:
https://github.com/helm/helm/blob/master/pkg/kube/client.go#L415
which just allows them to be way more robust about handling API errors in a fine-grained way, rather than the sledgehammer approach Tilt takes here

[nicks]

This should be fixed in tilt v0.17.4 !