Manual restart of pod from dashboard

[majelbstoat]

Sometimes pods get into a weird state and the best thing to do is just blow it away and start from scratch. Currently, the only way I know how to do this is to re-run tilt up, but this restarts all pods. (I also have a standalone rolling-update script which works by patching an environment variable into the container spec, but it would be nice to do it all from the same place.)

I'd love to be able to do this from within the cli dashboard, perhaps by selecting the workload and pressing r?

[landism]

thanks for filing! this is a thing we've been talking about and want to implement

[maiamcc]

Closing this in favor of #1012.

[maiamcc]

Oh whoops my mistake, this is a slightly different case than #1012 (which wants a build-and-deploy, whereas this ticket wants to restart a pod; e.g. if there are no code changes since the last deploy, then a build-and-deploy wouldn't affect anything). Re-opening.

[andymartin-sch]

We've been rolling out our Tilt setup to more developers on our team, and it's come up a few times that it was surprising that the "Force Update" functionality didn't restart the pod if no changes were made to the image.

Offhand, I can't think of any reason why someone who clicks the Force Update button would not be okay with the pod being restarted, but maybe there is one? Either way, I think having an easy way to force a restart of a pod would line up well with the "'Turn it off and on again" philosophy mentioned in the blog post.

Right now our fallback is to tell the user to delete the pod with kubectl, but we are SO CLOSE to being able to let "normal" app developers never use any command-line tools during their development workflows, so it would be nice if we could eliminate this use case of kubectl with a friendly button in Tilt.

Let me know if any other info from our use case would help!

[rpocase]

Just adding another use case to this - I'm working on integrating istio into certain profiles and actually have istio redeployed/reconfigured as part of the Tiltfile. E.g., adding sidecar injection. My options now are to either tilt down or individually shutdown pods. Being able to selectively redeploy a pod from the interface would be a bit easier as I work through issues with each service individually.

[maiamcc]

Thanks for the extra info, both of you! As I'm thinking about this more: are either of you using live update? Because restarting a pod wipes out any in-place changes we've made during live update, this feature is more technically complicated than I originally thought.

Would an acceptable starting point be a command that forces a rebuild AND forces a pod update? Otherwise, I'm not sure we could guarantee that you'd get the correct code on the restarted pod.*

*I mean we actually could, we already check for the case where we've lost in-place changes--but when we detect this case, we just rebuild and redeploy your code, so it amounts to the same thing, but slightly slower and more confusing.

edited to add elaboration:

there are two cases to consider--

1. pod that has NOT been live updated, i.e. no in-place changes
   a. we can safely force a rebuild of this resource (i.e full docker build), though it probably won’t restart the pod if nothing has changed
   b. (it’s not currently supported but would be safe) to just force a restart of the pod — it would spin up again with the same container image
2. pod HAS been live updated, i.e. has in-place changes — the image specified in the k8s yaml doesn’t match the state of the pod
   a. we can safely force a rebuild of this resource (i.e. full docker build), and it will restart the pod b/c the image tag will have changed
   b. we CANNOT just force a restart of this pod, at least not cleanly — it will spin back up running old code, tilt will detect this as an error case and do a full rebuild+deploy (edited)

[rpocase]

I'm using live_update for a handful of my pods. I think a force rebuild would be acceptable assuming layers are reused as appropriate. In my case, a rebuild would still be pretty fast because I SHOULD only hit source code syncs (package.json install is an earlier layer). This definitely hurts more for a fully compiled language.

[maiamcc]

(To be clear, we're not prioritizing this feature yet, but I did a little digging to see what the scope might be, if we choose to implement this soon.)

In a slack discussion, @andymartin-sch proposed the following options:

Option A:
  i. full rebuild of the container image
  ii. force a restart of the pod (even if the image is not changed by the previous step)
Option B:
  i. a forced restart (using old container image)
  ii. do a one-off execution of all the live_update steps

Option A seems sufficient to both Andy and myself. It's doubtful a user would want to rebuild the docker image WITHOUT restarting the pod; somewhat more plausible but still unlikely that a user would want to restart the pod without updating it to run the most current code.

Option B is tricky to implement today because we don't currently have a concept of a one-off execution of live_update steps; i.e. we can't sync your entire sync directory up to the container, we can only sync individual files that we've determined match a sync rule. We may implement this sort of behavior in future (e.g. to support #2582) and can revisit Option B then.

[nicks]

I liked Maia's point that

It's doubtful a user would want to rebuild the docker image WITHOUT restarting the pod

So i put up a PR that changed the behavior of the exiting "force update" button. The suggestion is that the button should rebuild the docker image (taking advantage of the docker layer cache) and restart the pod - #3413

[wu-victor]

The behavior described in #1928 (comment) above, is now available in the latest Tilt release: https://github.com/tilt-dev/tilt/releases/tag/v0.14.3

[andymartin-sch]

@victorwuky @nicks Would it be possible to make this also cover restarting resources that don't have a corresponding docker_build() invocation in the Tiltfile?

We use the Tiltfile Config to opt-out of building some resources locally (by conditionally running docker_build()only if the service is present in the config's to-edit field). In v0.14.3, it looks like clicking the "Live Update" button will restart the pod for services that have a docker_build() call but not those without.

I can spin up a separate issue if that's preferable, thanks!

[wu-victor]

Yes please @andymartin-sch . It would be great to have a separate issue to manage that additional case.

[nicks]

filed as #3454

[andymartin-sch]

Thanks!

[dimaqq]

Ugh, I think I've hit this very issue and spent way too much time trying to diagnose it.

In my case, I was tweaking RBAC and the new account name just wasn't propagating to a container in a pod.

Solved by ^C and tilt up.