Skip to content

pod_watch: only emit events from pods whose specs we have deployed since startup#2439

Merged
landism merged 4 commits into
masterfrom
mlandis/ch3772/tilt-resource-status-sometimes-mixes-old
Oct 29, 2019
Merged

pod_watch: only emit events from pods whose specs we have deployed since startup#2439
landism merged 4 commits into
masterfrom
mlandis/ch3772/tilt-resource-status-sometimes-mixes-old

Conversation

@landism

@landism landism commented Oct 28, 2019

Copy link
Copy Markdown
Member

Problem

When a user starts Tilt, Tilt will build a new image with a new hash, create new Deployment YAML, and apply it. If there is already a deployment running, this application will edit the existing deployment, leaving the UID unchanged. Thus, when k8s tells Tilt about the pod the deployment had before Tilt even started, Tilt says "hey, I know that deployment's UID!" and treats it as its own.
This has a few concrete minor effects:

  1. Tilt prints logs from pods that predate code the user tilt upd with
  2. Tilt's "ready" state can flash green for a moment, in between the moment Tilt has completed kubectl apply and knows the Deployment's UID and the moment k8s has started deploying the new pod and told Tilt about it. (because the old pod matches the deployment and is healthy)
  3. Due to (2), the resource_deps startup behavior is pretty confusing and buggy-seeming: https://github.com/windmilleng/tilt.build/pull/318/files#diff-4605e4a7d790d4cf0f1dd26e49ea8455R47

Solution

Inject a hash of the pod template spec into every pod template spec we deploy. Then, similar to DeployedUIDSet, track those IDs for each manifest and only pay attention to updates for pods whose template specs we've deployed since startup.

@landism
landism requested review from maiamcc and nicks October 28, 2019 15:05
Comment thread internal/k8s/pod_template.go Outdated
Comment thread internal/engine/image_build_and_deployer.go Outdated
Comment thread internal/engine/k8swatch/pod_watch.go
Comment thread internal/engine/k8swatch/pod_watch.go Outdated
Comment thread internal/engine/k8swatch/pod_watch.go Outdated

@maiamcc maiamcc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 🙌

Comment thread internal/k8s/pod_template.go Outdated

require.Equal(f.T(), 1, len(entities), "expected only one entity. Yaml contained: %s", f.k8s.Yaml)

d := entities[0].Obj.(*v1.Deployment)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe d, ok := here and require.True(t, ok, "expected entity to be a deployment") -- if it's NOT a deployment for some reason, this is easier to debug than the weird nil pointer errors you'll get otherwise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(but also why are you assuming the first entity is always a deployment?)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe d, ok := here and require.True(t, ok, "expected entity to be a deployment") -- if it's NOT a deployment for some reason, this is easier to debug than the weird nil pointer errors you'll get otherwise

how about require.IsType?

also why are you assuming the first entity is always a deployment?

More like "assuming the entity is always a deployment" - this is after requiring that there's exactly one entity. All the tests that hit this code only have one entity, which is a deployment. I added a comment above the length assertion to make it easier for someone to understand what's going on if they add/modify a test such that that's no longer true.

Comment thread internal/engine/image_build_and_deployer_test.go Outdated

if ok {
set, ok := podTemplateSpecHashes[mn]
if ok && !hasOKPodTemplateSpecHash(pod, set) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe worth a descriptive comment e.g. "the pod template hash doesn't correspond to the most recently deployed one--this pod is probably from an old Tilt run" or something.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think the existing comment on hasOKPodTemplateSpecHash suffices?
Or do you want to add a descriptive comment to each of its call sites?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh whoops, i missed that. ya that's fine.

Comment thread internal/store/runtime_state.go
@nicks

nicks commented Oct 29, 2019

Copy link
Copy Markdown
Contributor

what's the status of this pr? lost track of whether it's tabled, or needs review, or waiting to be merged, something else?

@landism

landism commented Oct 29, 2019

Copy link
Copy Markdown
Member Author

what's the status of this pr? lost track of whether it's tabled, or needs review, or waiting to be merged, something else?

When I left yesterday, it was awaiting approval. I failed to ping you for approval yesterday after addressing your comments.
I now need to address Maia's comments and merge, unless you have any concerns.


if ok {
set, ok := podTemplateSpecHashes[mn]
if ok && !hasOKPodTemplateSpecHash(pod, set) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh whoops, i missed that. ya that's fine.

@landism
landism merged commit 58c787b into master Oct 29, 2019
@landism
landism deleted the mlandis/ch3772/tilt-resource-status-sometimes-mixes-old branch October 29, 2019 16:59
landism pushed a commit that referenced this pull request Nov 1, 2019
landism added a commit that referenced this pull request Nov 1, 2019
maiamcc pushed a commit that referenced this pull request Nov 7, 2019
…have deployed since startup (#2439)" (#2474)"

This reverts commit 30b3378.
maiamcc pushed a commit that referenced this pull request Nov 7, 2019
…have deployed since startup (#2439)" (#2474)"

This reverts commit 30b3378.
maiamcc pushed a commit that referenced this pull request Nov 7, 2019
…have deployed since startup (#2439)" (#2474)"

This reverts commit 30b3378.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants