[Bug Fix]: Avoid evicting more pods than necessary by adding Timestamps for fsstats and ignoring stale stats

[dashpole]

Continuation of #33121. Credit for most of this goes to @sjenning. I added volume fs timestamps.

why is this a bug
This PR attempts to fix part of #31362 which results in multiple pods getting evicted unnecessarily whenever the node runs into resource pressure. This PR reduces the chances of such disruptions by avoiding reacting to old/stale metrics.
Without this PR, kubernetes nodes under resource pressure will cause unnecessary disruptions to user workloads.
This PR will also help deflake a node e2e test suite.

The eviction manager currently avoids evicting pods if metrics are old. However, timestamp data is not available for filesystem data, and this causes lots of extra evictions.
See the inode eviction test flakes for examples.
This should probably be treated as a bugfix, as it should help mitigate extra evictions.

cc: @kubernetes/sig-storage-pr-reviews @kubernetes/sig-node-pr-reviews @vishh @derekwaynecarr @sjenning

[k8s-reviewable]

This change is 

[dashpole]

@k8s-bot bazel test this

[dashpole]

/release-note-none

[vishh]

nit: One field per line.

[vishh]

Only one nit.

/lgtm
/approve

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

The following people have approved this PR: dashpole, vishh

Needs approval from an approver in each of these OWNERS Files:

• pkg/kubelet/OWNERS [vishh]
• pkg/volume/OWNERS
• test/e2e_node/OWNERS [vishh]

We suggest the following people:
cc @matchstick @timstclair
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[vishh]

@saad-ali @matchstick I understand the metrics code under /pkg/volume well since I orginally reviewed it. Hence I'm approving this PR to get it through the merge queue soon and deflake node e2es. I hope I don't cause much inconvenience my manually approving this PR

[dashpole]

@k8s-bot kops aws e2e test this
@k8s-bot cvm gce e2e test this

[smarterclayton]

@k8s-bot kops aws e2e test this

[smarterclayton]

@k8s-bot cvm gce e2e test this

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

[vishh]

@dashpole did this PR help with the eviction test flakiness?

[dashpole]

Yes, I believe it did. Look at the flaky testgrid. Ignore the section that is almost entirely red March 4th-7th, since during that period tests were being run in parallel, which broke everything. Compare March 2-3 (before this change) with March 8-9 (after). Failure rate over those periods went from 56% to 38%. I took a quick look at a failure yesterday, and it was simply using stats that were collected 7 seconds before use, which still showed pressure, and caused the extra pod to be evicted. I think this change helped though

[vishh]

Awesome! Like we discussed offline, now we need to bring down the cost of collecting disk usage metrics and then collect more often than every minute.

…

On Thu, Mar 9, 2017 at 1:24 PM, David Ashpole ***@***.***> wrote: Yes, I believe it did. Look at the flaky testgrid <https://k8s-testgrid.appspot.com/google-node#kubelet-flaky-gce-e2e&width=20>. Ignore the section that is almost entirely red March 4th-7th, since during that period tests were being run in parallel, which broke everything. Compare March 2-3 (before this change) with March 8-9 (after). Failure rate over those periods went from 56% to 38%. I took a quick look at a failure yesterday, and it was simply using stats that were collected 7 seconds before use, which still showed pressure, and caused the extra pod to be evicted. I think this change helped though — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42435 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGvIKAWl2JLxCI8BSL7Tl0J8zi5HXURUks5rkG4AgaJpZM4MRffP> .