Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get filesystem stats for files on Windows #99221

Merged

Conversation

jsturtevant
Copy link
Contributor

@jsturtevant jsturtevant commented Feb 19, 2021

What type of PR is this?

/kind bug

What this PR does / why we need it:

Pod log stats for Windows doesn't work. With this update the stats for log files are included in summary:

http://localhost:8001/api/v1/nodes/2733k8s01000000/proxy/stats/summary (output truncated, two containers 1 with log output, 1 without)

 "pods": [
  {
   "podRef": {
    "name": "iis-1809-668c5b9c87-fqqml",
    "namespace": "default",
    "uid": "84ff19a1-f0bb-4ff9-bc0b-b291d0f86818"
   },
   "startTime": "2021-02-19T00:59:15Z",
   "containers": [
    {
     "name": "iis",
     "startTime": "2021-02-19T00:59:16Z",
     "cpu": {
      "time": "2021-02-19T01:14:27Z",
      "usageNanoCores": 4680917,
      "usageCoreNanoSeconds": 16203125000
     },
     "memory": {
      "time": "2021-02-19T01:14:27Z",
      "workingSetBytes": 59080704
     },
     "rootfs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 71303168,
      "inodesUsed": 0
     },
     "logs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 0,
      "inodesUsed": 0
     }
    }
   ],
  },
  {
   "podRef": {
    "name": "logger-f76f4d988-k5n9m",
    "namespace": "default",
    "uid": "6a3c95d2-27cf-4439-927e-724f069aee89"
   },
   "startTime": "2021-02-19T01:12:34Z",
   "containers": [
    {
     "name": "testwin",
     "startTime": "2021-02-19T01:13:06Z",
     "cpu": {
      "time": "2021-02-19T01:14:27Z",
      "usageNanoCores": 190263476,
      "usageCoreNanoSeconds": 19796875000
     },
     "memory": {
      "time": "2021-02-19T01:14:27Z",
      "workingSetBytes": 66772992
     },
     "rootfs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 37748736,
      "inodesUsed": 0
     },
     "logs": {
      "time": "2021-02-19T01:14:27Z",
      "availableBytes": 7994449920,
      "capacityBytes": 31687962624,
      "usedBytes": 172746066,
      "inodesUsed": 0
     }
    }

Which issue(s) this PR fixes:

Fixes #99173

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Pod Log stats for windows now reports metrics

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note kind/bug size/XS cncf-cla: yes do-not-merge/needs-sig needs-triage needs-priority sig/storage and removed do-not-merge/needs-sig labels Feb 19, 2021
@k8s-ci-robot k8s-ci-robot requested review from jingxu97 and saad-ali Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Feb 19, 2021

/sig windows

@k8s-ci-robot k8s-ci-robot added the sig/windows label Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Feb 19, 2021

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted and removed needs-triage labels Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Feb 19, 2021

/priority important-soon
/milestone v1.21

@k8s-ci-robot k8s-ci-robot added the priority/important-soon label Feb 19, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Feb 19, 2021
@k8s-ci-robot k8s-ci-robot removed the needs-priority label Feb 19, 2021
@jsturtevant jsturtevant force-pushed the windows-host-stats-provider branch from fac60e0 to b33c11b Compare Feb 19, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Feb 19, 2021

Reviewing the logs I am also finding the following error which is also related to the changes with the host_stats_provider

E0218 23:30:13.293729    4360 cri_stats_provider.go:407] unable to fetch pod etc hosts stats: failed to get stats CreateFile \var\lib\kubelet\pods\830b35a0-d8f3-421f-afc7-2861ace4ff20/etc-hosts: The system cannot find the file specified.

/etc/hosts file isn't used in windows:

// Kubernetes only mounts on /etc/hosts if:
// - container is not an infrastructure (pause) container
// - container is not already mounting on /etc/hosts
// - OS is not Windows
// Kubernetes will not mount /etc/hosts if:
// - when the Pod sandbox is being created, its IP is still unknown. Hence, PodIP will not have been set.
mountEtcHostsFile := len(podIPs) > 0 && runtime.GOOS != "windows"

pushing a fix for this as well

@k8s-ci-robot k8s-ci-robot added size/S area/kubelet sig/node and removed size/XS labels Feb 19, 2021
@jingxu97
Copy link
Contributor

@jingxu97 jingxu97 commented Feb 19, 2021

is that possible to add a test for it?

@ehashman ehashman added this to Needs Reviewer in SIG Node PR Triage Feb 22, 2021
@k8s-ci-robot k8s-ci-robot added area/test sig/testing labels Feb 23, 2021
@erismaster
Copy link

@erismaster erismaster commented Mar 8, 2021

Hi 👋 Bug Triage here, just a reminder that code freeze is end of day, tomorrow the 9th! PRs need to be merged in before then, or they will require an approved exception to go into the 1.21 milestone.

@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 8, 2021

/retest

@marosset
Copy link
Contributor

@marosset marosset commented Mar 8, 2021

/test pull-kubernetes-e2e-aks-engine-azure-windows
/test pull-kubernetes-e2e-aks-engine-windows-contianerd

@marosset
Copy link
Contributor

@marosset marosset commented Mar 8, 2021

/test pull-kubernetes-e2e-aks-engine-windows-containerd

@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 8, 2021

The flakes for tests pull-kubernetes-e2e-azure-file-windows-containerd and /test pull-kubernetes-e2e-azure-disk-windows should be resolved by #95018

@jsturtevant jsturtevant force-pushed the windows-host-stats-provider branch from b688199 to c9eff4e Compare Mar 8, 2021
@marosset
Copy link
Contributor

@marosset marosset commented Mar 8, 2021

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Mar 8, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 8, 2021

/test pull-kubernetes-node-e2e

@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 8, 2021

Without these updates the kubelet logs for Windows are really noisy with error messages. Hope we can get it in for into the release. Thanks!

/assign @jingxu97 @dashpole @saad-ali @ehashman

@jingxu97
Copy link
Contributor

@jingxu97 jingxu97 commented Mar 9, 2021

/approve

@jingxu97
Copy link
Contributor

@jingxu97 jingxu97 commented Mar 9, 2021

/assign @Random-Liu @yujuhong

@marosset
Copy link
Contributor

@marosset marosset commented Mar 9, 2021

/assign @saad-ali @dchen1107

@dchen1107
Copy link
Member

@dchen1107 dchen1107 commented Mar 10, 2021

@saad-ali will take a look soon. Thanks!

Copy link
Member

@saad-ali saad-ali left a comment

/lgtm
/approve

@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 10, 2021

Thanks @saad-ali @jingxu97

@dchen1107 it looks like we need an approval from Kubelet owners as well. Would you be able to take a quick look?

@mrunalp
Copy link
Contributor

@mrunalp mrunalp commented Mar 10, 2021

/approve

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Mar 10, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jingxu97, jsturtevant, mrunalp, saad-ali

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved label Mar 10, 2021
@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 10, 2021

pull-kubernetes-e2e-aks-engine-azure-windows is a known test issue with an open fix: #99860

looks like pull-kubernetes-integration is unrealted and has flake open: #98606
/test pull-kubernetes-integration

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Mar 10, 2021

@jsturtevant: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-azure-file-windows-containerd b688199f18f76cd497524ed4f6c1cac9d98abeed link /test pull-kubernetes-e2e-azure-file-windows-containerd
pull-kubernetes-e2e-aks-engine-azure-windows b688199f18f76cd497524ed4f6c1cac9d98abeed link /test pull-kubernetes-e2e-aks-engine-azure-windows

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jsturtevant
Copy link
Contributor Author

@jsturtevant jsturtevant commented Mar 10, 2021

/retest

@ehashman ehashman moved this from Needs Approver to Done in SIG Node PR Triage Mar 10, 2021
@k8s-ci-robot k8s-ci-robot merged commit 4b6e3e1 into kubernetes:master Mar 10, 2021
20 checks passed
SIG-Windows automation moved this from In Review (v1.21) to Done (v1.21) Mar 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved area/kubelet area/test cncf-cla: yes kind/bug lgtm priority/important-soon release-note sig/node sig/storage sig/testing sig/windows size/L triage/accepted
Development

Successfully merging this pull request may close these issues.