Skip to content

Log parent quota#8390

Merged
roboquat merged 1 commit into
mainfrom
fo/p-quota
Feb 22, 2022
Merged

Log parent quota#8390
roboquat merged 1 commit into
mainfrom
fo/p-quota

Conversation

@Furisto
Copy link
Copy Markdown
Member

@Furisto Furisto commented Feb 22, 2022

Description

Log the quota of the parent. Today with https://github.com/gitpod-io/ops/issues/1244 and in yesterdays incident we saw that the parent value can be the reason for not being able to set cgroup values. It should therefore be included in the error to speed up analysis.

Related Issue(s)

n.a.

How to test

  • Edit ws-daemon config-map
 "cpulimit": {
          "enabled": true,
          "totalBandwidth": "12",
          "limit": "2",
          "burstLimit": "6",
          "controlPeriod": "15s",
          "cgroupBasePath": "/mnt/node-cgroups"
        },
  • Edit ws-manager config map
  "limits": {
              "cpu": "4",
              "memory": "",
              "ephemeral-storage": ""
            }
  • Restart ws-daemon and ws-manager
  • Open workspace and run stress in the workspace

Release Notes

NONE

@Furisto Furisto requested a review from a team February 22, 2022 15:17
@github-actions github-actions Bot added the team: workspace Issue belongs to the Workspace team label Feb 22, 2022
@kylos101
Copy link
Copy Markdown
Contributor

@Furisto may I ask you to share how you tested? I'm not sure if this is something you can do in core-dev, or workspace-preview?

@Furisto
Copy link
Copy Markdown
Member Author

Furisto commented Feb 22, 2022

@kylos101 I think you could overwrite the configmap of the ws-manager. Let me check...

@Furisto
Copy link
Copy Markdown
Member Author

Furisto commented Feb 22, 2022

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot set CFS quota of 600000 (period is 100000, parent quota is 400000):\n github.com/gitpod-io/gitpod/ws-daemon/pkg/cpulimit.CgroupCFSController.SetLimit\n github.com/gitpod-io/gitpod/ws-daemon/pkg/cpulimit/cfs.go:55\n - write /mnt/node-cgroups/cpu/kubepods/burstable/pod3c3aed9f-859c-4880-a5ee-3a228c197dc3/bfb29396e078697c68937e6ef81a64763b6bad965c3b75f3b565c100a7b8103c/cpu.cfs_quota_us: invalid argument","instanceId":"91cfb18d-2c2a-4cae-afa9-e44710eef399","level":"warning","message":"cannot set CPU limit","serviceContext":{"service":"ws-daemon","version":"commit-4c27eb3eb8d2689ee94aae2529078e131bfd55f2"},"severity":"WARNING","time":"2022-02-22T17:24:58Z","userId":"","workspaceId":"gitpodio-templatetypescr-zgeoncnwteu"}

@Furisto
Copy link
Copy Markdown
Member Author

Furisto commented Feb 22, 2022

@kylos Updated test instructions

@roboquat roboquat merged commit 7ca6184 into main Feb 22, 2022
@roboquat roboquat deleted the fo/p-quota branch February 22, 2022 18:16
@roboquat roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Feb 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployed: workspace Workspace team change is running in production deployed Change is completely running in production release-note-none size/S team: workspace Issue belongs to the Workspace team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants