Revert "Revert "[Re-Apply][Distroless] Convert the GCE manifests for master containers.""

[yuwenma]

We fixed the duplicate log issue in klog (kubernetes/klog#65). The new klog release (v0.3.2) is introduced to k/k in #78465.

Does this PR introduce a user-facing change?:

NONE

What type of PR is this?

/kind api-change
/kind bug

/kind cleanup

/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

[yuwenma]

/hold (Wait until #78465 is merged)
/assign @wojtek-t @MaciekPytel
/cc @dims @tallclair

[yuwenma]

/assign @MaciekPytel This PR is a re-apply of #76396. We pushed a fix in klog.

[xichengliudui]

/test pull-kubernetes-kubemark-e2e-gce-big

[mborsz]

Could we add also --stderrthreshold=FATAL to avoid logging anything to stderr (it is the case before the change)?

My understanding is that this contributes to the issue -- any data wrote to stderr must be read by docker and from my experience docker reads that data quite slow.

I think this generates significant part of the issue.

[mborsz]

I meant adding stderrthreshold here and in other components as well.

[yuwenma]

For the new klog logic, if logtostderr and alsotostderr are both false, stderrthreshold will not be considered (since no stderr would ever happpen).
See here

In such case, no data will be written to std error whatever the log severity is.

[mborsz]

I see. Makes sense.

[mborsz]

I think this change makes sense. I think we should add --stderrthreshold=FATAL as well to avoid logging anything to stderr (it is the case before the change), which is redirected to pipe with docker on the other side. From my experience docker usually reads data quite slowly from that pipe so this can lead to potential scalability issues.

Another thing is that given we have seen that changes like that can affect performance of 5k node tests, I suggest running manual test first before we merge this. Could you run something like that?

[yuwenma]

I think this change makes sense. I think we should add --stderrthreshold=FATAL as well to avoid logging anything to stderr (it is the case before the change), which is redirected to pipe with docker on the other side. From my experience docker usually reads data quite slowly from that pipe so this can lead to potential scalability issues.

Another thing is that given we have seen that changes like that can affect performance of 5k node tests, I suggest running manual test first before we merge this. Could you run something like that?

Can you give some guidance how to run a 5k node tests?
Just a reminder that we are hitting the v1.15 code freeze (EOD this week). And there's another PR blocked by this change. Is there any chance to get this PR go in today?

[mborsz]

In my opinion this PR should work, but I really prefer testing this before we submit to avoid third revert.

/test pull-kubernetes-e2e-gce-large-performance

This should test this PR in 2k scale. 5k scale would be better, but we don't have resources until the end of the next week to test this in 5k scale.

[yuwenma]

/test pull-kubernetes-e2e-gce-large-performance

[mborsz]

I'm afraid the first test failure is not a flake.

I took a look at logs and I see few problems there:

• kube-apiserver is restarting few times

➜  e2e-021a5abcf8-a7a7f-master zgrep -- --address= kube-apiserver.log*
kube-apiserver.log:I0530 01:41:22.465795       1 flags.go:33] FLAG: --address="127.0.0.1"
kube-apiserver.log-20190530-1559174412.gz:I0529 22:21:45.415090       1 flags.go:33] FLAG: --address="127.0.0.1"
kube-apiserver.log-20190530-1559174412.gz:I0529 23:41:21.603442       1 flags.go:33] FLAG: --address="127.0.0.1"
➜  e2e-021a5abcf8-a7a7f-master

• some log entries are still repeated, e.g.:

➜  e2e-021a5abcf8-a7a7f-master zgrep 'E0529 23:41:19.450955' kube-apiserver.log*
kube-apiserver.log-20190530-1559174412.gz:E0529 23:41:19.450955       1 metrics.go:96] Error in audit plugin 'buffered' affecting 1 audit events: audit backend shut down
kube-apiserver.log-20190530-1559174412.gz:E0529 23:41:19.450955       1 metrics.go:96] Error in audit plugin 'buffered' affecting 1 audit events: audit backend shut down
kube-apiserver.log-20190530-1559174412.gz:E0529 23:41:19.450955       1 metrics.go:96] Error in audit plugin 'buffered' affecting 1 audit events: audit backend shut down

[mborsz]

I see what happened, this PR doesn't contain klog version update which happens in #78465

We need to rebase this PR to contain that commit and rerun the test, at least now we know that 2000 node tests reproduces the issue we saw in 5k node scale.

[mborsz]

As soon as pull-kubernetes-e2e-gce-large-performance passes I think it's good to submit.

/lgtm
/approve
/hold

[dims]

/test pull-kubernetes-e2e-gce-large-performance

[dims]

let's try this again for 1.17

/hold cancel

[dims]

/test pull-kubernetes-e2e-gce-large-performance

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[dims]

W0919 10:10:30.497] ERROR: (gcloud.compute.instances.create) Could not fetch resource:
W0919 10:10:30.498]  - Quota 'CPUS' exceeded.  Limit: 5200.0 in region us-east1.

:(

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[dims]

/lgtm

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[wojtek-t]

/hold

Holding for a moment given that we currently have a regression (since Friday). Forfunately we seem to know where the problem is already - an issue will be opened later today.

[dims]

@wojtek-t how are things? could we try again?

[wojtek-t]

We recovered from the regression. Maybe we can try.

/hold cancel

@mborsz @mm4tt - FYI