P0 - because it makes any debugging roughly impossible.

kubernetes/test-infra#777

@rmmh is working on this

It seems that after fixes from yesterday, we are still lacking logs from all kubemark runs. @gmarek

Lack of kubemark logs is significantly slowing down any scalability-related work, which is our P0 goal for this quarter, so please prioritize it.
@fgrzadkowski - FYI ^^

As an example:
https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-gce-scale/1473/?project=kubernetes-jenkins
[is a run from an hour ago] and it contains only started.json file.

I'm going to migrate this job over to where our code (rather than a jenkins plugin) captures and uploads logs, which we want to do anyway and will hopefully resolve this.

Odd... if we look at job/kubernetes-e2e-gce/configure we will a bunch of conditional build steps in the post-build actions configuration, which matches what is specified in yaml:
https://github.com/kubernetes/test-infra/blob/master/jenkins/job-configs/kubernetes-jenkins/kubernetes-e2e-gce.yaml#L46

We also have this specified in kubemark yaml:
https://github.com/kubernetes/test-infra/blob/master/jenkins/job-configs/kubernetes-jenkins/kubernetes-kubemark.yaml#L26

However I don't see where this is applied in the actual configuration: /job/kubernetes-kubemark-gce-scale/configure

I suspect that the job updater ran at some point this weekend while the conditional build step plugin wasn't working, so it wasn't able to add that step. I'm not sure why it hasn't rectified it since.

That seems to be the problem. I've cleared the job cache and done a full rebuild, so gce-scale and anything else that was updated why the postbuild plugin was missing should be fixed now.

Hmm - it doesn't seem to be fully fixed.

Basically, we used to have logs from kubemark master machine, e.g.:
https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-500-gce/6550/artifacts/?project=kubernetes-jenkins
(there is this kubemark-500-kubemark-master/ directory)

However, even now (this is the last run), we no longer have those logs:
https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-500-gce/6686/artifacts/?project=kubernetes-jenkins

So something is still broken...

Any thoughts about it? The logs are still missing, e.g.:
https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-gce-scale/1477/artifacts/?project=kubernetes-jenkins

All the files in _artifacts are being properly uploaded. Something else is failing to copy the logs onto the node.

In gce-scale/1477

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSDumping master and node logs to /workspace/_artifacts
Copying 'kern kube-apiserver kube-scheduler kube-controller-manager etcd glbc cluster-autoscaler docker kubelet supervisor/supervisord supervisor/kubelet-stdout supervisor/kubelet-stderr supervisor/docker-stdout supervisor/docker-stderr' from kubemark-2000-kubemark-master
Node SSH not supported for kubemark

In gce-scale/1465, it seems to be copying more:

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSDumping master and node logs to /workspace/_artifacts
scp: /var/log/glbc.log*: No such file or directory
scp: /var/log/cluster-autoscaler.log*: No such file or directory
scp: /var/log/supervisor/kubelet-stdout.log*: No such file or directory
scp: /var/log/supervisor/kubelet-stderr.log*: No such file or directory
scp: /var/log/supervisor/docker-stdout.log*: No such file or directory
scp: /var/log/supervisor/docker-stderr.log*: No such file or directory
ERROR: (gcloud.compute.copy-files) [/usr/bin/scp] exited with return code [1]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
Node SSH not supported for kubemark

Did something else change in kubemark between these time periods? Is scp from the master broken now? This is a separate issue.

I'm not aware of any other changes.
@gmarek - we should try to debug it tomorrow.

We did touch log-dump.sh recently in #34153. Might be related.

Yeah, I was just trying to work out if it was. I tried to keep the fake kubemark path alive as best I could.

@zmerlynn - if you could figure out if it was because of your PR (timing matches) that would be great...

@wojtek-t: #34647

I confirm that this is fixed now.
@zmerlynn - thanks a lot for fixing this!