podutils: Add more verbose metadata

[cblecker]

What would you like to be added:
podutils jobs don't provide much in the way of metadata details in either their started or finished json files.

bootstrap + scenario example with metadata:
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/72177/pull-kubernetes-verify/116969/finished.json

podutils example with very little metadata:
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/72177/pull-kubernetes-typecheck/34990/finished.json

Why is this needed:
Of particular value would be the repo commits (what is HEAD of PR branch, and what is HEAD of base branch), and the infra-commit.

/kind feature
/area prow prow/pod-utilities

[cblecker]

cc: @BenTheElder @krzyzacy @stevekuznetsov

[BenTheElder]

@krzyzacy was looking into this. I've definitely held off migrating release blocking CI in particular while we sort this out. What's the current state Sen?

[spiffxp]

/milestone v1.14
hopefully?

ref: #7364 (sortof a dupe?) and #3412 (why not doc the metadata while we're at it)

[stevekuznetsov]

Yeah I would love to see a straightforward doc of what tools use what fields ... from my external viewpoint the metadata is a huge mess of legacy right now...

[stevekuznetsov]

And nobody ever helped me understand it two years ago in #3412

[krzyzacy]

I think two things:

• gubernator & spyglass will display the metadata.
• you can also set custom column headers in testgrid from the metadata you have.

You can put whatever fields you want there, and it is not a must-have today.

[krzyzacy]

Oh didn't read the issue body - we should have all the repo info in https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/72529/pull-kubernetes-typecheck/35614/clone-records.json, maybe we can clean that up a little bit and display some info from there? @stevekuznetsov

[BenTheElder]

discussed a possible path forward on this today: we can put the resolved refs after checkout into the clone-records.json, thread here: https://kubernetes.slack.com/archives/CDECRSC5U/p1547584185236700

@cjwagner, thoughts on this? @stevekuznetsov seems OK with it, and I spoke to @krzyzacy about it.

[cjwagner]

👍
Would we then have initupload include the resolved refs from clone-records.json in started.json? Otherwise consumers will need to read clone-records.json in addition to started.json and finished.json.

[krzyzacy]

/assign
/assign @fejta

[krzyzacy]

I'll backfill more data for now, and resolve #10359 separately.

[krzyzacy]

Most of the metadata here is now backfilled in started.json, we still need to make downstream consume these. Maybe worth a separate issue.

/close

[k8s-ci-robot]

@krzyzacy: Closing this issue.

In response to this:

Most of the metadata here is now backfilled in started.json, we still need to make downstream consume these. Maybe worth a separate issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.