add debug or FAQ section to testing.md

[tpepper]

The test landscape is complex for new comers. As @spiffxp described at KubeCon in the sig-testing meetup there is a subset of tests that is highly likely to fail. A simple PR that hits these types of failures can turn away new contributors, or cause them to take actions that increase infrastructure load (eg: re-run the coin-flip equivalent test until a heads is achieved or periodically rebase hoping a fix is now present and the test passes). A FAQ or debug section in https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md would be useful to guide contributors toward understanding failures, seeing if they're hitting a top current flake (eg: http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1), checking if there is an https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Akind%2Fflake issue is already tracking the problem or if they should "/kind flake", enable them to find and watch for the specific change they need to see merged before bothering to "/retest", etc.

[tpepper]

/sig testing

[tpepper]

/sig contributor-experience

[tpepper]

@guineveresaenger as discussed at KubeCon: If this makes sense and is put in place, probably a good link to hit in the new contrib guide.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[nikhita]

/remove-lifecycle rotten
/area contributor-guide

@tpepper @spiffxp I'm guessing this is still needed, right?

/cc @castrojo

[MikeSpreitzer]

I agree this is a big problem are for new, and even not-so-new, contributors. One additional degree of difficulty comes from the fact that sometimes the failure is in the test harness itself rather than the code being tested; as far as I can tell, the developer is given no clean delineation between these kinds of failures.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

Somewhat related: #2759

[spiffxp]

/area deflake

Guess what, a lot of this content once existed in k/community, and then got moved into k/sig-release/ephemera. (ref: kubernetes/sig-release#428)

In lieu of debugging/troubleshooting (which could cover a lot of things IMO) I would suggest a "test failure triage" guide or something, although I admit "troubleshooting" is probably a more common phrase.

IMO would best live in the contributor guide, because it's sort of a specific kind of issue triage.

[spiffxp]

/priority important-soon
/kind documentation

[spiffxp]

/area deflake
(now that the label exists)

[mariantalla]

Can I take this one? I'm hoping to PR some additions as part of kubernetes/sig-release#428.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[guineveresaenger]

/remove-lifecycle stale

@mariantalla is this still something you're working on?

[mariantalla]

Hey @guineveresaenger - I've added all I had in mind with #3143. Does that cover everything under the scope of the issue?

[guineveresaenger]

Awesome, will take a look soon!

[guineveresaenger]

ah, heh! yes! we can totally close this. Thanks for pinging!
/close

[k8s-ci-robot]

@guineveresaenger: Closing this issue.

In response to this:

ah, heh! yes! we can totally close this. Thanks for pinging!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.