Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finish saving test results on failure #76039

Merged
merged 1 commit into from Apr 3, 2019

Conversation

@johnSchnake
Copy link
Contributor

commented Apr 2, 2019

The conformance image should be saving its results
regardless of the results of the tests. However,
with errexit set, when ginkgo gets test failures
it exits 1 which prevents saving the results
for Sonobuoy to pick up.

Fixes: #76036

/kind bug

Special notes for your reviewer:
Currently working on building this image and testing it works as expected.

Does this PR introduce a user-facing change?:

Ensures the conformance test image saves results before exiting when ginkgo returns non-zero value.
@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

/sig testing
/area conformance

@timothysc
Copy link
Member

left a comment

/approve
/lgtm

Please follow up with cherry-pick PR for 1.14 branch

@k8s-ci-robot k8s-ci-robot added the lgtm label Apr 2, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnSchnake, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

/test pull-kubernetes-integration

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

Manually confirmed that the patch does fix the error case mentioned in the ticket; a failing ginkgo run still ends up saving the results and getting marked as completed via Sonobuoy.

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

commented Apr 2, 2019

/test pull-kubernetes-e2e-gce

@ixdy
Copy link
Member

left a comment

/hold

@@ -13,7 +13,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.

set -o errexit

This comment has been minimized.

Copy link
@ixdy

ixdy Apr 2, 2019

Member

it'd be better to change the wait line to be something like

wait "$(pgrep ginkgo)" && ret=0 || ret=$?

and then the last line of this script would be

exit ${ret}

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Apr 3, 2019

Author Contributor

To clarify to ensure I understand correctly,

is the reason you want to do that so that it isn't just customized for Sonobuoy (waiting for the done file) but also a consumer could wait for the non-zero exit code for indication of pass/fail?

I guess those two things are at odds:

  • if we set errexit and the script fails anywhere results wont be "published" and Sonobuoy would hang
  • if we dont set errexit and the script fails anywhere the exit code wont be properly reported and it might report success

If my understanding above is accurate, I agree that the lesser evil is to do as you suggest. A hang would at least be indicative of a problem whereas a false-positive is a larger issue that could go undiscovered.

Also +1/TIL to that particular bash logic; I dont think I'd seen that exact trick used before but it seems pretty nifty.

This comment has been minimized.

Copy link
@ixdy

ixdy Apr 3, 2019

Member

we generally like keeping errexit on scripts so that we fail fast on syntax errors or other issues - you're sorta required to handle all error cases, or everything will just fail.

another alternative if we want to make sure logs are always uploaded is to add something like

trap saveResults EXIT

and then you wouldn't even need to explicitly call saveResults - it'd always run on script exit.

Finish saving test results on failure
The conformance image should be saving its results
regardless of the results of the tests. However,
with errexit set, when ginkgo gets test failures
it exits 1 which prevents saving the results
for Sonobuoy to pick up.

Fixes: #76036

@johnSchnake johnSchnake force-pushed the johnSchnake:conformanceErrExit branch from 8669cd6 to 4fec7c7 Apr 3, 2019

@k8s-ci-robot k8s-ci-robot removed the lgtm label Apr 3, 2019

@dims

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

/lgtm
/hold cancel

@k8s-ci-robot k8s-ci-robot added lgtm and removed do-not-merge/hold labels Apr 3, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

/priority important-soon

@k8s-ci-robot k8s-ci-robot merged commit 41691a0 into kubernetes:master Apr 3, 2019

17 checks passed

cla/linuxfoundation johnSchnake authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Job succeeded.
Details
pull-kubernetes-cross Skipped.
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

@johnSchnake johnSchnake deleted the johnSchnake:conformanceErrExit branch Apr 3, 2019

k8s-ci-robot added a commit that referenced this pull request Apr 4, 2019

Merge pull request #76101 from johnSchnake/automated-cherry-pick-of-#…
…76039-upstream-release-1.14

Automated cherry pick of #76039: Finish saving test results on failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.