Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds custom reporter and logic to generate conformance docs and list of tests #85000

Open
wants to merge 1 commit into
base: master
from

Conversation

@johnSchnake
Copy link
Contributor

johnSchnake commented Nov 8, 2019

What type of PR is this?
/kind documentation

What this PR does / why we need it:
The existing walk.go and conformance.txt have a few shortcomings
which we'd like to resolve:

  • difficult to get the full test name due to test context nesting
  • complicated AST logic and understanding necessary due to the
    different ways a test can be invoked and written

This changes the AST parsing logic to be much more simple and simply
looks for the comments at/around a specific line. This file/line
information (and the full test name) is gathered by a custom ginkgo
reporter which dumps the SpecSummary data to a file.

Also, the SpecSummary dump can, itself, be potentially useful for
other post-processing and debugging tasks.

Which issue(s) this PR fixes:
Fixes #84330

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 8, 2019

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: johnSchnake
To complete the pull request process, please assign smarterclayton
You can assign the PR to them by writing /assign @smarterclayton in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@johnSchnake johnSchnake force-pushed the johnSchnake:documenterReporter branch from ee8f410 to 600ba7f Nov 8, 2019
The existing walk.go and conformance.txt have a few shortcomings
which we'd like to resolve:
 - difficult to get the full test name due to test context nesting
 - complicated AST logic and understanding necessary due to the
different ways a test can be invoked and written

This changes the AST parsing logic to be much more simple and simply
looks for the comments at/around a specific line. This file/line
information (and the full test name) is gathered by a custom ginkgo
reporter which dumps the SpecSummary data to a file.

Also, the SpecSummary dump can, itself, be potentially useful for
other post-processing and debugging tasks.

Signed-off-by: John Schnake <jschnake@vmware.com>
@johnSchnake johnSchnake force-pushed the johnSchnake:documenterReporter branch from 600ba7f to c51ef5a Nov 8, 2019
// filenames have `/go/src/k8s.io` prefix which dont exist locally
for i, v := range frames {
frames[i].File = strings.Replace(v.File,
"/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes",

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Nov 8, 2019

Author Contributor

I need some additional 👀 on this part; this was hacked together based on my own work but I'm not sure if I can always expect the same prefix or what I should be using to strip this prefix as appropriate.

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

johnSchnake commented Nov 8, 2019

Flow

Check out my branch

cd ~/go/src/k8s.io/kubernetes

GITHUBUSER=johnSchnake
BRANCH=documenterReporter

git remote add $GITHUBUSER http://github.com/$GITHUBUSER/kubernetes.git
git fetch -a $GITHUBUSER
git checkout -b $BRANCH $GITHUBUSER/$BRANCH

FYI using this bazel version

cd ~/go/src/k8s.io/kubernetes
curl -L --output bazel https://github.com/bazelbuild/bazel/releases/download/0.29.1/bazel-0.29.1-linux-x86_64
chmod +x bazel

Bazel target (updated): list-conformance-tests

cd ~/go/src/k8s.io/kubernetes
./bazel build //test/conformance:list_conformance_tests

Confirm output

cat  bazel-bin/test/conformance/conformance.txt|yq -r .[0:2]

[
  {
    "testname": "Pod Lifecycle, post start exec hook",
    "codename": "[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart exec hook properly [NodeConformance] [Conformance]",
    "description": "When a post start handler is specified in the container lifecycle using a ‘Exec’ action, then the handler MUST be invoked after the start of the container. A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a post start that invokes the server pod using ExecAction to validate that the post start is executed.",
    "release": "v1.9",
    "file": "test/e2e/common/lifecycle_hook.go"
  },
  {
    "testname": "Pod Lifecycle, post start http hook",
    "codename": "[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance]",
    "description": "When a post start handler is specified in the container lifecycle using a HttpGet action, then the handler MUST be invoked after the start of the container. A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a post start that invokes the server pod to validate that the post start is executed.",
    "release": "v1.9",
    "file": "test/e2e/common/lifecycle_hook.go"
  }
]

Bazel target (new): gen-conformance-docs

 cd ~/go/src/k8s.io/kubernetes
 ./bazel build //test/conformance:gen_conformance_docs
bazel-bin/test/conformance/conformance.md

tests/conformance/conformance.md

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

johnSchnake commented Nov 9, 2019

/test pull-kubernetes-e2e-gce

@johnSchnake johnSchnake changed the title WIP Documenter reporter Adds custom reporter and logic to generate conformance docs and list of tests Nov 9, 2019
@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

johnSchnake commented Nov 9, 2019

The pull-kubernetes-conformance-image-test is currently broken in general; nothing about this PR: https://prow.k8s.io/job-history/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-conformance-image-test

@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

johnSchnake commented Nov 9, 2019

/assign @hh

@fejta

This comment has been minimized.

Copy link
Contributor

fejta commented Nov 9, 2019

/assign cheftako spiffxp
/uncc
/retest

@k8s-ci-robot k8s-ci-robot removed the request for review from fejta Nov 9, 2019
@johnSchnake

This comment has been minimized.

Copy link
Contributor Author

johnSchnake commented Nov 10, 2019

@hh

This comment has been minimized.

Copy link
Member

hh commented Nov 21, 2019

/assign @johnbelamaric

@hh

This comment has been minimized.

Copy link
Member

hh commented Nov 21, 2019

/retest

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Nov 21, 2019

@johnSchnake: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-conformance-image-test c51ef5a link /test pull-kubernetes-conformance-image-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.