New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conformance annotations require build checks to ensure format is adhered to #53810

Closed
bradtopol opened this Issue Oct 12, 2017 · 13 comments

Comments

Projects
None yet
8 participants
@bradtopol
Contributor

bradtopol commented Oct 12, 2017

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

/sig area/conformance

What happened:
Kubernetes conformance annotations are comments in the e2e test suite next to tests labeled [Conformance] and are used for generating a document that describes the list of conformance tests as identified by the Kubernetes Conformance workgroup. Currently these annotation comments do not undergo a build check to ensure the annotation format is being adhered to.
What you expected to happen:
Annotation comments should undergo a build check to ensure the annotation format is being adhered to.
How to reproduce it (as minimally and precisely as possible):
Create a Conformance annotation in the e2e test suite for a conformance test and do not follow the prescribed format of

/*
Testname: pods-exec-liveness-probe-not-restarted
Description: Make sure when exec liveness probe succeeds, the pod should
not be restarted.
*/
It("should not be restarted with a exec "cat /tmp/health" liveness probe [Conformance]", func() {

Anything else we need to know?:
See PR #53742 for more details on this.

Environment:

@mml

This comment has been minimized.

Show comment
Hide comment
@mml

mml Oct 12, 2017

Contributor

FYI I'm also working on a presubmit check to verify that any addition or removal of [Conformance] tags is reviewed big sig-architecture. There may be some overlap, so if you have an implementation plan in mind, please let me know.

Thanks!

Contributor

mml commented Oct 12, 2017

FYI I'm also working on a presubmit check to verify that any addition or removal of [Conformance] tags is reviewed big sig-architecture. There may be some overlap, so if you have an implementation plan in mind, please let me know.

Thanks!

@brahmaroutu

This comment has been minimized.

Show comment
Hide comment
@brahmaroutu

brahmaroutu Oct 12, 2017

Contributor

#52863 @mml The current tooling assumes that we leverage [Conformance] tag. Can you elaborate other options?

Contributor

brahmaroutu commented Oct 12, 2017

#52863 @mml The current tooling assumes that we leverage [Conformance] tag. Can you elaborate other options?

@mml

This comment has been minimized.

Show comment
Hide comment
@mml

mml Oct 13, 2017

Contributor

I'm doing the same thing. See my half-baked PR at #53909

Contributor

mml commented Oct 13, 2017

I'm doing the same thing. See my half-baked PR at #53909

@duglin

This comment has been minimized.

Show comment
Hide comment
@duglin

duglin Oct 18, 2017

Contributor

@mml can we discuss on the conformance mailing list so that we avoid duplicate work. There might be some overlap but not completely.

Contributor

duglin commented Oct 18, 2017

@mml can we discuss on the conformance mailing list so that we avoid duplicate work. There might be some overlap but not completely.

@bradtopol

This comment has been minimized.

Show comment
Hide comment
@bradtopol

bradtopol Oct 26, 2017

Contributor

So I have been adding lots of conformance annotations to the [Conformance] tests in the e2e test suite. Here are some real world issues I ran into that are related to this issue:

In addition to make sure the comments have a test name and description it would be very helpful to list all conformance tests that have no annotation comment associated with them and to also list if two tests have the same test-name. Both could happen easier than you think and having the tool report those situations would really help to ensure the accuracy of the annotations.

Contributor

bradtopol commented Oct 26, 2017

So I have been adding lots of conformance annotations to the [Conformance] tests in the e2e test suite. Here are some real world issues I ran into that are related to this issue:

In addition to make sure the comments have a test name and description it would be very helpful to list all conformance tests that have no annotation comment associated with them and to also list if two tests have the same test-name. Both could happen easier than you think and having the tool report those situations would really help to ensure the accuracy of the annotations.

@brahmaroutu

This comment has been minimized.

Show comment
Hide comment
@brahmaroutu

brahmaroutu Nov 17, 2017

Contributor

/assign

Contributor

brahmaroutu commented Nov 17, 2017

/assign

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Nov 17, 2017

Contributor

@brahmaroutu: GitHub didn't allow me to assign the following users: brahmaroutu.

Note that only kubernetes members can be assigned.

In response to this:

/assign

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Nov 17, 2017

@brahmaroutu: GitHub didn't allow me to assign the following users: brahmaroutu.

Note that only kubernetes members can be assigned.

In response to this:

/assign

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Feb 15, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented Feb 15, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@mml

This comment has been minimized.

Show comment
Hide comment
@mml

mml Feb 15, 2018

Contributor

/remove-lifecycle stale

Contributor

mml commented Feb 15, 2018

/remove-lifecycle stale

@mml

This comment has been minimized.

Show comment
Hide comment
@mml

mml Feb 16, 2018

Contributor

@brahmaroutu you are still working on this, correct?

Contributor

mml commented Feb 16, 2018

@brahmaroutu you are still working on this, correct?

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot May 17, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot commented May 17, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jun 16, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot commented Jun 16, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jul 16, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot commented Jul 16, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment