Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Label [Skipped] tests appropriately #19297

Merged
merged 1 commit into from
Jan 7, 2016
Merged

Label [Skipped] tests appropriately #19297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
43 changes: 12 additions & 31 deletions hack/jenkins/e2e.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,20 +272,14 @@ fi
# When 1.2.0-beta.0 comes out, e.g., this will become "ci/latest-1.2"
CURRENT_RELEASE_PUBLISHED_VERSION="ci/latest-1.1"

# Specialized to skip when running reboot tests.
REBOOT_SKIP_TESTS=(
"Restart\sshould\srestart\sall\snodes"
"\[Example\]"
)

# Specialized tests which should be skipped by default for projects.
GCE_DEFAULT_SKIP_TESTS=(
"${REBOOT_SKIP_TESTS[@]}"
"\[Example\]" # previously in REBOOT_SKIP_TESTS..., dates back before version control (#10078)
"\[Skipped\]"
"Reboot"
"ServiceLoadBalancer"
)

# PROVIDER SKIPS --------------------------------------

# Tests which cannot be run on GKE, e.g. because they require
# master ssh access.
GKE_REQUIRED_SKIP_TESTS=(
Expand Down Expand Up @@ -314,6 +308,7 @@ AWS_REQUIRED_SKIP_TESTS=(
"GCE\sL7\sLoadBalancer\sController" # GCE L7 loadbalancing
)

# END PROVIDER SKIPS --------------------------------------

# Tests which kills or restarts components and/or nodes.
DISRUPTIVE_TESTS=(
Expand All @@ -333,10 +328,6 @@ GCE_FLAKY_TESTS=(
# comments below, and for poorly implemented tests, please quote the
# issue number tracking speed improvements.
GCE_SLOW_TESTS=(
# Before enabling this loadbalancer test in any other test list you must
# make sure the associated project has enough quota. At the time of this
# writing a GCE project is allowed 3 backend services by default. This
# test requires at least 5.
"\[Slow\]"
)

Expand All @@ -348,21 +339,6 @@ GCE_PARALLEL_SKIP_TESTS=(
"\[Disruptive\]"
)

# Tests that should not run on soak cluster.
GCE_SOAK_CONTINUOUS_SKIP_TESTS=(
"GCE\sL7\sLoadBalancer\sController" # issue: #17119
"Density.*30\spods"
"Elasticsearch"
"external\sload\sbalancer"
"identically\snamed\sservices"
"network\spartition"
"Services.*Type\sgoes\sfrom"
"${DISRUPTIVE_TESTS[@]}" # avoid component restarts.
)

GCE_RELEASE_SKIP_TESTS=(
)

# Define environment variables based on the Jenkins project name.
# NOTE: Not all jobs are defined here. The hack/jenkins/e2e.sh in master and
# release branches defines relevant jobs for that particular version of
Expand Down Expand Up @@ -619,10 +595,13 @@ case ${JOB_NAME} in
: ${E2E_UP:="false"}
# Clear out any orphaned namespaces in case previous run was interrupted.
: ${E2E_CLEAN_START:="true"}
# We should be testing the reliability of a long-running cluster. The
# DISRUPTIVE_TESTS kill/restart components or nodes in the cluster,
# defeating the purpose of a soak cluster. (#15722)
: ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty \
${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}} \
${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}} \
${GCE_SOAK_CONTINUOUS_SKIP_TESTS[@]:+${GCE_SOAK_CONTINUOUS_SKIP_TESTS[@]}} \
${DISRUPTIVE_TESTS[@]:+${DISRUPTIVE_TESTS[@]}} \
)"}
: ${KUBE_GCE_INSTANCE_PREFIX:="gce-soak-weekly"}
: ${PROJECT:="kubernetes-jenkins"}
Expand Down Expand Up @@ -650,7 +629,6 @@ case ${JOB_NAME} in
: ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty \
"\[Skipped\]" \
${GKE_DEFAULT_SKIP_TESTS[@]:+${GKE_DEFAULT_SKIP_TESTS[@]}} \
${REBOOT_SKIP_TESTS[@]:+${REBOOT_SKIP_TESTS[@]}} \
${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}} \
${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}} \
)"}
Expand Down Expand Up @@ -693,11 +671,14 @@ case ${JOB_NAME} in
: ${E2E_CLEAN_START:="true"}
: ${PROJECT:="kubernetes-jenkins"}
: ${E2E_OPT:="--check_version_skew=false"}
# We should be testing the reliability of a long-running cluster. The
# DISRUPTIVE_TESTS kill/restart components or nodes in the cluster,
# defeating the purpose of a soak cluster. (#15722)
: ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty \
${GKE_REQUIRED_SKIP_TESTS[@]:+${GKE_REQUIRED_SKIP_TESTS[@]}} \
${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}} \
${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}} \
${GCE_SOAK_CONTINUOUS_SKIP_TESTS[@]:+${GCE_SOAK_CONTINUOUS_SKIP_TESTS[@]}} \
${DISRUPTIVE_TESTS[@]:+${DISRUPTIVE_TESTS[@]}} \
)"}
;;

Expand Down
5 changes: 5 additions & 0 deletions test/e2e/ingress.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,11 @@ func (cont *IngressController) Cleanup(del bool) error {
return fmt.Errorf(errMsg)
}

// Before enabling this loadbalancer test in any other test list you must
// make sure the associated project has enough quota. At the time of this
// writing a GCE project is allowed 3 backend services by default. This
// test requires at least 5.
//
// Slow by design (10 min)
// Flaky issue #17518
var _ = Describe("GCE L7 LoadBalancer Controller [Serial] [Slow] [Flaky]", func() {
Expand Down
4 changes: 3 additions & 1 deletion test/e2e/restart.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ const (
restartPodReadyAgainTimeout = 5 * time.Minute
)

var _ = Describe("Restart", func() {
// TODO(ihmccreery): This is skipped because it was previously in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have an issue to track this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, but that's what my next task is, going through all the [Skipped] tests and filing issues for them.

// REBOOT_SKIP_TESTS, dates back before version control (#10078)
var _ = Describe("Restart [Skipped]", func() {
var c *client.Client
var ps *podStore
var skipped bool
Expand Down
4 changes: 3 additions & 1 deletion test/e2e/serviceloadbalancers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,9 @@ func (s *ingManager) test(path string) error {
})
}

var _ = Describe("ServiceLoadBalancer", func() {
// TODO(ihmccreery) Skipped originally in #14988, never sent follow-up PR, as
// far as I can tell.
var _ = Describe("ServiceLoadBalancer [Skipped]", func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will make this skipped by default in non-Jenkins runs, too - is this what we want?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no reason why Jenkins should be a special snowflake. Part of the benefit of this effort will be reconciling Jenkins with "real-world" runs.

// These variables are initialized after framework's beforeEach.
var ns string
var repoRoot string
Expand Down