Initial docs about writing extended test

[mfojtik]

(this is just a draft ;-)

TODO

• Describe CLI interface (oc.Run())
• Describe Kubernetes E2E framework (point to godoc?)
• Provide example extended test with explanation
• Add more todo items

[stevekuznetsov]

link to the dir?

[**`test/extended/util`**](../test/extended/util)

[stevekuznetsov]

Add some information and a link to this README after this section here.

[stevekuznetsov]

Add the how-to:
go get github.com/onsi/ginkgo/ginkgo

[jhadvig]

@mfojtik I've added the CLI interface part. PTAL

[mfojtik]

@stevekuznetsov thanks! I moved the docs into README.md

[gabemontero]

s/OpenShift helpers are/OpenShift helpers that are/

[bparees]

@mfojtik this looks like a good start, one thing i'd add is an explanation of how the whole flow works, ie starting with the test-extend.sh script, what parameters that takes to decide what to run, etc.

[gabemontero]

We may want to have a ref to the common bash functions in hack/util.sh
that hack/test-extended/default/run.sh and
hack/test-extended/forcepull/run.sh now leverage.

Thoughts @bparees @mfojtik @jhadvig ?

On Fri, Aug 21, 2015 at 1:13 PM, Ben Parees notifications@github.com
wrote:

@mfojtik https://github.com/mfojtik this looks like a good start, one
thing i'd add is an explanation of how the whole flow works, ie starting
with the test-extend.sh script, what parameters that takes to decide what
to run, etc.

—
Reply to this email directly or view it on GitHub
#4258 (comment).

[bparees]

@gabemontero +1

[bparees]

one more thing to add: how to run a specific test within a group, assuming that's possible...

[mfojtik]

@bparees @gabemontero thanks! I going to add all docs you mention

[bparees]

@mfojtik is there a way to run a single test (single .go file or single Describe within it?) or is "group" the finest level of granularity?

[bparees]

name the file where this is bundled.

[gabemontero]

Only had 4 minor typo kind of corrections on top of the @bparees comment about listing the location of the base helpers.... pending further input from others, igtm

[stevekuznetsov]

@bparees The run scripts for extended tests are currently changing - they will soon have the following syntax:
hack/test-extended.sh bucket filter where bucket defines the bucket to be run, and filter is piped into gingko -focus= which runs any tests that match the filter. Unless we create some sort of look-up table from test to bucket, though, you will not be able to specify a test by itself and instead need to specify its' bucket. I assume you would be able to specify all buckets, but unless there is a ginkgo dry-run (@mfojtik ??), we don't have a way of finding if a gikgo -focus matches any tests in a bucket and instead would need to do the entire bucket set-up just to realize there are no tests to run.

[bparees]

having just written my first extended test (yay!), a couple things i found confusing related to buckets/focuses...

we have scripts like:
hack/test-extended/forcepull/run.sh

which runs ginkgo with the "forcepull" focus

but then forcepull.go is located under "test/extended/builds/forcepull.go"

and then everything else under "builds" is under the "default" focus.

If i wanted to figure out what "hack/test-extended/forcepull/run.sh" was going to run, i'd have intuitively looked for "test/extended/forcepull/somefiles.go"

it seems like a bit of a mish-mash. intuitively i'd expect that tests with the same "focus" would live in the same directory and there would be some script i could run (or argument to a script) which would run all the tests in that directory. (again ideally i'd be able to pick a specific test from a specific directory, too)

@stevekuznetsov what defines a "bucket" in this impending refactor? perhaps that will make it more like what i was thinking...

[gabemontero]

Yeah I originally had a separate forcepull subdir but then switched based on comment from @mfojtik. Forcepull is still build related (hence same dir) but needs to run separately (hence a different focus). Admittedly without understanding that distinction it seems confusing. I ultimately don't feel strongly either way, but did like the idea of exposing forcepull at the subdir / group level.

Gabe

On Aug 24, 2015, at 6:34 PM, Ben Parees notifications@github.com wrote:

having just written my first extended test (yay!), a couple things i found confusing related to buckets/focuses...

we have scripts like:
hack/test-extended/forcepull/run.sh

which runs ginkgo with the "forcepull" focus

but then forcepull.go is located under "test/extended/builds/forcepull.go"

and then everything else under "builds" is under the "default" focus.

If i wanted to figure out what "hack/test-extended/forcepull/run.sh" was going to run, i'd have intuitively looked for "test/extended/forcepull/somefiles.go"

it seems like a bit of a mish-mash. intuitively i'd expect that tests with the same "focus" would live in the same directory and there would be some script i could run (or argument to a script) which would run all the tests in that directory. (again ideally i'd be able to pick a specific test from a specific directory, too)

@stevekuznetsov what defines a "bucket" in this impending refactor? perhaps that will make it more like what i was thinking...

—
Reply to this email directly or view it on GitHub.

[mfojtik]

@bparees @gabemontero right, it is a little bit confusing, but I don't want to introduce a lot of extra Go packages for different test groups. Especially when the tests really are 'builds' test or 'images' tests.

I think one solution could be to use --dry-run option when executing the tests (or provide that option to the shell launcher) to see what tests are going to be executed.

[bparees]

@mfojtik if the only alternative is a lot of go packages, i agree, just wanted to raise the discussion if only to ensure it gets documented clearly.

[mfojtik]

[merge]

We will improve these docs as we go.

[openshift-bot]

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_origin/3119/) (Image: devenv-fedora_2222)

[openshift-bot]

[Test]ing while waiting on the merge queue

[openshift-bot]

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/4480/)

[openshift-bot]

Evaluated for origin test up to ebffbc9

[stevekuznetsov]

@bparees The original intent of the buckets (as per @deads2k) was functional separation - don't start up a cluster with a router if there is no router necessary for it.

Thinking more about it, however, as these are extended tests we don't particularly need them to be isolated, we care if they fail when the router is up even if they don't run the router. It would make sense to only have buckets if they have some sort of conflicting setup needs for the cluster, which I am not sure we have right now.

[bparees]

i'm not sure if buckets and focus are two separate ways of grouping things but basically there are two needs here:

1a) defining the configuration a particular set of tests need. this would include "tests that don't need a router" (though i don't see the point in grouping that separately...if you don't need a router, don't use it..don't start a whole separate openshift server w/o a router for that purpose...but perhaps there's a more legitimate use case here).

1b) tests which need to be run serially, not in parallel (the forcepull test which mucks with the images on the host filesystem and therefore could screw up other tests running at the same time is an example of this)

2. grouping by functional area...if i'm changing build code, i want an easy way to run the build-related extended tests, not the entire bucket of everything which is going to be huge and slow.

So whatever we're driving at, i'd expect it to provide those 2.5 levels of division. And it should be easy for a user to at least understand how to run buckets grouped by (2).

As a user it's probably not as important I be able to run "all the tests that don't need a router" or "all the tests that can't be run in parallel".

so if we're already there, great..if not, let's figure out how to restructure things so we can get those characteristics. And if we can do it in such a way that the directory structure/scripts make more logical sense, even better :)

[deads2k]

I only care about functional area. Ben's item 2.

I see the parallel/serial and configuration as the responsibility of the individual bucket owner. Maybe interesting, but not essential or particularly useful in all cases.

I'd like to see the bash for startup behave similarly to the integration tests where we make a config that is easy to mutate and then start it. That should help dry out the scripts and reduce pain without making painful, prescriptive metadata that doesn't really help reduce overall complexity.

[mfojtik]

re [merge]

[openshift-bot]

Evaluated for origin merge up to ebffbc9