Missing documents for what senarios are being tested

[mohammedzee1000]

[kind/Documentation]

What did you find missing in the documentation?

There is no document that provides a broad overview of what is being tested, and to what extent it is being tested. What i mean by this is a document that lists:

1. The test scenarios of odo that are being tested

2. Against which environments, are each of the scenarios tested in?

3. Against which versions of OpenShift are each of the scenarios tested against?

4. Under which category of testing does this happen (e2e/integration/etc)?

Note: I am talking actual scenarios and not just commands (although a scenario could be a single command). In fact some tests, may not even involve running commands

Already looked at #430 and while the table there says which commands are tested or not to an extent by e2e it does not give the extent.

What is the relevence of it?

Having a detailed view will help us know how far we are, from having a decently tested project, which we can vouch for without having to go look at actual code. It should also give us a biases to say go and talk to OpenShift team to get tests added to prow there, allowing us to coordinate better with OpenShift team.

[amitkrout]

There is no document that provides a broad overview of what is being tested, and to what extent it is being tested. What i mean by this is a document that lists: 1. The test scenarios of odo that are being tested

Yes, the test scenario are documented but not in that impressive way (Mixture of context and implementation in same file). Unlike other gherkin supported test frame work, GinkGo does not support in that way where we have doc file in one place (.feature file) and backend implementation in another file (.go file). Even there is no such plan for adding gherkin support [5] in GinkGo. So I have three suggestions atm.

[5] onsi/ginkgo#535

1. Isolate the implementation code and Context as much as possible by keeping the implementation code in a separate file.
2. Create package for individual test and put doc file test-coverage.md along with its .go implementation file including changes mentioned in point 1. But in this case maintaining the test doc file will be very painful.

The third option is hard or near to impossible atm but I can still see this is a best option as part of test documentation is concerned

3. Switch to a BDD framework that supports BDD+gherkin

2. Against which environments, are each of the scenarios tested in?3. Against which versions of OpenShift are each of the scenarios tested against?

Good to have these info in somewhere Odo doc (Could be a new doc file or in an existing doc file) if not yet documented.

4. Under which category of testing does this happen (e2e/integration/etc)?

Could not get it. Would you share what is your test expectation in the category of e2e and integration test. An example would really be helpful

Already looked at #430 and while the table there says which commands are tested or not to an extent by e2e it does not give the extent.

Right, This is just a premature test doc file

[girishramnani]

we have tests running on 3 latest versions of openshift. ( v4.3,2.1 at the time of writing ) and have neatly defined spec and can run tests which are scoped into "component", "storage", "url" etc.
In my opinion, the scope of this issue has been implemented. Hence closing. If someone feels there is more need to close this issue then please re-open this issue.