Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bats tests destroy the namespace's non-OW resources #168

Open
joshuaauerbachwatson opened this issue Aug 25, 2021 · 2 comments
Open

Bats tests destroy the namespace's non-OW resources #168

joshuaauerbachwatson opened this issue Aug 25, 2021 · 2 comments

Comments

@joshuaauerbachwatson
Copy link
Contributor

joshuaauerbachwatson commented Aug 25, 2021

I re-titled and broadened this issue, which I opened Aug 25. The problem is worse than I then thought.

If a developer runs the bats test suite, it does at least the following destructive things to the non-OW resources. OW resources are handled more thoughtfully by using package scoping.

  1. The object tests and the kv tests include a test for the clean operation, and so the data bucket and kv store end up being wiped out.
  2. Although there is no explicit web clean operation in the tests, the web_include test has the cleanNamespace: true property in its config file, so the result is to clear the web bucket as well.

IMO the clean capabilities should not be tested in the regular bats suite because the developer may well be using the current namespace for something else. All tests involving kv, web, and object should use a dedicated sub-area named after the test, similar to how OW resources are protected.

In addition to these destructive operations, there are certain tests (like the object list test) that assume that a bucket was cleaned earlier. This was the original content of this issue. So, in revising the test suite to eliminate destructive operations, these details should be fixed as well.

@joshuaauerbachwatson joshuaauerbachwatson changed the title Object list test fails when data bucket does not start out empty Bats tests are destructive of the namespace's buckets and (falsely) assume total ownership of them Oct 4, 2021
@joshuaauerbachwatson joshuaauerbachwatson changed the title Bats tests are destructive of the namespace's buckets and (falsely) assume total ownership of them Bats tests destroy the namespace's non-OW resources Oct 4, 2021
@joshuaauerbachwatson
Copy link
Contributor Author

I recognize that it is useful for the tests to be comprehensive and it might be desirable for them to assume complete ownership of a test namespace. That would be an alternative fix for this problem. BUT, in that case:

  1. The developer should be instructed to create a special namespace for running the tests.
  2. The identity of that namespace should be obtained by the tests from the environment or by some other automated means.
  3. The current namespace should not be assumed to be available for arbitrary alteration.

@joshuaauerbachwatson
Copy link
Contributor Author

The 'object list' test was failing routinely, which is annoying when trying to assess the health of the code after a commit.

I put in a skip for it stating the reason.

This does not address the general problem of this issue. The skip should probably be replaced by a rewrite of the test, along with removing the destructive behavior of other tests. This is not urgent since we are not getting outside submissions of PRs and so the likelihood of customers running the tests with their own namespaces is very low.

joshuaauerbachwatson added a commit that referenced this issue Dec 18, 2021
* WIP for --build-env

* Adjust to change in the Flags type

* Adjust call for extra argument in signature

* Add tests for --build-env feature

* Add tests, update deployer dependency (build-env)

* Skip the unreliable object list test (issue #168)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant