split unit and integration tests for blueprint

[kellyselden]

The blueprint-test is huge and has both unit and integration tests. This splits them up. Should probably be applied across the board.

[stefanpenner]

these look kinda unit testy, can you share your thoughts?

[kellyselden]

My thoughts are unit tests test parts of a blueprint model, with dependencies like the filesystem mocked. Integration tests involve an actual (albeit fixture) blueprint and test all the pieces together including filesystem artifacts.

My reasoning in a nutshell is: When I want to write a new test, I know whether I want to test the unit or test the whole blueprint, but when go to find where to put the test, I have to read through the entire test file of mixed unit/integration tests to find where to put my test.

[stefanpenner]

My thoughts are unit tests test parts of a blueprint model, with dependencies like the filesystem mocked.

Some thoughts:

• un-mocked FS is still fine for unit testing as the FS is basically an invariant of our system. Mocking it provides unknown value, but more risk. E.g. the FS mocks are less trust worthy then the real FS. Rather, if an algorithm doesn't care about the FS it should likely then not interact with it directly. If it does care about the FS it should just interact with it directly. So if we have parts of blueprints, that shouldn't directly interact with the file-system, but they do. We will need to tease that apart, so the FS interactions can be unit tested.

• For our use-cases, un-mocked interactions with non-invariants (like project, or ui or specific tasks) would not be unit testing. If they are not full system, integration is fine if they are acceptance is appropriate.

---

After reading through blueprint's "unit" tests earlier, I agree some are integration tests. But many are merely interacting with mocks and fixtures data, which suggests they are unit tests. For those interacting with very much discrete high level un-mocked units, they should be considered integration tests.

[homu]

☔ The latest upstream changes (presumably #6475) made this pull request unmergeable. Please resolve the merge conflicts.

[stefanpenner]

@kellyselden ping?

[kellyselden]

the FS is basically an invariant of our system.

I think the FS is a huge variant. Wether it's Windows vs Unix, sudo vs non sudo, local vs CI, I think there are a number of reasons why a unit test would pass because you are calling all the right fs functions, and the integration test fail because the FS interaction has a failure. Having this difference can save time when tracking down bugs.

Mocking it provides unknown value, but more risk

Only if you rely solely on unit tests.

But many are merely interacting with mocks and fixtures data, which suggests they are unit tests

I agree mocks imply unit test, but fixtures imply integration. For example, if my unit I'm testing does a string replace, but the line endings in the fixture files are messing with my test, it's no longer a unit test, but an integration test IMO.

[stefanpenner]

the FS is basically an invariant of our system.
I think the FS is a huge variant. Wether it's Windows vs Unix, sudo vs non sudo, local vs CI, I think there are a number of reasons why a unit test would pass because you are calling all the right fs functions, and the integration test fail because the FS interaction has a failure. Having this difference can save time when tracking down bugs.

Unit tests, that interact first-class with FS, can hit fixtures. If they are indirectly depending on the FS, it should be mocked.

---

Im fine with merging this,

[kellyselden]

@homu r+

[homu]

📌 Commit d801b65 has been approved by kellyselden

[homu]

⚡ Test exempted - status