fix: Make tests more robust & fix path-handling bugs on macOS #183
+653
−56
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So it looks like basically there was a bug where the indexer was
generating 0 documents on macOS (related to the recent bug fix in
https://github.com/sourcegraph/scip-python/pull/177/files where we
started normalizing the file paths passed to setTrackedFiles). 😬
However, the tests were passing on macOS not because they were
correct, but because the zero documents case bypassed the
updating/diffing process. 😢
This patch changes the test diffing logic to be more robust, in handling
various scenarios such as 0 documents. It also introduces a couple
of flags, one for filtering tests, and one for running tests with fail-fast
behavior.
On fixing the tests, I discovered some bugs due to improper handling
of case-sensitivity, so I fixed those (which is why the snapshots are unchanged
and the tests are passing on macOS). For some of these, I'm not super
sure about Pyright behavior, so I added a little assertions library
inspired by Antithesis so that we can test the coverage of different
cases by our test suite on macOS.
See also: https://antithesis.com/docs/best_practices/sometimes_assertions/