Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a couple of changes to improve the E2E test stability and robustness. The "specs" tests run the E2E tests currently, but later they will be also used for "system tests" (see #62).
There are a couple of improvements included in this PR, but the most important is that it provides a solution for using cached packages. This issue we have discovered with @livioc on #54: If you modify the generation part (e.g.
MsTestV2GeneratorProvider
) and rerun the related "specs" test, the test will pick up the previously generated nuget package from the package cache and therefore the test will not reflect your changes (you still see the old behavior).The reason for this is that these E2E test use the generated NuGet packages for generating the sample projects. Between two compilations these packages have the same version number, therefore if a version has been used, nuget automatically caches that in the local nuget cache (
C:\Users\<your-user>\.nuget\packages
), so the subsequent execution will re-use the cached version. The solution was to manually clean the Reqnroll related packages from the local nuget cache.There has been tries to avoid this problem: in SpecFlow we tried to generate commit specific nuget versions (this caused other problems and did not solve the changes within commits) and also there was a code that tried to delete the packages from the local nuget cache (most of the cases did not work because of file locking issues).
This PR uses a different approach: it reconfigures nuget for the test projects to use an alternative local nuget cache folder somewhere in $TEMP, so that the execution always starts with an empty nuget cache. (There is a related issue in nuget where this idea was mentioned.) The nuget cache has several megabytes, so instead of making a local nuget cache for every sample project, we introduce a "Test Run" folder, that contains all sample project folders and the local nuget cache. Each time a new test run folder is created in $TEMP, but we have a cleanup mechanism that removes old test run folders.
I have tested that by replaying the changes of #54 and it works fine, all changes of the code are immediately reflected in tests.
Other improvements in this PR:
-a
argument of nuget has been replacedTypes of changes