Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve E2E test stability #77

Merged
merged 8 commits into from
Mar 13, 2024
Merged

Improve E2E test stability #77

merged 8 commits into from
Mar 13, 2024

Conversation

gasparnagy
Copy link
Contributor

@gasparnagy gasparnagy commented Mar 11, 2024

This PR contains a couple of changes to improve the E2E test stability and robustness. The "specs" tests run the E2E tests currently, but later they will be also used for "system tests" (see #62).

There are a couple of improvements included in this PR, but the most important is that it provides a solution for using cached packages. This issue we have discovered with @livioc on #54: If you modify the generation part (e.g. MsTestV2GeneratorProvider) and rerun the related "specs" test, the test will pick up the previously generated nuget package from the package cache and therefore the test will not reflect your changes (you still see the old behavior).

The reason for this is that these E2E test use the generated NuGet packages for generating the sample projects. Between two compilations these packages have the same version number, therefore if a version has been used, nuget automatically caches that in the local nuget cache (C:\Users\<your-user>\.nuget\packages), so the subsequent execution will re-use the cached version. The solution was to manually clean the Reqnroll related packages from the local nuget cache.

There has been tries to avoid this problem: in SpecFlow we tried to generate commit specific nuget versions (this caused other problems and did not solve the changes within commits) and also there was a code that tried to delete the packages from the local nuget cache (most of the cases did not work because of file locking issues).

This PR uses a different approach: it reconfigures nuget for the test projects to use an alternative local nuget cache folder somewhere in $TEMP, so that the execution always starts with an empty nuget cache. (There is a related issue in nuget where this idea was mentioned.) The nuget cache has several megabytes, so instead of making a local nuget cache for every sample project, we introduce a "Test Run" folder, that contains all sample project folders and the local nuget cache. Each time a new test run folder is created in $TEMP, but we have a cleanup mechanism that removes old test run folders.

I have tested that by replaying the changes of #54 and it works fine, all changes of the code are immediately reflected in tests.

Other improvements in this PR:

  • For those tests that only assert for the total number of tests we also verify that those were passing (in MsTest DataRow support #54 many problems were hidden because of that).
  • The TestProjectGenerator related projects are now target netstandard2.0 and net6.0 only.
  • The obsolete -a argument of nuget has been replaced
  • Improve "specs" to assert for a successful compilation except otherwise stated (in MsTest DataRow support #54 we also had this) and provide better error message
  • Added some additional helper methods to TestProjectGenerator that will be needed for "system tests"
  • Get rid of the warnings in the generated Log.cs files

Types of changes

  • Refactoring (so no functional change)

@gasparnagy gasparnagy marked this pull request as ready for review March 11, 2024 15:50
@gasparnagy gasparnagy merged commit 6b76b61 into main Mar 13, 2024
7 checks passed
@gasparnagy gasparnagy deleted the improve-e2e-test-stability branch March 13, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant