ci: Integration Test Framework

[Jason3S]

I have a few goals in mind because I want to release v5, but I need assurance that its output matches v4.

1. Ensure cspell v5 results are compatible with v4
2. Ensure that on going development in v5 or v4 do no break existing usages.
3. Check how changes to dictionaries change the spelling results.

Logic of current set of integration tests:

1. Run cspell in a repo for a given commit
2. Compare results against a snapshot.

I want to be able to:

1. Add new integration tests (especially for languages that the spell checker does not do a great job with GO, C++, Python)
2. Be able to update snapshots easily.

[coveralls]

Coverage increased (+2.2%) to 100.0% when pulling 10e6856 on cspell5-integration-tests into 4c1df9a on master.

[codecov]

Codecov Report

Merging #431 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #431   +/-   ##
=======================================
  Coverage   98.42%   98.42%           
=======================================
  Files          85       85           
  Lines        3609     3609           
  Branches      604      604           
=======================================
  Hits         3552     3552           
  Misses         54       54           
  Partials        3        3           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c1df9a...10e6856. Read the comment docs.

[nschonni]

You might also want to take a look at what was setup over here https://github.com/DavidAnson/markdownlint
The external projects aren't added as submodules, but cloned as part of the CI testing, or by initializing/downloading the repos locally in another script.
It avoids needing to keep submodules up to date. There is a Dependabot task that can do that, but might cause a alot of noise with multiple repos

[Jason3S]

You might also want to take a look at what was setup over here https://github.com/DavidAnson/markdownlint
The external projects aren't added as submodules, but cloned as part of the CI testing, or by initializing/downloading the repos locally in another script.
It avoids needing to keep submodules up to date. There is a Dependabot task that can do that, but might cause a alot of noise with multiple repos

You make an excellent point. Using submodules is a pain. I had considered downloading repos and will most likely go that way.
I decided to use submodules for the moment to focus in the testing. I will improve the process a bit later.

I have a few goals in mind because I want to release v5, but I need assurance that its output matches v4.

1. Ensure cspell v5 results are compatible with v4
2. Ensure that on going development in v5 or v4 do no break existing usages.
3. Check how changes to dictionaries change the spelling results.

Logic of current set of integration tests:

1. Run cspell in a repo for a given commit
2. Compare results against a snapshot.

I want to be able to:

1. Add new integration tests (especially for languages that the spell checker does not do a great job with GO, C++, Python)
2. Be able to update snapshots easily.