Test Framework

[haxmeadroom]

Add some basic pre-commit testing that runs winchecksec on a few exe's and checks the results.

[woodruffw]

By pre-commit do you mean a commit hook, or in the CI? I'd prefer it if we add this to the CI so that external contributors don't need to share or configure our commit hook setup, but not sure if that's what you meant anyways 🙂

[haxmeadroom]

Maybe pre-commit isn't the right word. I like to make test and have say pytest run some regression tests before committing.

[woodruffw]

Gotcha. That SGTM, as long as we can also enable it in the CI!

[haxmeadroom]

Is there a collection anywhere of PE's that have various flags set we could throw into a test/ directory and run against? We could compile a helloworld.c with different flags I suppose.

[haxmeadroom]

I've never done CI so I have no idea how it works. Are there certain things to do / not do that make it easier to integrate with CI?

[woodruffw]

Is there a collection anywhere of PE's that have various flags set we could throw into a test/ directory and run against? We could compile a helloworld.c with different flags I suppose.

We could probably fetch some of the SysInternals EXEs and use them for tests, although I'm not sure about licensing. A helloworld.c that we build a few different times with different flags sounds good to me, though.

I've never done CI so I have no idea how it works. Are there certain things to do / not do that make it easier to integrate with CI?

Nothing in particular; if the tests can be run with make test and exit with a non-zero code on failure, that would be perfect 🙂

[woodruffw]

I created pegoat to supply PEs for our testing needs. We can either submodule it here or do some kind of artifact management with GitHub Actions to get its build products for testing.

[ekilmer]

either submodule it here or do some kind of artifact management with GitHub Actions to get its build products for testing.

Submodule would enable people who clone the repo to easily build, contribute, and run tests locally.

GitHub Actions would (hopefully) enable more elegant build artifact caching to skip building the examples every time.

Doing both would mean updating commit SHAs in two places, which is a bit annoying.

Hmmmm. If we had an initial stage, similar to linting, that had build caching turned on, we'd be able to build the examples in the submodule quickly, and it would only be a single place to update the git SHA. Not sure how to deal with CMakeLists.txt files, but if we had a fast way to do something like make test-binaries which is run before each platform's build, then that would be nice. Then, the built binaries could be distributed to each platform's test job.