Use GitHub Actions to Run Tests on Windows and Ubuntu

[moleske]

After watching @gaussianrecurrence have some feed back about Windows being broken in the PR, I thought I'd share what I use to try to cover the bases. This uses GitHub Actions to run tests on Ubuntu 16.04 & 18.04 and Windows 2016 with VS2017 and Windows 2019 with VS2019. It is slow and prone to flakiness, but it is easy to rerun. Also not all builds work on VS2019 so those weren't added. It also doesn't run acceptance test nor build with clang at the moment. It does the rat and clang format checks though.

The software available on the GitHub workers is here. There is the possibility to use own workers which could be more powerful, but I didn't look at that.

I don't actually expect to merge this, as Geode seems all in on Concourse. I like me some Concourse, but I don't think it is easy to collaborate on Concourse for either Geode or Geode Native. It is especially hard to test changes yourself if you don't have a concourse lying around with all the secrets you want.

If you want to see this in action, you can look at the PRs I opened against my fork of the repo to see what success and flakiness appears. To see the run of this PR branch, check this out. If you want to see my original setup, which includes Mac (which is currently broken for reasons I don't understand) and a workflow of things I think should work but don't, you can check out my test actions branch and the results of that running.

Feel free to push changes to this branch to try things out. Currently it does not save error logs or run with -VV for details, but of which might be useful. GitHub allows the uploading of artifacts to runs, so it would be possible on failure to upload logs. I only tried uploading the entire build, but the entire build is way to big and often timed out on me. But a few log files seem like it should work. Might also be neat if there was a way to pass a built artifact around rather than what it current does of building it for each test. Regardless, this seemed like a good first step to make progress on this front rather than try to port the Geode Concourse setup over. Maybe Concourse can be used for some other publishing steps, or maybe this will just not be used and provide inspiration on other ways to organize. I think this would have helped @gaussianrecurrence find some of the Windows problems before @pdxcodemonkey found them on his private CI he has. I also believe this will help us spend less time on CI since I think a fair amount of work goes into maintaining the running Concourse for Geode, which would free to people to work on other problems.

[gaussianrecurrence]

Thanks really for sharing this. I actually spend lots of time in doing the verification manually. This is what I was looking for.
Really thanks again, I am crying of joy

[alb3rtobr]

I don't actually expect to merge this, as Geode seems all in on Concourse.

Well, but geode-native is not in Concourse, its using Travis.
Although you dont merge it to develop branch, I think it would be useful to have your test-actions branch contributed to geode-native anyway. And document how someone can use this branch to test a contribution before creating a PR.

In my case I have a fork of apache/geode-native, and I added your repo as remote. Then I created branch tracking your branch with your github actions changes (git co -b test-actions moleske/test-actions).
On the head of that branch, I plan to create a branch per Geode ticket I want to verify, so I can test my code before creating a PR :)

Im glad you shared this, Im currently working on a geode-native client that broke something in windows, and I dont have a windows machine to test it!

[gaussianrecurrence]

Btw @moleske,

@alb3rtobr tried to run the pipeline and the MacOS failed due to this: https://github.com/alb3rtobr/geode-native/runs/1225137425?check_suite_focus=true#step:8:6242

You mentioned that it was failing due to some reason, but in there it clearly seems that the problem is due to dropping a const-qualifier. Is it possible that MacOS compilation is actually broken? :S

[moleske]

You mentioned that it was failing due to some reason, but in there it clearly seems that the problem is due to dropping a const-qualifier. Is it possible that MacOS compilation is actually broken? :S

It actually compiles on my Mac, but I have older version of AppleClang, so I think that's it works for me and not on the GItHub Mac. Turns out I have the same AppleClang, but I don't have Xcode installed. So I don't know what's up between mine and GitHub's🤔

Nope I was right the first time, I do have older AppleClang. Thanks all for watching the real time confusion I went through

Last edit (probably), I use OpenSSL 1.1.1g and GitHub is using OpenSSL 1.0.2t. Not sure if that is an issue also 🤷 . Based on the fact that windows and ubuntu builds are using 1.1.1 I'm currently assuming that having mac use 1.1.1 instead of 1.0.2 will solve the problem. If you end up trying that before me let me know and then can update this branch with mac os builds (which would be sweet)

[moleske]

@gaussianrecurrence @alb3rtobr I got the mac one, it was indeed an openssl version problem. Check it out. If you want to see the wip playground I play in (with stuff that is broken), it is in https://github.com/moleske/geode-native/tree/test-actions/

[moleske]

Closing this as there's minimal interest, and I don't think it would be good to take this as lazy consensus (rather have something more active, or at least fist of five style.) If you want to follow or use this, follow my test-actions branch. Recently added --output-on-failure and --repeat until-pass:3 for tests to show detailed errors and to repeat tests that are flakier then one would like. Also add missing cli unit tests and don't fail fast the matrix builds. Still think this will be easier to community maintain over concourse but 🤷‍♀️