Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider CI/testing #95

Closed
PaulBone opened this issue Dec 29, 2018 · 6 comments
Closed

Consider CI/testing #95

PaulBone opened this issue Dec 29, 2018 · 6 comments

Comments

@PaulBone
Copy link
Member

@PaulBone PaulBone commented Dec 29, 2018

I'm not an expert on CI/CD. I'm fairly sure CD is not what we want. And
I'm not sure of the precise meaning of CI. So bear with me if I get
something wrong.

Compiler

Plasma has a compiler written in Mercury that has no system specific parts
beyond what Mercury has in its own runtime. it has a tiny bit of foreign
code in C but it's pretty simple.

We should test with:

  • 32 and 64 bit systems,
  • stable and development versions of Mercury.

Note that development versions of Mercury are currently broken on 32 bit
systems.

Runtime

The rumtime is currently fairly system agnostic, at least until C++ was
added. It uses a mix of C and C++, and should be tested with at least two
compilers. There is also a build option to enable extra asserts and other
development options.

We should test with:

  • 32 and 64 bit systems
  • -DPZ_DEV

If possible:

  • le and be if possible
  • More than one OS (Linux and OS X have been tried so far, if it was known
    to work on something else then I'd add this to "should").
  • Systems that assert for unaligned accesses if possible.
  • valgrind / memcheck / ubcheck / etc.

I'm also in the process of adding a "zealous" mode to the GC, which performs
a lot more checking but runs slower. And this is why I'm interested in CI.
I don't think we want this to run when people type "make test", not by
default. But it does make sense to run it for each commit / pull request.

So there's a few things that would be useful to begin testing now. Or at
the very least to think about testing. So here's where I throw it over to
you (the reader) since you probably know more about this than me.

New GC mode

So the first question, is for the new GC mode should this be enabled as a
compiler option or an environment variable? I'm imagining compiling it when
PZ_DEV is defined and enable it when an environment variable is set. I
imagine that typical CI stuff can handle this but please tell me if it
isn't.

Hosted CI?

When I've worked on other projects, things like "Travis" would run when I
opened a pull request. Has anyone worked with or set this up before? Can
it support what we need?

Alternatively I've setup Jenkins before, although I never finished the full
configuration, for a non-hosted thing I think Jenkins seems good. Does
anyone have any suggestions between hosted/non-hosted and which service to
use?

TAP

It was suggested (Bug #28, Bug #6) that Test Anything Protocol (TAP) can be used to gather results from a test suite and integrate with a CI tool. Should we use this? is it the best option? What other work needs to be done to the test suite / Makefile to make them work with a CI tool?

Thanks.

Also here on the mailing list: http://www.plasmalang.org/list-archives/dev/2018-December/000027.html

@nox

This comment has been minimized.

Copy link

@nox nox commented Dec 29, 2018

For personal projects I use https://bors.tech which lets me use Travis in a controlled way: I write bors r+ in the PR thread and Homu tells Travis to run the tests on a merge of the current master branch with the code in the PR, if the tests pass, the merge commit becomes the new master tip, thus ensuring it stays green (barring any intermittent failures, which you know are a problem often times given we are colleagues 😂).

@PaulBone

This comment has been minimized.

Copy link
Member Author

@PaulBone PaulBone commented Dec 29, 2018

(barring any intermittent failures, which you know are a problem often times given we are colleagues joy).

Yes. it took me at least 6 months to develop a knack for knowing which intermittents might be my fault and which probably (hopefully) aren't.

@PaulBone

This comment has been minimized.

Copy link
Member Author

@PaulBone PaulBone commented Dec 29, 2018

That sounds good. I'll check it out.

@PaulBone

This comment has been minimized.

Copy link
Member Author

@PaulBone PaulBone commented Jan 20, 2020

@brendanzab just showed me his configuration for github actions. This looks really neat and probably what we want for now. I'm still intrigued by bors, but it's probably more than we need for now.

https://help.github.com/en/actions/automating-your-workflow-with-github-actions

@brendanzab

This comment has been minimized.

Copy link

@brendanzab brendanzab commented Jan 20, 2020

The advantage of bors is that you don't have to sit around waiting for the green CI thing to show - you just say bors r+! and forget it. Still might be worth exploring down the line!

@PaulBone

This comment has been minimized.

Copy link
Member Author

@PaulBone PaulBone commented Feb 21, 2020

Regarding bors, I've read up on it and I'm all-on-board. But CI is a prerequisite so we'll do that first.

For CI there's things we can do now, and some things we can do later.

Now

  • tags / versioning for docker images.
  • tags / versioning for actions
  • clang / gcc
  • rotd / stable Mercury Package stable Mercury first.
  • Test script can list tests rather than dots in CI.
  • Various C compiler versions (oldest supported and newest stable will be good). (later)
  • Test with -Wall -Werror on newest C++ compiler.
  • Status / badges / notifications / PR comments etc.
  • -DPZ_DEV and non-dev build options.
  • BUILD_TYPE in run-tests.sh
  • aggressive GC mode.
  • Linting build
  • Test building docs

Later

Some of these can be lifted out of "make test" so that we speed-up "make test" but still catch problems in CI.

  • valgrind / memsan?
  • ubsan?
  • bors
  • Test building website (low priority, need to test for broken links also).
  • deployment eg: builds, update website.
  • Artifacts (lots, diffs for broken output, core dumps).

Much Later

Plasma has not been tested on these combinations at all, so CI doesn't yet make sense. Or cloud hosting isn't available.

  • x86, x86_64, arm, arm64 (needs different cloud CI providers? / real hardware)
  • More linux distros / FreeBSD / *BSD? (low priority / unported)
  • macos (needs some work to get Mercury in a docker or something)
  • Windows (Plasma is untested/unported to Windows for now)
  • Performance checking / graphs / regression warnings (needs real hardware).

Test suite

Test suite improvments including TAP is #6 and #28

@PaulBone PaulBone self-assigned this Feb 21, 2020
@PaulBone PaulBone closed this in 85c2265 Feb 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.