Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider CI/testing #95

PaulBone opened this issue Dec 29, 2018 · 6 comments

Consider CI/testing #95

PaulBone opened this issue Dec 29, 2018 · 6 comments
meta: triaged project skill: scripting status: resolved type: maintenance


Copy link

@PaulBone PaulBone commented Dec 29, 2018

I'm not an expert on CI/CD. I'm fairly sure CD is not what we want. And
I'm not sure of the precise meaning of CI. So bear with me if I get
something wrong.


Plasma has a compiler written in Mercury that has no system specific parts
beyond what Mercury has in its own runtime. it has a tiny bit of foreign
code in C but it's pretty simple.

We should test with:

  • 32 and 64 bit systems,
  • stable and development versions of Mercury.

Note that development versions of Mercury are currently broken on 32 bit


The rumtime is currently fairly system agnostic, at least until C++ was
added. It uses a mix of C and C++, and should be tested with at least two
compilers. There is also a build option to enable extra asserts and other
development options.

We should test with:

  • 32 and 64 bit systems
  • -DPZ_DEV

If possible:

  • le and be if possible
  • More than one OS (Linux and OS X have been tried so far, if it was known
    to work on something else then I'd add this to "should").
  • Systems that assert for unaligned accesses if possible.
  • valgrind / memcheck / ubcheck / etc.

I'm also in the process of adding a "zealous" mode to the GC, which performs
a lot more checking but runs slower. And this is why I'm interested in CI.
I don't think we want this to run when people type "make test", not by
default. But it does make sense to run it for each commit / pull request.

So there's a few things that would be useful to begin testing now. Or at
the very least to think about testing. So here's where I throw it over to
you (the reader) since you probably know more about this than me.

New GC mode

So the first question, is for the new GC mode should this be enabled as a
compiler option or an environment variable? I'm imagining compiling it when
PZ_DEV is defined and enable it when an environment variable is set. I
imagine that typical CI stuff can handle this but please tell me if it

Hosted CI?

When I've worked on other projects, things like "Travis" would run when I
opened a pull request. Has anyone worked with or set this up before? Can
it support what we need?

Alternatively I've setup Jenkins before, although I never finished the full
configuration, for a non-hosted thing I think Jenkins seems good. Does
anyone have any suggestions between hosted/non-hosted and which service to


It was suggested (Bug #28, Bug #6) that Test Anything Protocol (TAP) can be used to gather results from a test suite and integrate with a CI tool. Should we use this? is it the best option? What other work needs to be done to the test suite / Makefile to make them work with a CI tool?


Also here on the mailing list:

@PaulBone PaulBone added build system skill: scripting meta: meta project labels Dec 29, 2018
@PaulBone PaulBone added this to Triage in Plasma Programming Language via automation Dec 29, 2018
Copy link

@nox nox commented Dec 29, 2018

For personal projects I use which lets me use Travis in a controlled way: I write bors r+ in the PR thread and Homu tells Travis to run the tests on a merge of the current master branch with the code in the PR, if the tests pass, the merge commit becomes the new master tip, thus ensuring it stays green (barring any intermittent failures, which you know are a problem often times given we are colleagues 😂).

Copy link
Member Author

@PaulBone PaulBone commented Dec 29, 2018

(barring any intermittent failures, which you know are a problem often times given we are colleagues joy).

Yes. it took me at least 6 months to develop a knack for knowing which intermittents might be my fault and which probably (hopefully) aren't.

Copy link
Member Author

@PaulBone PaulBone commented Dec 29, 2018

That sounds good. I'll check it out.

@PaulBone PaulBone added meta: triaged status: accepted and removed meta: meta labels Mar 23, 2019
@PaulBone PaulBone added this to the Language groundwork milestone Mar 23, 2019
@PaulBone PaulBone removed this from the Language groundwork milestone Mar 23, 2019
@PaulBone PaulBone added this to the Ease of development milestone Mar 23, 2019
@PaulBone PaulBone added the type: maintenance label Mar 24, 2019
Copy link
Member Author

@PaulBone PaulBone commented Jan 20, 2020

@brendanzab just showed me his configuration for github actions. This looks really neat and probably what we want for now. I'm still intrigued by bors, but it's probably more than we need for now.

Copy link

@brendanzab brendanzab commented Jan 20, 2020

The advantage of bors is that you don't have to sit around waiting for the green CI thing to show - you just say bors r+! and forget it. Still might be worth exploring down the line!

Copy link
Member Author

@PaulBone PaulBone commented Feb 21, 2020

Regarding bors, I've read up on it and I'm all-on-board. But CI is a prerequisite so we'll do that first.

For CI there's things we can do now, and some things we can do later.


  • tags / versioning for docker images.
  • tags / versioning for actions
  • clang / gcc
  • rotd / stable Mercury Package stable Mercury first.
  • Test script can list tests rather than dots in CI.
  • Various C compiler versions (oldest supported and newest stable will be good). (later)
  • Test with -Wall -Werror on newest C++ compiler.
  • Status / badges / notifications / PR comments etc.
  • -DPZ_DEV and non-dev build options.
  • aggressive GC mode.
  • Linting build
  • Test building docs


Some of these can be lifted out of "make test" so that we speed-up "make test" but still catch problems in CI.

  • valgrind / memsan?
  • ubsan?
  • bors
  • Test building website (low priority, need to test for broken links also).
  • deployment eg: builds, update website.
  • Artifacts (lots, diffs for broken output, core dumps).

Much Later

Plasma has not been tested on these combinations at all, so CI doesn't yet make sense. Or cloud hosting isn't available.

  • x86, x86_64, arm, arm64 (needs different cloud CI providers? / real hardware)
  • More linux distros / FreeBSD / *BSD? (low priority / unported)
  • macos (needs some work to get Mercury in a docker or something)
  • Windows (Plasma is untested/unported to Windows for now)
  • Performance checking / graphs / regression warnings (needs real hardware).

Test suite

Test suite improvments including TAP is #6 and #28

@PaulBone PaulBone self-assigned this Feb 21, 2020
@PaulBone PaulBone added status: resolved and removed status: accepted labels Feb 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
meta: triaged project skill: scripting status: resolved type: maintenance
No open projects

No branches or pull requests

3 participants