Integrate dtcw tests into CI build

[rdmueller]

the new tests for the wrapper really help a lot!

Can we setup a test which downloads the current wrapper from https://doctoolchain.org/dtcw to test the correct deployment of it?

Would this make sense?

[mh182]

What do you mean with "test the correct deployment of it"?

The deployment of dtcw is just the execution of curl. If you mean testing the deployment of docToolchain with dtcw this is already done with the tests provided in #1077.

In the PR I created two end-to-end (e2e) test:

• local installation of docToolchain
• installation with sdk

What I would like to have is Github workflows running the tests on different environments (Linux, macOS) whenever someone changes dtcw. But since the tests need a docker container, we would need a container image for macOS.

I haven't worked with Github workflows that much.

Maybe I'm able to adapt the tests which use mocks (the non e2e tests) so that they do not need the docker container. Then we just could add a workflow tests for Ubunt and macOS.

Is that what you were asking for?

[mh182]

I started to work on a GitHub Action workflow running the test suite whenever dtcw changes.

I followed the approach to run the tests within a docker container. This works fine as long an Ubuntu runner is used. But I would like to test the scripts on macOS too, as requested in #543.

Running the tests in the virtual machine provided by the GHA fails. The tests assume a clean environment without any tooling (like curl, wget, docker etc.). The tests mock-commands which interfere with the commands of the virtual machine.

Since we have to run the test suite on a VM for macOS anyway, I have to change the tests, so there is no collision between the test mocks and the installed commands.

[rdmueller]

What do you mean with "test the correct deployment of it"?

as far as I could see, the tests use the dtcw from the repository, not the one downloaded from the website (the deployed one).

[rdmueller]

Is that what you were asking for?

I am quite happy with what you came up with. I think it will be quite hard to come up with tests for macOS and powershell.
I played around a bit with powershell on linux to test, but this isn't the same as powershell on windows, so I stopped it.

My initial thought from this issue was that we should set up a test which starts with the curl command to download the wrapper from the website. It was more or less a reminder for me to checkout the test framework you came up with :-)

[rdmueller]

I started to work on a GitHub Action workflow running the test suite whenever dtcw changes.

that is a great idea!

[mh182]

Is that what you were asking for?

I am quite happy with what you came up with. I think it will be quite hard to come up with tests for macOS and powershell. I played around a bit with powershell on linux to test, but this isn't the same as powershell on windows, so I stopped it.

Well, dtcw is going to be used on macOS. So we have to test in there.

I'm almost certain that we can run the same tests on macOS and Linux (or other POSIX compliant OS). I have to extend https://github.com/grayhemp/bats-mock, the mocking framework so that I can mock commands as "command not found". I'm currently working on that.

My initial thought from this issue was that we should set up a test which starts with the curl command to download the wrapper from the website. It was more or less a reminder for me to checkout the test framework you came up with :-)

So you want to test if the URL shown in the documentation points to dtcw?

Because the test suite already performs end-to-end tests for installing docToolchain and Java with dtcw.

The CI test pipeline for dtcw would consist of 3 jobs:

1. shellcheck the bash scripts
2. Run the mocked tests (approx 20 seconds)
3. Run e2e tests: for local install, for install with sdk

The jobs are run sequentially, meaning they would stop in case an error occurs.

Regarding the dtcw PowerShell implementation. I'm really sorry, but I have not the means, nor the capacity to do the changes there. I hope someone is going to stand up and port the current dtcw shell code to PowerShell version.

[mh182]

What do you mean with "test the correct deployment of it"?

as far as I could see, the tests use the dtcw from the repository, not the one downloaded from the website (the deployed one).

IMHO, it doesn't make much sense to test the published dtcw script. I assume we only release dtcw after it went through the CI build with all the tests OK.

I currently struggle with GitHub actions. I would like to change the default-build so that It doesn't run when dtcw is changed. There is no point in running the Java tests when the shell script changed. I added a ignore-path filter for dtcw but it still keeps beeing built.

[mh182]

Just to give this ticket a boundary. The ticker is going to be resolved when

• we have a new GHA workflow which triggers the dtcw test suite for Linux and macOS
• the workflow is only triggered on changes in dtcw or the test suite for it, but not changes on the other parts of docToolchain

The end-to-end tests with dtcw should not be executed with this workflow, but just before the release.