Converting to Github Actions or Azure Pipelines.

[vamega]

I notice that the CI seems to having issues on Travis at the moment.
I think part of it might stem from the need for the tests to hit a server that is run externally to the tests.

Would you be open to running the tests in Github Actions or Azure Pipelines, since I think those services would allow us to start an Exasol docker container alongside the tests against which the tests could be run.

If that's something you'd be willing to consider, I'll look into figuring out how to get that working.

[jank]

This would be a major step forward for automating the build. Builds are currently ran against an EXASOL cluster hosted by Exasol. We have a test user there but need to be careful in how we expose the credentials (only to core committers). Also, this limits parallelism in the build and we cannot manage the EXASOL cluster version as this is done by EXASOL.
So running this on GitHub Actions that allows spinning up an own EXASOL docker image sounds quite attractive and would allow a lot more contributors to run builds.

[vamega]

Yes, I'm hoping to find some time to work on this soon.
The work I have done is on this branch: https://github.com/vamega/sqlalchemy_exasol/tree/github-actions

I ran into an issue while trying to debug why the Exasol docker container wasn't starting up on Github Actions. I'm suspecting it might be something related to the docker containers requirement of O_DIRECT which may not be supported on the runners that Github provides.

I made a community post about it here: https://github.community/t5/GitHub-Actions/Debugging-service-container-startup-failure/m-p/33908, but never got a response.

I haven't found the time recently to dig into this, and I think it's going to be at least two weeks before I'll have the time to dig into it. I'm not particularly familiar with Github Actions which makes this something I'm slower at doing.

However it's something I want to see done, so I'm glad you're open to the switch. Makes me more motivated for finding the time to finish it.

[jank]

Looks promising. Also make sure to reach out to the folks at EXASOL, maybe they have an idea on why the EXASOL container does not spin up.

[jank]

@vamega have you made any progress? I was trying to read up on how services in GitHub actions are configured. I stumbled on this post that refers to setting the workdir options: --workdir /home/dynamodblocal
https://github.community/t5/GitHub-Actions/Github-Actions-services-not-reachable/td-p/30050
However, I am not sure if EXASOL is sensitive to a home directory (have not found anything in their documentation).

[vamega]

@BY-jk - Sorry my own work has been very busy and I haven't had the chance to investigate this more thoroughly.

I think an option like workdir might help, since I suspect the issue is the need for O_DIRECT support for the filesystem used to store the Exasol data.

I was hoping to replicate that by using a mergerfs partition at home which I know does not support O_DIRECT when the parameter direct_io is set to false.

I have seen your post on the Exasol docker-db repository, hopefully they will be able to provide more insight into this.

My question on the Github community site hasn't received any traction. What I'd really like is the ability to debug the docker container that is run in the pipeline, but I don't know of a way to do that at the moment.

[jank]

@vamega if you are still on this. EXASOL commented on this in their docker image project over here: https://github.com/exasol/docker-db/issues/44#issuecomment-616532017

[vamega]

I had been blocked on the Github Actions issue. I'll look at it this weekend. Thanks for pointing that out to me.

…

On Mon, Apr 20, 2020 at 9:01 AM Jan ***@***.***> wrote: @vamega <https://github.com/vamega> if you are still on this. EXASOL commented on this in their docker image project over here: exasol/docker-db#44 (comment) <https://github.com/exasol/docker-db/issues/44#issuecomment-616532017> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWRMFDLUIYH5SFRBQRVELRNRBQ7ANCNFSM4I4QNKHA> .

[jank]

More context was provided, especially a hint at an exasol docker image to take as a starting point.
https://github.com/exasol/docker-db/issues/44#issuecomment-617083378

[vamega]

Thanks for pointing that out.
Will take a stab at this over the weekend.

[vamega]

@BY-jk , only managed to do some reading on this over the weekend.
The repository linked may not be appropriate for what Github Actions, but I think I now know how I can configure the base docker container to test my hypothesis.

Will keep this issue updated with any progress I make.

[tkilias]

@BY-jk @vamega I am the maintainer of the mentioned repository. Github Actions doesn't provide O_DIRECT which can be disabled in the configuration. We can try this and then use it for integration testing.

[tkilias]

@BY-jk I opened a draft pull request for Github Actions with Exasol Docker-DB. It is currently, unfinished and I might need some input from you to finish it.
It uses integration-test-docker-environment, because it greatly simplifies the start of the configured docker-db's and also waits for the database to be actually available. It also supports the configuration for different Exasol versions which is usually a bit cumbersome, because the config file format changed with the Exasol versions. We are using it also in our own test system for the Script Language Container, as such it will be maintained also in the future.

[jank]

close with #91