MySQL fails to run on Github Actions using Kind #1179
Labels
Comments
indeed, this smells to some filesystem permissions ... |
|
@tamalsaha seems you are using https://github.com/engineerd/setup-kind |
|
Without looking too closely yet this sounds like a PV using fsGroup, that won't work out of the box until the next kind release. If you use kind from HEAD it should work. |
|
https://github.com/tamalsaha/mysql-kind-demo/blob/master/.github/workflows/ci.yml even has the logic to setup the rancher driver, just commented out ...? |
|
I am going to try kind from HEAD and report back. But For this test, we don't need the rancher driver. |
|
I have tried Kind from HEAD and I am still getting the same error. https://github.com/tamalsaha/mysql-kind-demo/blob/master/.github/workflows/kind-master.yml |
|
There is something particularly weird about the virtual environment provided by GitHub actions. I created a similar machine DigitalOcean with Then creates a kind cluster and everything worked. I also added strace to the mysql image and you can see the error: |
ah, this is a directory in the mysql image? not a hostPath or a volume? I saw a PVC in your manifests when I took a quick look last night and guessed wrongly :-) |
|
|
seeing a few comments around that suggest a relation to apparmor |
|
Do you think https://unix.stackexchange.com/questions/229782/overlayfs-doesnt-work-with-unprivileged-user-namespace might be somehow related? The thing is I can |
possibly. I don't know how github actions setup docker in this environment or what security mechanisms / filesystems they have enabled... |
|
GitHub uses packer to build their VM. https://github.com/actions/virtual-environments |
|
can you test #1183 ? |
|
Checking from your fork. |
|
I used your fork tamalsaha/mysql-kind-demo@913a250 But I'm still getting the same error: |
5 tasks
|
:/ Can you obtain some more details about the github actions environment?
|
|
Here are some info: |
|
Also, I don't know where to check for: how is docker configured? anything interesting like user namespace remapping? |
|
@BenTheElder , I think this answer probably gives the clue docker-library/mysql#617 (comment) I am still trying to understand it. |
the daemon config.json. most of this is surfaced in
this bit from that's where docker stores runtime data (containers, images, volumes... ) we can check the permissions on that for sanity |
ah that makes sense! |
|
this explanation is good -- moby/moby#7512 (comment) an option is to disable apparmor on the host for mysql kind needs docs for this under known-issues I'm not sure that we can do much more. kind would of course not run priv containers if we didn't need to, similarly we need to support priv containers inside the kind nodes because kubernetes system components run this way (eg kube-proxy). containerd / kubelet rootless might be viable in the near future, but it's not quite ready, and i'm not sure if we can ship it right away when it's ready because most of our users's hosts won't be ready kubernetes/enhancements#1371 cgroups v1 -> v2 is going to be "fun" for hacks like KIND |
BenTheElder
added
kind/external
|
I don't think KIND can do much else here, we should document this in known-issues though. |
|
for now documented https://kind.sigs.k8s.io/docs/user/known-issues/#apparmor |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
We are trying to run an official MySQL pod inside a Kind cluster on GitHub actions. Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo
The problem is mysql pod fails to run with the error message:
You can see the full log here: https://github.com/tamalsaha/mysql-kind-demo/commit/dfef4e188accdb6334b75f4658bd0fc1dd35c732/checks?check_suite_id=364071606
The puzzling this is that if I just run
docker run mysql:8.0.14 mysqld --verbose --helpit works! You can see full log here:https://github.com/tamalsaha/mysql-kind-demo/commit/d452e672fa2219fb5ea5174bd99108d6d0b2728a/checks?check_suite_id=364034120
When I try these on my Ubuntu 18.04 Desktop (using pod in a Kind cluster and Direct Docker), both modes work.
You can see the diff between pod log from my Desktop (left) and Github action (right). They mostly look identical: https://www.diffchecker.com/D64wMscf
Can you help my understand why
mysqld --verbose --helpfails when run as a pod inside a Kind v1.16.3 cluster on GitHub actions?What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo
If you want to run locally, try this:
Anything else we need to know?:
One hack we found is that if we mount an emptyDir to /etc/mysql/conf.d folder,
mysqld --verbose --helpcommand starts working.Environment:
kind version): v0.6.1kubectl version): v1.16.3docker info):/etc/os-release):The text was updated successfully, but these errors were encountered: