Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL fails to run on Github Actions using Kind #1179

Closed
tamalsaha opened this issue Dec 18, 2019 · 26 comments
Closed

MySQL fails to run on Github Actions using Kind #1179

tamalsaha opened this issue Dec 18, 2019 · 26 comments

Comments

@tamalsaha
Copy link
Contributor

@tamalsaha tamalsaha commented Dec 18, 2019

What happened:
We are trying to run an official MySQL pod inside a Kind cluster on GitHub actions. Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo

The problem is mysql pod fails to run with the error message:

+ mysqld --verbose --help
mysqld: Can't read dir of '/etc/mysql/conf.d/' (OS errno 13 - Permission denied)
mysqld: [ERROR] Fatal error in defaults handling. Program aborted!

You can see the full log here: https://github.com/tamalsaha/mysql-kind-demo/commit/dfef4e188accdb6334b75f4658bd0fc1dd35c732/checks?check_suite_id=364071606

The puzzling this is that if I just run docker run mysql:8.0.14 mysqld --verbose --help it works! You can see full log here:
https://github.com/tamalsaha/mysql-kind-demo/commit/d452e672fa2219fb5ea5174bd99108d6d0b2728a/checks?check_suite_id=364034120

When I try these on my Ubuntu 18.04 Desktop (using pod in a Kind cluster and Direct Docker), both modes work.

You can see the diff between pod log from my Desktop (left) and Github action (right). They mostly look identical: https://www.diffchecker.com/D64wMscf

Can you help my understand why mysqld --verbose --help fails when run as a pod inside a Kind v1.16.3 cluster on GitHub actions?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo

If you want to run locally, try this:

kind create cluster
kubectl apply -f https://github.com/tamalsaha/mysql-kind-demo/raw/master/hack/kubernetes/mysql.yaml

Anything else we need to know?:

One hack we found is that if we mount an emptyDir to /etc/mysql/conf.d folder, mysqld --verbose --help command starts working.

Environment:

  • kind version: (use kind version): v0.6.1
  • Kubernetes version: (use kubectl version): v1.16.3
  • Docker version: (use docker info):
Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.3.1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 3.0.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-1027-azure
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.783GiB
 Name: fv-az86
 ID: KRSI:74QW:YCD2:XEK7:4WO2:DOAO:FRYG:MP4B:J2YH:N63L:5SGR:JV5A
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
  • OS (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
@tamalsaha tamalsaha added the kind/bug label Dec 18, 2019
@aojea

This comment has been minimized.

Copy link
Contributor

@aojea aojea commented Dec 18, 2019

When I try these on my Ubuntu 18.04 (using pod in a Kind cluster and Direct Docker), both modes work.

indeed, this smells to some filesystem permissions ... 🤔 I wonder if there are some restrictions on the docker folders https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners

@aojea

This comment has been minimized.

Copy link
Contributor

@aojea aojea commented Dec 18, 2019

@tamalsaha seems you are using https://github.com/engineerd/setup-kind
Can you open an issue there and reference this one?
maybe the author of that project is more familiar with Github Actions and can shed some light on this issue

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

Without looking too closely yet this sounds like a PV using fsGroup, that won't work out of the box until the next kind release. If you use kind from HEAD it should work.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

https://github.com/tamalsaha/mysql-kind-demo/blob/master/.github/workflows/ci.yml even has the logic to setup the rancher driver, just commented out ...?

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

I am going to try kind from HEAD and report back. But /etc/mysql/conf.d folder is not mounted to any disk. This is just Docker overlay fs.

For this test, we don't need the rancher driver.

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

There is something particularly weird about the virtual environment provided by GitHub actions. I created a similar machine DigitalOcean with uname -a

root@kind-mysql-1904:~# uname -a
Linux kind-mysql-1904 5.0.0-31-generic #33-Ubuntu SMP Mon Sep 30 18:51:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Then creates a kind cluster and everything worked.

I also added strace to the mysql image and you can see the error:

2019-12-18T20:40:53.9611864Z + strace mysqld --verbose --help
2019-12-18T20:40:53.9612291Z execve("/usr/sbin/mysqld", ["mysqld", "--verbose", "--help"], [/* 18 vars */]) = 0
2019-12-18T20:40:53.9612494Z brk(NULL)                               = 0x55c482303000
2019-12-18T20:40:53.9612925Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9613123Z readlink("/proc/self/exe", "/usr/sbin/mysqld", 4096) = 16
2019-12-18T20:40:53.9613645Z access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9614210Z open("/usr/lib/mysql/private/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9614693Z stat("/usr/lib/mysql/private/tls/x86_64", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9615203Z open("/usr/lib/mysql/private/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9615646Z stat("/usr/lib/mysql/private/tls", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9616157Z open("/usr/lib/mysql/private/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9616621Z stat("/usr/lib/mysql/private/x86_64", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9617114Z open("/usr/lib/mysql/private/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9617354Z stat("/usr/lib/mysql/private", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-12-18T20:40:53.9617882Z open("/usr/sbin/../lib/mysql/private/tls/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9618552Z stat("/usr/sbin/../lib/mysql/private/tls/x86_64", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9619087Z open("/usr/sbin/../lib/mysql/private/tls/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9619568Z stat("/usr/sbin/../lib/mysql/private/tls", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9620098Z open("/usr/sbin/../lib/mysql/private/x86_64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9620572Z stat("/usr/sbin/../lib/mysql/private/x86_64", 0x7ffdce84fa90) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9621080Z open("/usr/sbin/../lib/mysql/private/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9621440Z stat("/usr/sbin/../lib/mysql/private", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-12-18T20:40:53.9621623Z open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9621805Z fstat(3, {st_mode=S_IFREG|0644, st_size=8636, ...}) = 0
2019-12-18T20:40:53.9621991Z mmap(NULL, 8636, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f9b9783b000
2019-12-18T20:40:53.9622166Z close(3)                                = 0
2019-12-18T20:40:53.9622622Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9623147Z open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9623363Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0Pa\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9623555Z fstat(3, {st_mode=S_IFREG|0755, st_size=135440, ...}) = 0
2019-12-18T20:40:53.9623995Z mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9b97839000
2019-12-18T20:40:53.9624244Z mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b973fe000
2019-12-18T20:40:53.9624409Z mprotect(0x7f9b97416000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9624665Z mmap(0x7f9b97615000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f9b97615000
2019-12-18T20:40:53.9625170Z mmap(0x7f9b97617000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b97617000
2019-12-18T20:40:53.9625344Z close(3)                                = 0
2019-12-18T20:40:53.9625754Z open("/usr/lib/mysql/private/libprotobuf-lite.so.3.6.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9625978Z read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\261\1\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9626172Z fstat(3, {st_mode=S_IFREG|0644, st_size=3922528, ...}) = 0
2019-12-18T20:40:53.9626398Z mmap(NULL, 2436280, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b971ab000
2019-12-18T20:40:53.9626578Z mprotect(0x7f9b971fc000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9626900Z mmap(0x7f9b973fb000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x50000) = 0x7f9b973fb000
2019-12-18T20:40:53.9627082Z close(3)                                = 0
2019-12-18T20:40:53.9627570Z open("/usr/lib/mysql/private/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9628061Z open("/usr/sbin/../lib/mysql/private/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9628751Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9629158Z open("/lib/x86_64-linux-gnu/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9629391Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\v\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9629584Z fstat(3, {st_mode=S_IFREG|0644, st_size=39256, ...}) = 0
2019-12-18T20:40:53.9629817Z mmap(NULL, 2322912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96f73000
2019-12-18T20:40:53.9629994Z mprotect(0x7f9b96f7b000, 2097152, PROT_NONE) = 0
2019-12-18T20:40:53.9630259Z mmap(0x7f9b9717b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f9b9717b000
2019-12-18T20:40:53.9632892Z mmap(0x7f9b9717d000, 184800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b9717d000
2019-12-18T20:40:53.9633102Z close(3)                                = 0
2019-12-18T20:40:53.9633595Z open("/usr/lib/mysql/private/librt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9634094Z open("/usr/sbin/../lib/mysql/private/librt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9634531Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9634946Z open("/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9635183Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340 \0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9635546Z fstat(3, {st_mode=S_IFREG|0644, st_size=31744, ...}) = 0
2019-12-18T20:40:53.9635784Z mmap(NULL, 2128832, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96d6b000
2019-12-18T20:40:53.9635969Z mprotect(0x7f9b96d72000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9636234Z mmap(0x7f9b96f71000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f9b96f71000
2019-12-18T20:40:53.9636396Z close(3)                                = 0
2019-12-18T20:40:53.9636922Z open("/usr/lib/mysql/private/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9637428Z open("/usr/sbin/../lib/mysql/private/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9637967Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9638743Z open("/usr/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9638993Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\212\1\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9639188Z fstat(3, {st_mode=S_IFREG|0644, st_size=442984, ...}) = 0
2019-12-18T20:40:53.9639423Z mmap(NULL, 2538440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96aff000
2019-12-18T20:40:53.9639605Z mprotect(0x7f9b96b62000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9639859Z mmap(0x7f9b96d61000, 40960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x62000) = 0x7f9b96d61000
2019-12-18T20:40:53.9640039Z close(3)                                = 0
2019-12-18T20:40:53.9640560Z open("/usr/lib/mysql/private/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9641091Z open("/usr/sbin/../lib/mysql/private/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9641527Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9642056Z open("/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9642312Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\360\7\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9642511Z fstat(3, {st_mode=S_IFREG|0644, st_size=2715840, ...}) = 0
2019-12-18T20:40:53.9642745Z mmap(NULL, 4825088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96665000
2019-12-18T20:40:53.9642912Z mprotect(0x7f9b968d1000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9643187Z mmap(0x7f9b96ad0000, 180224, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26b000) = 0x7f9b96ad0000
2019-12-18T20:40:53.9643742Z mmap(0x7f9b96afc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b96afc000
2019-12-18T20:40:53.9643925Z close(3)                                = 0
2019-12-18T20:40:53.9644398Z open("/usr/lib/mysql/private/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9644901Z open("/usr/sbin/../lib/mysql/private/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9645444Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9645829Z open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9646057Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\r\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9646245Z fstat(3, {st_mode=S_IFREG|0644, st_size=14640, ...}) = 0
2019-12-18T20:40:53.9646673Z mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9b97837000
2019-12-18T20:40:53.9646910Z mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96461000
2019-12-18T20:40:53.9647087Z mprotect(0x7f9b96464000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9647347Z mmap(0x7f9b96663000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f9b96663000
2019-12-18T20:40:53.9647595Z close(3)                                = 0
2019-12-18T20:40:53.9648425Z open("/usr/lib/mysql/private/libaio.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9648977Z open("/usr/sbin/../lib/mysql/private/libaio.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9649408Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9649809Z open("/lib/x86_64-linux-gnu/libaio.so.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9650033Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\6\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9650227Z fstat(3, {st_mode=S_IFREG|0644, st_size=5552, ...}) = 0
2019-12-18T20:40:53.9650463Z mmap(NULL, 2101264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b9625f000
2019-12-18T20:40:53.9650646Z mprotect(0x7f9b96260000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9650891Z mmap(0x7f9b9645f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x7f9b9645f000
2019-12-18T20:40:53.9651528Z mmap(0x7f9b96460000, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b96460000
2019-12-18T20:40:53.9651705Z close(3)                                = 0
2019-12-18T20:40:53.9652170Z open("/usr/lib/mysql/private/libnuma.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9652644Z open("/usr/sbin/../lib/mysql/private/libnuma.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9653064Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9653463Z open("/usr/lib/x86_64-linux-gnu/libnuma.so.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9653694Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\00003\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9653886Z fstat(3, {st_mode=S_IFREG|0644, st_size=44192, ...}) = 0
2019-12-18T20:40:53.9654099Z mmap(NULL, 2140320, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b96054000
2019-12-18T20:40:53.9654775Z mprotect(0x7f9b9605e000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9655090Z mmap(0x7f9b9625d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7f9b9625d000
2019-12-18T20:40:53.9655262Z close(3)                                = 0
2019-12-18T20:40:53.9655954Z open("/usr/lib/mysql/private/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9657036Z open("/usr/sbin/../lib/mysql/private/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9657655Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9658402Z open("/usr/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9658661Z read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\267\10\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9658845Z fstat(3, {st_mode=S_IFREG|0644, st_size=1566168, ...}) = 0
2019-12-18T20:40:53.9659094Z mmap(NULL, 3674720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b95cd2000
2019-12-18T20:40:53.9659275Z mprotect(0x7f9b95e44000, 2097152, PROT_NONE) = 0
2019-12-18T20:40:53.9659548Z mmap(0x7f9b96044000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x172000) = 0x7f9b96044000
2019-12-18T20:40:53.9660098Z mmap(0x7f9b96050000, 12896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b96050000
2019-12-18T20:40:53.9660280Z close(3)                                = 0
2019-12-18T20:40:53.9661129Z open("/usr/lib/mysql/private/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9661684Z open("/usr/sbin/../lib/mysql/private/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9662231Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9662599Z open("/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9662976Z read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200V\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9663169Z fstat(3, {st_mode=S_IFREG|0644, st_size=1063328, ...}) = 0
2019-12-18T20:40:53.9663393Z mmap(NULL, 3158248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b959ce000
2019-12-18T20:40:53.9663555Z mprotect(0x7f9b95ad1000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9663815Z mmap(0x7f9b95cd0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x102000) = 0x7f9b95cd0000
2019-12-18T20:40:53.9663987Z close(3)                                = 0
2019-12-18T20:40:53.9664497Z open("/usr/lib/mysql/private/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9664992Z open("/usr/sbin/../lib/mysql/private/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9665397Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9665800Z open("/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9666030Z read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220*\0\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9666220Z fstat(3, {st_mode=S_IFREG|0644, st_size=92584, ...}) = 0
2019-12-18T20:40:53.9666433Z mmap(NULL, 2188336, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b957b7000
2019-12-18T20:40:53.9666608Z mprotect(0x7f9b957cd000, 2093056, PROT_NONE) = 0
2019-12-18T20:40:53.9666868Z mmap(0x7f9b959cc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f9b959cc000
2019-12-18T20:40:53.9667152Z close(3)                                = 0
2019-12-18T20:40:53.9667635Z open("/usr/lib/mysql/private/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9668366Z open("/usr/sbin/../lib/mysql/private/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9668842Z access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9669349Z open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9669598Z read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\4\2\0\0\0\0\0"..., 832) = 832
2019-12-18T20:40:53.9669780Z fstat(3, {st_mode=S_IFREG|0755, st_size=1689360, ...}) = 0
2019-12-18T20:40:53.9670267Z mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9b97835000
2019-12-18T20:40:53.9670514Z mmap(NULL, 3795296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9b95418000
2019-12-18T20:40:53.9670698Z mprotect(0x7f9b955ad000, 2097152, PROT_NONE) = 0
2019-12-18T20:40:53.9671088Z mmap(0x7f9b957ad000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f9b957ad000
2019-12-18T20:40:53.9671664Z mmap(0x7f9b957b3000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9b957b3000
2019-12-18T20:40:53.9671845Z close(3)                                = 0
2019-12-18T20:40:53.9672309Z mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9b97833000
2019-12-18T20:40:53.9672488Z arch_prctl(ARCH_SET_FS, 0x7f9b97834000) = 0
2019-12-18T20:40:53.9672642Z mprotect(0x7f9b957ad000, 16384, PROT_READ) = 0
2019-12-18T20:40:53.9672811Z mprotect(0x7f9b959cc000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9672978Z mprotect(0x7f9b95cd0000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9673432Z mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9b97831000
2019-12-18T20:40:53.9673600Z mprotect(0x7f9b96044000, 40960, PROT_READ) = 0
2019-12-18T20:40:53.9673767Z mprotect(0x7f9b9625d000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9673931Z mprotect(0x7f9b9645f000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9674095Z mprotect(0x7f9b96663000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9674246Z mprotect(0x7f9b97615000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9674412Z mprotect(0x7f9b96ad0000, 122880, PROT_READ) = 0
2019-12-18T20:40:53.9674697Z mprotect(0x7f9b96d61000, 16384, PROT_READ) = 0
2019-12-18T20:40:53.9674865Z mprotect(0x7f9b96f71000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9675016Z mprotect(0x7f9b9717b000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9675184Z mprotect(0x7f9b973fb000, 8192, PROT_READ) = 0
2019-12-18T20:40:53.9675352Z mprotect(0x55c4804c4000, 1335296, PROT_READ) = 0
2019-12-18T20:40:53.9675520Z mprotect(0x7f9b9783e000, 4096, PROT_READ) = 0
2019-12-18T20:40:53.9675668Z munmap(0x7f9b9783b000, 8636)            = 0
2019-12-18T20:40:53.9675831Z set_tid_address(0x7f9b978342d0)         = 29
2019-12-18T20:40:53.9675995Z set_robust_list(0x7f9b978342e0, 24)     = 0
2019-12-18T20:40:53.9676296Z rt_sigaction(SIGRTMIN, {sa_handler=0x7f9b97403bd0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f9b9740f0e0}, NULL, 8) = 0
2019-12-18T20:40:53.9676638Z rt_sigaction(SIGRT_1, {sa_handler=0x7f9b97403c60, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f9b9740f0e0}, NULL, 8) = 0
2019-12-18T20:40:53.9676827Z rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
2019-12-18T20:40:53.9677048Z getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
2019-12-18T20:40:53.9677242Z brk(NULL)                               = 0x55c482303000
2019-12-18T20:40:53.9677427Z brk(0x55c482335000)                     = 0x55c482335000
2019-12-18T20:40:53.9677579Z open("/proc/self/status", O_RDONLY)     = 3
2019-12-18T20:40:53.9677756Z fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
2019-12-18T20:40:53.9677897Z read(3, "Name:\tmysqld\nUmask:\t0022\nState:\t"..., 1024) = 1024
2019-12-18T20:40:53.9678261Z read(3, "00,00000000,00000000,00000000,00"..., 1024) = 300
2019-12-18T20:40:53.9678421Z close(3)                                = 0
2019-12-18T20:40:53.9678635Z open("/sys/devices/system/node", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9678822Z fstat(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
2019-12-18T20:40:53.9678989Z getdents(3, /* 10 entries */, 32768)    = 312
2019-12-18T20:40:53.9679170Z open("/sys/devices/system/node/node0/meminfo", O_RDONLY) = 4
2019-12-18T20:40:53.9679447Z fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
2019-12-18T20:40:53.9679639Z read(4, "Node 0 MemTotal:        7112216 "..., 4096) = 1105
2019-12-18T20:40:53.9679808Z read(4, "", 4096)                       = 0
2019-12-18T20:40:53.9679971Z close(4)                                = 0
2019-12-18T20:40:53.9680120Z getdents(3, /* 0 entries */, 32768)     = 0
2019-12-18T20:40:53.9680280Z close(3)                                = 0
2019-12-18T20:40:53.9680445Z sched_getaffinity(0, 512, [0, 1])       = 16
2019-12-18T20:40:53.9680657Z open("/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
2019-12-18T20:40:53.9680832Z fstat(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
2019-12-18T20:40:53.9680998Z getdents(3, /* 19 entries */, 32768)    = 576
2019-12-18T20:40:53.9681162Z getdents(3, /* 0 entries */, 32768)     = 0
2019-12-18T20:40:53.9681324Z close(3)                                = 0
2019-12-18T20:40:53.9681481Z open("/proc/self/status", O_RDONLY)     = 3
2019-12-18T20:40:53.9681661Z fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
2019-12-18T20:40:53.9681795Z read(3, "Name:\tmysqld\nUmask:\t0022\nState:\t"..., 1024) = 1024
2019-12-18T20:40:53.9681986Z read(3, "00,00000000,00000000,00000000,00"..., 1024) = 300
2019-12-18T20:40:53.9682140Z read(3, "", 1024)                       = 0
2019-12-18T20:40:53.9682302Z close(3)                                = 0
2019-12-18T20:40:53.9682483Z futex(0x7f9b9605103c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
2019-12-18T20:40:53.9682669Z futex(0x7f9b96051048, FUTEX_WAKE_PRIVATE, 2147483647) = 0
2019-12-18T20:40:53.9682822Z get_mempolicy(NULL, NULL, 0, NULL, 0)   = 0
2019-12-18T20:40:53.9683328Z access("/usr/local/sbin/mysqld", X_OK)  = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9683763Z access("/usr/local/bin/mysqld", X_OK)   = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9683942Z access("/usr/sbin/mysqld", X_OK)        = 0
2019-12-18T20:40:53.9684212Z lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-12-18T20:40:53.9684412Z lstat("/usr/sbin", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-12-18T20:40:53.9684628Z lstat("/usr/sbin/mysqld", {st_mode=S_IFREG|0755, st_size=61336880, ...}) = 0
2019-12-18T20:40:53.9685091Z stat("/etc/my.cnf", 0x7ffdce84b1d0)     = -1 ENOENT (No such file or directory)
2019-12-18T20:40:53.9685293Z stat("/etc/mysql/my.cnf", {st_mode=S_IFREG|0664, st_size=1174, ...}) = 0
2019-12-18T20:40:53.9685466Z open("/etc/mysql/my.cnf", O_RDONLY)     = 3
2019-12-18T20:40:53.9685646Z fstat(3, {st_mode=S_IFREG|0664, st_size=1174, ...}) = 0
2019-12-18T20:40:53.9685838Z read(3, "# Copyright (c) 2017, Oracle and"..., 4096) = 1174
2019-12-18T20:40:53.9686307Z open("/etc/mysql/conf.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 EACCES (Permission denied)
2019-12-18T20:40:53.9686474Z write(2, "mysqld: ", 8mysqld: )                 = 8
2019-12-18T20:40:53.9687027Z write(2, "Can't read dir of '/etc/mysql/co"..., 72Can't read dir of '/etc/mysql/conf.d/' (OS errno 13 - Permission denied)) = 72
2019-12-18T20:40:53.9687183Z write(2, "\n", 1
2019-12-18T20:40:53.9687443Z )                       = 1
2019-12-18T20:40:53.9687587Z close(3)                                = 0
2019-12-18T20:40:53.9687728Z write(2, "mysqld: ", 8mysqld: )                 = 8
2019-12-18T20:40:53.9688239Z write(2, "[ERROR] Fatal error in defaults "..., 58[ERROR] Fatal error in defaults handling. Program aborted!) = 58
2019-12-18T20:40:53.9688392Z write(2, "\n", 1
2019-12-18T20:40:53.9688516Z )                       = 1
2019-12-18T20:40:53.9688678Z exit_group(1)                           = ?
2019-12-18T20:40:53.9688814Z +++ exited with 1 +++
2019-12-18T20:41:53.7045783Z Cleaning up orphan processes
@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

I am going to try kind from HEAD and report back. But /etc/mysql/conf.d folder is not mounted to any disk. This is just Docker overlay fs.

ah, this is a directory in the mysql image? not a hostPath or a volume? I saw a PVC in your manifests when I took a quick look last night and guessed wrongly :-)

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

seeing a few comments around that suggest a relation to apparmor

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

Do you think https://unix.stackexchange.com/questions/229782/overlayfs-doesnt-work-with-unprivileged-user-namespace might be somehow related?

The thing is I can ls -l /etc/mysql/conf.d, I can touch a new file from that directory inside the pod. Why does mysqld fails?

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

Do you think https://unix.stackexchange.com/questions/229782/overlayfs-doesnt-work-with-unprivileged-user-namespace might be somehow related?

possibly. I don't know how github actions setup docker in this environment or what security mechanisms / filesystems they have enabled...

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

GitHub uses packer to build their VM. https://github.com/actions/virtual-environments

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 18, 2019

can you test #1183 ?
not sure the best way to get it into your github actions (could clone it, check out the ref, make install?)

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 18, 2019

Checking from your fork.

@tamalsaha

This comment has been minimized.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 19, 2019

:/

Can you obtain some more details about the github actions environment?

  • what docker version / how was it installed (snap? etc.?)
  • how is docker configured? anything interesting like user namespace remapping?
  • what filesystem is in use and what storage driver?
  • what permissions does the docker data root have?
  • what security mechanisms are in use? apparmor? seccomp?
@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 19, 2019

Here are some info:

$ uname -a
Linux fv-az131 5.0.0-1027-azure #29~18.04.1-Ubuntu SMP Mon Nov 25 21:18:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 3.0.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-1027-azure
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.783GiB
 Name: fv-az131
 ID: KRSI:74QW:YCD2:XEK7:4WO2:DOAO:FRYG:MP4B:J2YH:N63L:5SGR:JV5A
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 19, 2019

Also, I don't know where to check for:

how is docker configured? anything interesting like user namespace remapping?
what permissions does the docker data root have?

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

@tamalsaha tamalsaha commented Dec 20, 2019

@BenTheElder , I think this answer probably gives the clue docker-library/mysql#617 (comment)

I am still trying to understand it.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 20, 2019

how is docker configured? anything interesting like user namespace remapping?

the daemon config.json. most of this is surfaced in docker info

what permissions does the docker data root have?

this bit from docker info:
Docker Root Dir: /var/lib/docker

that's where docker stores runtime data (containers, images, volumes... )

we can check the permissions on that for sanity

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 20, 2019

@BenTheElder , I think this answer probably gives the clue docker-library/mysql#617 (comment)

ah that makes sense!

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Dec 20, 2019

this explanation is good -- moby/moby#7512 (comment)

an option is to disable apparmor on the host for mysql

kind needs docs for this under known-issues

I'm not sure that we can do much more. kind would of course not run priv containers if we didn't need to, similarly we need to support priv containers inside the kind nodes because kubernetes system components run this way (eg kube-proxy).

containerd / kubelet rootless might be viable in the near future, but it's not quite ready, and i'm not sure if we can ship it right away when it's ready because most of our users's hosts won't be ready kubernetes/enhancements#1371

cgroups v1 -> v2 is going to be "fun" for hacks like KIND 🙃

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Jan 2, 2020

I don't think KIND can do much else here, we should document this in known-issues though.

@BenTheElder

This comment has been minimized.

Copy link
Member

@BenTheElder BenTheElder commented Jan 7, 2020

@BenTheElder BenTheElder closed this Jan 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.