Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow moby-engine instead of docker-ce? #1294

Closed
jonpugh opened this issue Nov 16, 2018 · 5 comments
Closed

Allow moby-engine instead of docker-ce? #1294

jonpugh opened this issue Nov 16, 2018 · 5 comments

Comments

@jonpugh
Copy link
Contributor

@jonpugh jonpugh commented Nov 16, 2018

I'm on the latest Fedora 29. I couldn't install the RPM out of the box.

It depends on docker-ce but I had moby-engine instead.

Is it possible to allow that dependency instead?

docker-ce isn't available in 29 yet, I used this workaround to get docker-ce running: docker/for-linux#430 (comment)

@dustinleblanc
Copy link
Sponsor Collaborator

@dustinleblanc dustinleblanc commented Nov 16, 2018

@jonpugh the work around is to run Lando from source: https://docs.devwithlando.io/installation/installing.html#from-source

@stale
Copy link

@stale stale bot commented Jan 31, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions and please check out this if you are wondering why we auto close issues.

@stale stale bot added the stale label Jan 31, 2019
@stale stale bot closed this as completed Feb 7, 2019
@pirog pirog removed the stale label Feb 24, 2020
@ekes
Copy link

@ekes ekes commented May 5, 2020

So Fedora 32!

Compiling from source works, no errors.

But the first lando project I tried failed with two containers giving an error that they aren't cgroups v2 compatible. So force the kernel back to v1. Now it's getting images but not making the containers at all.

I'll post here if I come up with ideas. Any other suggestions welcome.

@dustinleblanc
Copy link
Sponsor Collaborator

@dustinleblanc dustinleblanc commented May 5, 2020

@ekes are docker and docker-compose available? I know the Fedora community has some really, really cool looking new container toys in their new setups (with silverblue, etc), and I can imagine some of things are not actually running on top of docker (using buildah, podman, etc). We currently only support Docker and compose, so it's important to know for sure what you're using.

@ekes
Copy link

@ekes ekes commented May 6, 2020

I've found two options for getting Lando working with Fedora 32. In short below, the full debugging and reasoning for each step below.

Method one: using moby-engine docker

  • Disable cgroups v2 in the kernel: sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" Reboot.
  • Install docker via moby: sudo dnf install moby-engine
  • Switch firewalld to use iptables (default is nftables). In /etc/firewalld/firewalld.conf set FirewallBackend=iptables. Restart firewall sudo systemctl restart firewalld.
  • Disable SELinux enforcing sudo setenforce 0 (this is temporary you can also do this permenantly if you really want in /etc/selinux/config)
  • Install lando from source https://docs.lando.dev/basics/installation.html#from-source.

Method two: using docker-ce

  • Disable cgroups v2 in the kernel: sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" Reboot.
  • Add the docker-ce repo sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo but then edit the repo /etc/yum.repos.d/docker-ce.repo to access the Fedora 31 packages s/$releasever/31/.
  • Install docker: sudo dnf install docker-ce
  • Switch firewalld to use iptables (default is nftables). In /etc/firewalld/firewalld.conf set FirewallBackend=iptables. Restart firewall sudo systemctl restart firewalld.
  • Install lando from source https://docs.lando.dev/basics/installation.html#from-source. Or not tested, but the package will probably work here as docker-ce is installed.

Short notes

To undo steps:

  • SELinux sudo setenforce 1
  • Firewall change firewalld.conf back to FirewallBackend=nftables restart.
  • Allow cgroups v2 sudo grubby --update-kernel=ALL --remove-args="systemd.unified_cgroup_hierarchy=0" Reboot.

The SELinux line is only in the moby installed docker. No idea why both moby-engine and docker-ce bring in the dependency container-selinux from the fedora repo at time of installing 2:2.132.0-1.fc32 Some SEAlert messages in the long notes.

The Firewall is because lando is setting iptables rules, they silently (no errors thrown) seem not to work with nftables.

Disabling cgroups v2 seems to mainly be because even the moby docker doesn't quite work with it on, works a bit, but the breaks. More in the detailed debug.

Longer notes

Error messages and process to get to the above. Might give clues how to fix these things, or better ways of doing things.

Started out with moby and cgroups v2 enabled

 moby-engine               x86_64         19.03.8-1.ce.gitafacb8b.fc32

docker --version
Docker version 19.03.8, build afacb8b

 docker-compose                      noarch            1.25.4-1.fc32

docker-compose --version
docker-compose version 1.25.4, build unknown

/usr/share/lando/bin/docker-compose --version
docker-compose version 1.25.5, build 8a1c60f6

At times symlinked the packaged docker version rather than the downloaded one, at it worked with lando, not extensively tested, but lando works and doesn't complain about having a too-shiny-new version.

The cgroups error refers to runc, a bit of prodding around suggested this might be running it in non-root mode (but even hacking lando a bit to run it in root doesn't change the error -- well it wanted testing!). Anyway it's something else about runc and cgroups v2, I guess this is one of several things that want booting upsteam into a Fedora issue, but I've really not got a clue where to unpick what is what to make issues for all this.

The message for the first service (it repeats the same message for more than one service):

ERROR: for test_appserver_1  Cannot start service appserver: OCI runtime create failed: this version of runc doesn't work on cgroups v2: unknown

ERROR: for appserver  Cannot start service appserver: OCI runtime create failed: this version of runc doesn't work on cgroups v2: unknown

ERROR: Encountered errors while bringing up the project.
ERROR ==>  message=, stack=Error
    at /home/user/src/lando/lando/lib/shell.js:156:44
From previous event:
    at Shell.sh (/home/user/src/lando/lando/lib/shell.js:148:6)
    at Object.exports.dc (/home/user/src/lando/lando/lib/bootstrap.js:110:16)
    at compose (/home/user/src/lando/lando/lib/bootstrap.js:189:43)
    at /home/user/src/lando/lando/lib/router.js:121:61
    at /home/user/src/lando/lando/lib/router.js:21:80
From previous event:
    at retryEach (/home/user/src/lando/lando/lib/router.js:21:42)
    at Object.exports.start (/home/user/src/lando/lando/lib/router.js:121:36)
    at /home/user/src/lando/lando/lib/router.js:77:22
From previous event:
    at /home/user/src/lando/lando/lib/router.js:74:47
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at Object.exports.run (/home/user/src/lando/lando/lib/router.js:69:66)
    at run (/home/user/src/lando/lando/lib/engine.js:18:59)
    at /home/user/src/lando/lando/lib/router.js:26:15
From previous event:
    at Object.exports.eventWrapper (/home/user/src/lando/lando/lib/router.js:26:4)
    at Engine.engineCmd (/home/user/src/lando/lando/lib/engine.js:18:104)
    at Engine.run (/home/user/src/lando/lando/lib/engine.js:353:17)
    at Object.exports.runBuild (/home/user/src/lando/lando/plugins/lando-services/lib/utils.js:125:23)
    at AsyncEvents.<anonymous> (/home/user/src/lando/lando/plugins/lando-services/app.js:80:20)
    at AsyncEvents.handle (/home/user/src/lando/lando/lib/events.js:84:25)
    at /home/user/src/lando/lando/lib/events.js:117:21
From previous event:
    at AsyncEvents.emit (/home/user/src/lando/lando/lib/events.js:111:20)
    at /home/user/src/lando/lando/lib/app.js:419:29
From previous event:
    at App.start (/home/user/src/lando/lando/lib/app.js:419:6)
    at /home/user/src/lando/lando/lib/app.js:350:22   
From previous event:
    at App.rebuild (/home/user/src/lando/lando/lib/app.js:350:6)
    at Object.run (/home/user/src/lando/lando/plugins/lando-core/tasks/rebuild.js:32:20)
    at /home/user/src/lando/lando/lib/cli.js:378:54
From previous event:
    at /home/user/src/lando/lando/lib/cli.js:376:10
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at Object.handler (/home/user/src/lando/lando/lib/cli.js:310:37)
    at Object.runCommand (/home/user/src/lando/lando/node_modules/yargs/lib/command.js:238:44)
    at Object.parseArgs [as _parseArgs] (/home/user/src/lando/lando/node_modules/yargs/yargs.js:1063:30)
    at Function.get [as argv] (/home/user/src/lando/lando/node_modules/yargs/yargs.js:1004:21)
    at Cli.init (/home/user/src/lando/lando/lib/cli.js:231:51)
    at Cli.run (/home/user/src/lando/lando/lib/cli.js:435:10)
    at Object.<anonymous> (/home/user/src/lando/lando/bin/lando.js:55:7)
    at Module._compile (internal/modules/cjs/loader.js:1158:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
    at Module.load (internal/modules/cjs/loader.js:1002:32)
    at Function.Module._load (internal/modules/cjs/loader.js:901:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
    at internal/main/run_main_module.js:18:47, __stackCleaned__=true
Starting landocasetupkenobi38ahsoka9ea429ec103411ace56c18021c9f0302d5948351_ca_1 ... error

Next test disabling cgroups v2

So for each of these steps I'm docker container prune docker image prune docker network prune just to clear up.

Running sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" rebooting, and still using the above. So this is before the firewall has been changed, or selinux looked at. It silently fails to find the container (including when running lando -v)

test  09:33:36 INFO ==> starting app... 
Starting landocasetupkenobi38ahsoka9ea429ec103411ace56c18021c9f0302d5948351_ca_1 ... done
ERROR: No container found for ca_1
test  09:33:38 ERROR ==>  message=, stack=Error
    at /home/user/src/lando/lando/lib/shell.js:156:44
From previous event:
    at Shell.sh (/home/user/src/lando/lando/lib/shell.js:148:6)
    at Object.exports.dc (/home/user/src/lando/lando/lib/bootstrap.js:110:16)
    at compose (/home/user/src/lando/lando/lib/bootstrap.js:189:43)
    at /home/user/src/lando/lando/lib/router.js:85:15
From previous event:
    at /home/user/src/lando/lando/lib/router.js:85:4
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at Object.exports.run (/home/user/src/lando/lando/lib/router.js:69:66)
    at run (/home/user/src/lando/lando/lib/engine.js:18:59)
    at /home/user/src/lando/lando/lib/router.js:26:15
From previous event:
    at Object.exports.eventWrapper (/home/user/src/lando/lando/lib/router.js:26:4)
    at Engine.engineCmd (/home/user/src/lando/lando/lib/engine.js:18:104)
    at Engine.run (/home/user/src/lando/lando/lib/engine.js:353:17)
    at AsyncEvents.<anonymous> (/home/user/src/lando/lando/plugins/lando-core/index.js:77:27)
    at AsyncEvents.handle (/home/user/src/lando/lando/lib/events.js:84:25)
    at /home/user/src/lando/lando/lib/events.js:117:21
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at AsyncEvents.emit (/home/user/src/lando/lando/lib/events.js:111:20)
    at /home/user/src/lando/lando/lib/router.js:25:22
From previous event:
    at Object.exports.eventWrapper (/home/user/src/lando/lando/lib/router.js:25:4)
    at Engine.engineCmd (/home/user/src/lando/lando/lib/engine.js:18:104)
    at Engine.start (/home/user/src/lando/lando/lib/engine.js:425:17)
    at /home/user/src/lando/lando/plugins/lando-proxy/app.js:131:29
From previous event:
    at AsyncEvents.<anonymous> (/home/user/src/lando/lando/plugins/lando-proxy/app.js:117:8)
    at AsyncEvents.handle (/home/user/src/lando/lando/lib/events.js:84:25)
    at /home/user/src/lando/lando/lib/events.js:117:21
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at AsyncEvents.emit (/home/user/src/lando/lando/lib/events.js:111:20)
    at /home/user/src/lando/lando/lib/app.js:419:29
From previous event:
    at App.start (/home/user/src/lando/lando/lib/app.js:419:6)
    at Object.run (/home/user/src/lando/lando/plugins/lando-core/tasks/start.js:17:20)
    at /home/user/src/lando/lando/lib/cli.js:378:54
From previous event:
    at /home/user/src/lando/lando/lib/cli.js:376:10
    at processImmediate (internal/timers.js:456:21)
From previous event:
    at Object.handler (/home/user/src/lando/lando/lib/cli.js:310:37)
    at Object.runCommand (/home/user/src/lando/lando/node_modules/yargs/lib/command.js:238:44)
    at Object.parseArgs [as _parseArgs] (/home/user/src/lando/lando/node_modules/yargs/yargs.js:1063:30)
    at Function.get [as argv] (/home/user/src/lando/lando/node_modules/yargs/yargs.js:1004:21)
    at Cli.init (/home/user/src/lando/lando/lib/cli.js:231:51)
    at Cli.run (/home/user/src/lando/lando/lib/cli.js:435:10)
    at Object.<anonymous> (/home/user/src/lando/lando/bin/lando.js:55:7)
    at Module._compile (internal/modules/cjs/loader.js:1158:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
    at Module.load (internal/modules/cjs/loader.js:1002:32)
    at Function.Module._load (internal/modules/cjs/loader.js:901:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
    at internal/main/run_main_module.js:18:47, __stackCleaned__=true

Switching to Docker CE

So swapped the repos, clean up docker, remove moby, install docker-ce. This brings with it:

 docker-ce                   x86_64           3:19.03.8-3.fc31           docker-ce-stable

 container-selinux           noarch           2:2.132.0-1.fc32           updates                     48 k
 containerd.io               x86_64           1.2.13-3.1.fc31            docker-ce-stable            23 M
 docker-ce-cli               x86_64           1:19.03.8-3.fc31           docker-ce-stable            39 M
 libcgroup                   x86_64           0.42.2-1.fc32              fedora                      68 k

Now running lando start

Creating test_appserver_1 ... done
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0curl: (6) Could not resolve host: github.com

Ah!HA! curl works on the commandline. It's the firewall. Switch to iptables as shown above, and we're good to go.

Switching back to moby-engine with iptables

So was it the firewall causing the silent container failure. Well maybe, but still, tidy up docker, uninstall docker-ce, install moby-engine, run lando and:

Creating test_appserver_1 ... done
ERROR: No container found for appserver_1
ERROR ==>  message=, stack=Error
    at /home/user/src/lando/lando/lib/shell.js:156:44
From previous event:
    at Shell.sh (/home/user/src/lando/lando/lib/shell.js:148:6)
    at Object.exports.dc (/home/user/src/lando/lando/lib/bootstrap.js:110:16)
    at compose (/home/user/src/lando/lando/lib/bootstrap.js:189:43)
    at /home/user/src/lando/lando/lib/router.js:85:15

Oh! Well.

SELinux

Hold up there are suddenly loads of messages in sealert. SELinux is now blocking things it wasn't with docker-ce.

The first one is accessing lando-entrypoint.sh, having solved that it just throws more, stuff like:

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that find should be allowed execute access on the add-cert.sh file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'find' --raw | audit2allow -M my-find
# semodule -X 300 -i my-find.pp

Additional Information:
Source Context                system_u:system_r:container_t:s0:c423,c846
Target Context                unconfined_u:object_r:user_home_t:s0
Target Objects                add-cert.sh [ file ]
Source                        find
Source Path                   find
Port                          <Unknown>
Host                          localhost.localdomain
Source RPM Packages           
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-3.14.5-32.fc32.noarch
Local Policy RPM              selinux-policy-targeted-3.14.5-32.fc32.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     localhost.localdomain
Platform                      Linux localhost.localdomain 5.6.8-300.fc32.x86_64 #1 SMP
                              Wed Apr 29 19:01:34 UTC 2020 x86_64 x86_64
Alert Count                   4
First Seen                    2020-05-06 10:46:25 CEST
Last Seen                     2020-05-06 10:46:39 CEST
Local ID                      c1a6c2d4-8bad-4909-8b01-5dc7263420d9

Raw Audit Messages
type=AVC msg=audit(1588754799.245:4297): avc:  denied  { execute } for  pid=54953 comm="find" name="add-cert.sh" dev="dm-3" ino=2230575 scontext=system_u:system_r:container_t:s0:c423,c846 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0


Hash: find,container_t,user_home_t,file,execute
*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that chown should be allowed setattr access on the test directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'chown' --raw | audit2allow -M my-chown
# semodule -X 300 -i my-chown.pp

Additional Information:
Source Context                system_u:system_r:container_t:s0:c423,c846
Target Context                unconfined_u:object_r:user_home_t:s0
Target Objects                test [ dir ]
Source                        chown
Source Path                   chown
Port                          <Unknown>
Host                          localhost.localdomain
Source RPM Packages           
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-3.14.5-32.fc32.noarch
Local Policy RPM              selinux-policy-targeted-3.14.5-32.fc32.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     localhost.localdomain
Platform                      Linux localhost.localdomain 5.6.8-300.fc32.x86_64 #1 SMP
                              Wed Apr 29 19:01:34 UTC 2020 x86_64 x86_64
Alert Count                   1
First Seen                    2020-05-06 10:46:26 CEST
Last Seen                     2020-05-06 10:46:26 CEST
Local ID                      0db55e73-22af-4142-a1a9-cef347c506db

Raw Audit Messages
type=AVC msg=audit(1588754786.771:4239): avc:  denied  { setattr } for  pid=54493 comm="chown" name="test" dev="dm-3" ino=2769424 scontext=system_u:system_r:container_t:s0:c423,c846 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=0


Hash: chown,container_t,user_home_t,dir,setattr

find, chmod, openssl, cp, apache2, lando-entrypoint, sh, grep, all largely read edit setattr to files; one more that's not traefik getting to its socket.

The observant amongst you will see my solution to this in the dump:

SELinux is preventing traefik from connectto access on the unix_stream_socket /run/docker.sock.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that traefik should be allowed connectto access on the docker.sock unix_stream_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'traefik' --raw | audit2allow -M my-traefik
# semodule -X 300 -i my-traefik.pp

Additional Information:
Source Context                system_u:system_r:container_t:s0:c455,c879
Target Context                system_u:system_r:container_runtime_t:s0
Target Objects                /run/docker.sock [ unix_stream_socket ]
Source                        traefik
Source Path                   traefik
Port                          <Unknown>
Host                          localhost.localdomain
Source RPM Packages           
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-3.14.5-32.fc32.noarch
Local Policy RPM              selinux-policy-targeted-3.14.5-32.fc32.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Permissive
Host Name                     localhost.localdomain
Platform                      Linux localhost.localdomain 5.6.8-300.fc32.x86_64 #1 SMP
                              Wed Apr 29 19:01:34 UTC 2020 x86_64 x86_64
Alert Count                   3
First Seen                    2020-05-06 10:55:05 CEST
Last Seen                     2020-05-06 10:57:44 CEST
Local ID                      07dd6b1c-c2f2-4bc0-be13-dee7c47db7cf

Raw Audit Messages
type=AVC msg=audit(1588755464.503:4772): avc:  denied  { connectto } for  pid=60981 comm="traefik" path="/run/docker.sock" scontext=system_u:system_r:container_t:s0:c455,c879 tcontext=system_u:system_r:container_runtime_t:s0 tclass=unix_stream_socket permissive=1


Hash: traefik,container_t,container_runtime_t,unix_stream_socket,connectto

Yep I just switched to permissive mode. I'm sure the SELinux rules can be corrected, but it's not a cut and paste and import job, so one more change sudo setenforce 0 and lando works with the moby installed docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants