Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NixOS: Run Docker containers as declarative systemd services #55179

Merged
merged 12 commits into from Mar 24, 2019

Conversation

@benley
Copy link
Member

commented Feb 4, 2019

Motivation for this change

https://discourse.nixos.org/t/deploying-docker-containers-declaratively/693

#37553 #26075

Related to #26075 but the implementation is less general; this one just offers an interface to run containers as systemd units from pre-existing Docker images.

cc @copumpkin

TODO:

  • NixOS tests
  • Documentation
  • Environment vars
  • Publishable ports
  • Volume mounts
  • Log driver option, systemd/journald logging integration
  • override UID/GID, workdir, entrypoint, cmd
  • systemd unit generator
@nixos-discourse

This comment has been minimized.

Copy link

commented Feb 4, 2019

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/deploying-docker-containers-declaratively/693/9

@nlewo
Copy link
Member

left a comment

It would be nice to add a test for this module also.

Show resolved Hide resolved nixos/modules/virtualisation/docker-containers.nix Outdated
Show resolved Hide resolved nixos/modules/virtualisation/docker-containers.nix Outdated
@benley

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

It would be nice to add a test for this module also.

I absolutely will add tests for this stuff.

@costrouc

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2019

Is it too complicated for this PR to address mapping ports as well? Would this require systemd.sockets? I'm thinking with port mapping this PR would cover 90% of all use cases.

@benley

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

Is it too complicated for this PR to address mapping ports as well? Would this require systemd.sockets? I'm thinking with port mapping this PR would cover 90% of all use cases.

It wouldn't be too hard to handle mapping ports here, as long as we don't try to cover all possible networking setups. I'll have a go at adding it.

@benley

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

(and no, I don't believe this will require interacting with systemd.sockets)

@benley

This comment has been minimized.

Copy link
Member Author

commented Feb 4, 2019

I've added support for publishing ports, setting environment variables, and overriding user and workdir.

@benley

This comment was marked as resolved.

Copy link
Member Author

commented Feb 5, 2019

There are some docbook validation errors; I'm working on it.

@expipiplus1

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

Looking forward to using this!

I think it would be worthwhile to take a look at this repository: https://github.com/ibuildthecloud/systemd-docker as well as the linked discussions moby/moby#6791 and https://groups.google.com/forum/#!topic/coreos-dev/wf7G6rA7Bf4/discussion

@Infinisil
Copy link
Member

left a comment

The code looks good to me now (I haven't tested anything though)

@benley

This comment has been minimized.

Copy link
Member Author

commented Feb 17, 2019

is there something else necessary to get ofborg to run the tests again? It looks like I'm in the known-users list in the config.

@Infinisil

This comment has been minimized.

Copy link
Member

commented Feb 18, 2019

Weird, lemme try again

@GrahamcOfBorg test docker-containers

@eljojo

This comment has been minimized.

Copy link

commented Mar 2, 2019

👋 Hi everyone, I'm somewhat new to nixos but would like to try out this branch in my local system.
Is there an easy way in which I can point to it? Any idea on when this will be publicly available?

after = [ "docker.service" "docker.socket" ];
requires = [ "docker.service" "docker.socket" ];
serviceConfig = {
ExecStart = concatStringsSep " \\\n " ([

This comment has been minimized.

Copy link
@danbst

danbst Mar 13, 2019

Contributor

why not concatStringsSep " "?

This comment has been minimized.

Copy link
@benley

benley Mar 18, 2019

Author Member

This makes the output considerably prettier, with indented line continuations instead of one very long line.

ExecStart = /nix/store/foolkasdjflkasdflajd-blah/bin/foo \
  --bar \
  --baz=buz \
  --etc-etc-etc \
  --really-long-option=laksjdfklajfdkajdslkfalkdsfjalsdjfalkfdsajfdsla

versus

ExecStart = /nix/store/foolkasdjflkasdflajd-blah/bin/foo --bar --baz=buz --etc-etc-etc --really-long-option=laksjdfklajfdkajdslkfalkdsfjalsdjfalkfdsajfdsla
@danbst

danbst approved these changes Mar 13, 2019

Copy link
Contributor

left a comment

docker-compose on steroids! Did run test, it worked for me, I guess ofborg has unrelated problems.

@benley benley changed the title WIP: Run Docker containers as declarative systemd services NixOS: Run Docker containers as declarative systemd services Mar 18, 2019

@benley benley requested a review from nlewo Mar 18, 2019

@tomberek

This comment has been minimized.

Copy link
Contributor

commented Mar 24, 2019

Tested. Seems to work well and respond as expected to configuration.nix changes. There are some nuances to deal with like obtaining images (preloader seems difficult to setup), using hostnames or /etc/hosts in containers to avoid hard-coding IPs. Overall, good work! This can serve as a band-aid to deploy applications that are only feasible to run as upstream-provided docker images (or are difficult to port, eg: i got onlyoffice to work with nextcloud using docker-containers).

@danbst danbst merged commit c940053 into NixOS:master Mar 24, 2019

10 checks passed

grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@danbst

This comment has been minimized.

Copy link
Contributor

commented Mar 24, 2019

thanks all!

@benley

This comment has been minimized.

Copy link
Member Author

commented Mar 25, 2019

Woot!

@benley benley deleted the benley:benley/docker-containers branch Mar 25, 2019

andrew-d added a commit to andrew-d/nixpkgs that referenced this pull request Apr 7, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

(cherry picked from commit c940053)
@expipiplus1

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2019

Just checking, but did anyone (@benley, @danbst) take a look at the link I posted: https://github.com/ibuildthecloud/systemd-docker as well as the linked discussions moby/moby#6791 and https://groups.google.com/forum/#!topic/coreos-dev/wf7G6rA7Bf4/discussion

It's not really my area, but I couldn't see any handling of this edge case in this PR

The short of it is that systemd does not actually supervise the Docker container but instead the Docker client. This makes systemd incapable of reliably managing Docker containers without hitting a bunch of really odd situations.

@nyanloutre

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2019

@benley

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2019

Just checking, but did anyone (@benley, @danbst) take a look at the link I posted: https://github.com/ibuildthecloud/systemd-docker as well as the linked discussions moby/moby#6791 and https://groups.google.com/forum/#!topic/coreos-dev/wf7G6rA7Bf4/discussion

It's not really my area, but I couldn't see any handling of this edge case in this PR

The short of it is that systemd does not actually supervise the Docker container but instead the Docker client. This makes systemd incapable of reliably managing Docker containers without hitting a bunch of really odd situations.

Yes, I studied these cases extensively as I was putting this together. The way this is implemented in NixOS, systemd does not attempt to own the container or the processes within it. Instead, it just asks docker to run a thing, keeps it in the foreground, and if the container exits then systemd will know because the docker run command terminates. When systemd wants to stop or restart a container, it also just asks docker to do it with docker stop.

I would love to swap out the implementation at some point to avoid using the Docker daemon, using runc or podman or something like those, and I did try to build this in a way that would make that possible.

LinArcX added a commit to LinArcX/nixpkgs that referenced this pull request Apr 13, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

andrew-d added a commit to andrew-d/nixpkgs that referenced this pull request Apr 13, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

(cherry picked from commit c940053)

andrew-d added a commit to andrew-d/nixpkgs that referenced this pull request Apr 13, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

(cherry picked from commit c940053)
@expipiplus1

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2019

@benley great, thanks!

andrew-d added a commit to andrew-d/nixpkgs that referenced this pull request Apr 25, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

(cherry picked from commit c940053)

andrew-d added a commit to andrew-d/nixpkgs that referenced this pull request May 26, 2019

NixOS: Run Docker containers as declarative systemd services (NixOS#5…
…5179)

* WIP: Run Docker containers as declarative systemd services

* PR feedback round 1

* docker-containers: add environment, ports, user, workdir options

* docker-containers: log-driver, string->str, line wrapping

* ExecStart instead of script wrapper, %n for container name

* PR feedback: better description and example formatting

* Fix docbook formatting (oops)

* Use a list of strings for ports, expand documentation

* docker-continers: add a simple nixos test

* waitUntilSucceeds to avoid potential weird async issues

* Don't enable docker daemon unless we actually need it

* PR feedback: leave ExecReload undefined

(cherry picked from commit c940053)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.