Improvements for declarative nixos containers #3021

chrisfarms · 2014-06-19T20:45:51Z

Use unique service units for each container rather than using
templates (containers@xxx) to work around reloading issues.
Fix triggering of switching to new container configuration when
the container's config has changed.
Add new options: macvlanInterface, macvlanAddress and
macvlanPrefixLength to enable a macvlan interface within the
container (eth1).
Do not force a veth pair to be created when privateNetwork is
true (Sometimes you just want to isolate the container).
veth pairs are now created only when localAddress or hostAddress
are set.
Add a wantedBy option so that you can control when the container
unit should be started (defaults to multi-user.target).

* Use unique service units for each container rather than using templates (containers@xxx) to work around reloading issues. * Fix triggering of switching to new container configuration when the container's config has changed. * Add new options: `macvlanInterface`, `macvlanAddress` and `macvlanPrefixLength` to enable a macvlan interface within the container (eth1). * Do not force a veth pair to be created when `privateNetwork` is true (Sometimes you just want to isolate the container). * veth pairs are now created only when `localAddress` or `hostAddress` are set. * Add a `wantedBy` option so that you can control when the container unit should be started (defaults to `multi-user.target`).

thoughtpolice · 2014-06-19T21:06:49Z

👍 to all these changes, in particular making declarative containers start on boot.

edolstra · 2014-06-19T21:16:15Z

Doesn't this break imperative containers? I used unique per-container units originally, but replaced it with a template to support imperative containers (ba88db3).

What were the reloading issues exactly?

chrisfarms · 2014-06-19T21:25:48Z

Oh really dammit, I tried searching for what it might affect and couldn't spot anything.

Can you point me towards where imperative containers might use the unit, and I'll investigate?

The reloading issues were that it simply didn't reload anything. Changes to the container's config would not trigger a switch in the container at all for anything I tried. It had a "FIXME" label next to the reload, so I'm guessing it never worked?

Separate units were also required so that each unit could be tailored individually. I've just added a wantedByso far, but when there are more complex dependencies between containers, it is going to be useful to allow more customisation of the container service.

chrisfarms · 2014-06-19T21:36:32Z

Ok ... yep you're right.

gets on it

…late instance' during switch if it does not have a unit file.

chrisfarms · 2014-06-19T23:58:56Z

I've discovered the root cause of the reloading issue ... switch-to-configuration treats all units in the form template@instance.service as if they must be instantiated from a template. However systemd allows you to create units in the form template@instance.service specifically for the cases where you want to override the template.

It looks like this appeared around aa6fd9f

@edolstra can you remember the reason for treating template instances that way?
In the latest commit I've reverted to using the template style to keep nixos-container happy, but to resolve the reloading issue I have changed the logic for detecting template instances during switch to check for the existence of unit files in /etc/systemd/system before deciding it is ephemeral. Does that make sense?

…er's journal is linked with the host. Setting this to 'guest' keeps the journal data in the container's filesystem while still being able to montior the container via 'journalctl -M <containername>' on the host. While setting to 'host' can give you a single collection of journals in /var/log/journal on the host.

…ties' option. This enables you to run more priviledges containers for example those that must access other /dev items

the-kenny · 2014-06-21T10:19:37Z

I think we should stay conservative with the 'start on boot' option. Adding an option is nice, but a default value which starts all containers on boot might be dangerous.

chrisfarms · 2014-06-22T13:06:06Z

I think intuitively declarative containers should default to auto-starting. I was certainly surprised when they didn't, Everything else that you define (like services) start after you enable them. If you don't want to be running persistent containers then I would expect most people to have reached for imperative containers not defining them in configuration.nix.

Maybe a better solution would be to add an enable option to each container so that they are more inline with service definitions. This might be nice to have anyway to make it easy to do things like enable/disable containers easily (say) by hostname and if the default was enable=false then anyone updating to this new style won't be bombarded with lots of unexpected running containers (but would have to go edit their config to enable their containers and set them up correctly). What do you think?

wmertens · 2014-06-22T17:14:54Z

👍 on what @chrisfarms says

When a container is enabled a seperate service unit is created to manage auto-starting of the container via the `wantedBy` option. When disabled the container is still built but will not be started by systemd, it can then be started/stopped manually via `systemctl` or `nixos-container`. Containers were not getting cleanly shutdown by systemd as machinectl did not wait for container to shutdown and the resulting RTMIN+4 signal would upset the service and trigger systemd to start killing off the processes (I believe this needs some work from upstream tbh). Now the container is shutdown via the command socket, and machinectl is used to wait resulting in a clean shutdown/exit.

alexanderkjeldaas · 2014-06-29T18:44:37Z

+1 to auto-start by default.

thoughtpolice · 2014-08-29T19:00:39Z

@edolstra What's the status on this? I'm strongly in favor of having this functionality, and this PR is going nowhere. I was about to implement it myself, but I remembered this. Can we please merge it? Containers are now optionally autostart with the changes @chrisfarms made. Unless this was obsoleted by something, I'm inclined to merge it soon.

lucabrunox · 2014-08-29T19:40:41Z

Anybody to explain what are the consequences of this PR? What are the usages?

Fuuzetsu · 2014-09-21T02:25:11Z

Another ping, I think unless there are objections raised then you can just merge it in @thoughtpolice , it's pointless to have this sit around for nothing with no discussion and it's not like 3 weeks is not a fair notice.

chrisfarms · 2014-09-21T10:22:58Z

2337a85 appears to have added auto start capability but it's not clear how to set the AUTO_START flag for declarative containers.

I'm happy to update this patch to work with the AUTO_START flag if there is still interest in the rest of these changes?

aristidb · 2014-11-09T10:45:56Z

What's the status of this?

fpletz · 2014-11-09T13:29:44Z

@chrisfarms 👍 on getting this updated and merged, especially the macvlan and privateNetwork changes.

fpletz · 2016-03-20T22:10:50Z

This will conflict with the changes in #14018.

@kampfschlaefer Do you maybe have time to integrate some of the changes here? Otherwise I'll try fo find some time in the next few weeks.

kampfschlaefer · 2016-03-21T16:37:20Z

@fpletz Thanks for showing me this PR. I might take a look at it but I would like to get my PR with its less changes into master sooner than later if possible.

kampfschlaefer · 2016-03-29T17:23:12Z

@fpletz

Do you maybe have time to integrate some of the changes here? Otherwise I'll try fo find some time in the next few weeks.

I think I like the explicit service definitions per container. I noticed that with hostbridge and/or extra network interfaces, there should be a dependency from containers to start only after all network devices or (better) after the network devices used by the container. And this is better solved when there is one explicit service per declarative container.

Also this would allow to define extra dependencies and extra before/after scripts per container… Gotta think about this.

Without the templating (which is still present for imperative containers), it will be possible to set individual dependencies. Like depending on the network only if the hostbridge or hardware interfaces are used. Ported from NixOS#3021

bjornfor · 2016-05-21T14:59:45Z

Ping all.

joachifm · 2016-08-20T10:41:50Z

Ping. Can we resolve this before release?

danbst · 2016-10-04T16:48:28Z

to be clear:

Use unique service units for each container rather than using templates (containers@xxx) to work around reloading issues.

this is already in master (#14018)

Fix triggering of switching to new container configuration when the container's config has changed.

still to be addressed (as explained in #3021 (comment))

Add new options: macvlanInterface, macvlanAddress and macvlanPrefixLength to enable a macvlan nterface within the container (eth1).

Do not force a veth pair to be created when privateNetwork is
true (Sometimes you just want to isolate the container).

veth pairs are now created only when localAddress or hostAddress
are set.

still to be addressed

Add a wantedBy option so that you can control when the container
unit should be started (defaults to multi-user.target).

I think it is possible to set wantedBy now through lib.mkOverride, but haven't checked that.

allow setting of additional container capabilities via 'grantCapabilities' option. This enables you to run more priviledges containers for example those that must access other /dev items

this is done in 0d1e1b1 as additionalCapabilities

spacekitteh · 2016-11-04T15:46:59Z

@grahamc please tag as security

danbst · 2017-08-22T12:28:15Z

Fix triggering of switching to new container configuration when the container's config has changed.

#28465

Add new options: macvlanInterface, macvlanAddress and macvlanPrefixLength to enable a macvlan nterface within the container (eth1).

#20935, partially

disassembler · 2017-11-01T00:07:04Z

Is there anything in this PR that still would be good to implement or should this be closed?

danbst · 2017-11-01T07:56:01Z

@disassembler most stuff is implemented

Do not force a veth pair to be created when privateNetwork is
true (Sometimes you just want to isolate the container).

veth pairs are now created only when localAddress or hostAddress
are set.

Add a wantedBy option so that you can control when the container
unit should be started (defaults to multi-user.target).

still to be addressed

disassembler · 2017-11-01T07:58:30Z

ok, this sounds good to me. Can you rebase on master and fix any conflicts?

danbst · 2017-11-01T08:01:12Z

probably autoStart by default is also what this PR makes

ok, this sounds good to me. Can you rebase on master and fix any conflicts?

ouch, I'm not PR author. Also, it may be not easy to perform rebase (3 year-old PR) and I'm not very interested in those featues

srghma · 2018-04-22T17:32:18Z

Such a shame that this wasn't merged

Previously, setting "privateNetwork = true" without specifying host and local addresses would create unconfigured interfaces: ve-$INSTANCE on the host and eth0 inside the container. These changes is rebased part of the original PR NixOS#3021.

arianvp · 2019-01-03T14:22:50Z

I think most of the stuff that this PR introduces are now available already. Can we close this?

Use unique service units for each container rather than using templates (containers@xxx) to work around reloading issues.

This we don't have yet, but interferes with imperative containers, and I'm not sure what the "Reloading" issues were. Perhaps we can create a new issue for this

Fix triggering of switching to new container configuration when
the container's config has changed.

#28465

Add new options: macvlanInterface, macvlanAddress and
macvlanPrefixLength to enable a macvlan interface within the
container (eth1).

#20935

Do not force a veth pair to be created when privateNetwork is
true (Sometimes you just want to isolate the container).

veth pairs are now created only when localAddress or hostAddress
are set.

These have both been fixed in: #49392 but still has some stuff to fix in #52417

Add a wantedBy option so that you can control when the container
unit should be started (defaults to multi-user.target)

This seems to have been fixed in #48771

danbst · 2019-01-03T14:29:36Z

This we don't have yet, but interferes with imperative containers, and I'm not sure what the "Reloading" issues were. Perhaps we can create a new issue for this

It is already done and merged in separate PRs. @Profpatsch can you close this?

nixos-discourse · 2019-01-14T18:18:11Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/triage-stale-issues-and-prs-automatically/1809/6

chrisfarms added 4 commits June 19, 2014 22:17

retab

62f50e8

fix comment

1d9a6de

fix issue caused by retab

c0c3037

Use unit templates for containers but only consider a unit as a 'temp…

6e36619

…late instance' during switch if it does not have a unit file.

chrisfarms added 3 commits June 20, 2014 13:25

fix machinectl typo

a5a4946

allow setting of additional container capabilities via 'grantCapabili…

7f8d0a2

…ties' option. This enables you to run more priviledges containers for example those that must access other /dev items

Fuuzetsu added the 6.topic: nixos label Sep 21, 2014

vcunat force-pushed the master branch from c86eaf7 to 71c34a4 Compare October 20, 2014 08:19

peti force-pushed the master branch from d0fb763 to 1b68c7b Compare October 31, 2014 12:02

aristidb added 9.needs: reporter feedback and removed 9.needs: reporter feedback labels Nov 9, 2014

thoughtpolice added the 0.kind: enhancement label Mar 11, 2015

kampfschlaefer mentioned this pull request Mar 29, 2016

containers: hostbridge and IPv6 #14018

Merged

fpletz added the 9.needs: clean-up label Apr 24, 2016

This was referenced Sep 6, 2016

Unable to update containers from deployed machine NixOS/nixops#503

Closed

unable to upgrade declarative nixos container. #12396

Closed

Ekleog mentioned this pull request May 8, 2017

[RFC 0012] Declarative virtual machines NixOS/rfcs#12

Closed

uvNikita mentioned this pull request Oct 29, 2018

nixos/containers: don't create veths if not configured #49392

Merged

9 tasks

veprbl added the 2.status: merge conflict label Nov 28, 2018

joachifm closed this Jan 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements for declarative nixos containers #3021

Improvements for declarative nixos containers #3021

chrisfarms commented Jun 19, 2014

thoughtpolice commented Jun 19, 2014

edolstra commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

the-kenny commented Jun 21, 2014

chrisfarms commented Jun 22, 2014

wmertens commented Jun 22, 2014

alexanderkjeldaas commented Jun 29, 2014

thoughtpolice commented Aug 29, 2014

lucabrunox commented Aug 29, 2014

Fuuzetsu commented Sep 21, 2014

chrisfarms commented Sep 21, 2014

aristidb commented Nov 9, 2014

fpletz commented Nov 9, 2014

fpletz commented Mar 20, 2016

kampfschlaefer commented Mar 21, 2016

kampfschlaefer commented Mar 29, 2016

bjornfor commented May 21, 2016

joachifm commented Aug 20, 2016

danbst commented Oct 4, 2016 •

edited

Loading

spacekitteh commented Nov 4, 2016

danbst commented Aug 22, 2017

disassembler commented Nov 1, 2017

danbst commented Nov 1, 2017

disassembler commented Nov 1, 2017

danbst commented Nov 1, 2017 •

edited

Loading

srghma commented Apr 22, 2018

arianvp commented Jan 3, 2019 •

edited

Loading

danbst commented Jan 3, 2019

nixos-discourse commented Jan 14, 2019

Improvements for declarative nixos containers #3021

Improvements for declarative nixos containers #3021

Conversation

chrisfarms commented Jun 19, 2014

thoughtpolice commented Jun 19, 2014

edolstra commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

chrisfarms commented Jun 19, 2014

the-kenny commented Jun 21, 2014

chrisfarms commented Jun 22, 2014

wmertens commented Jun 22, 2014

alexanderkjeldaas commented Jun 29, 2014

thoughtpolice commented Aug 29, 2014

lucabrunox commented Aug 29, 2014

Fuuzetsu commented Sep 21, 2014

chrisfarms commented Sep 21, 2014

aristidb commented Nov 9, 2014

fpletz commented Nov 9, 2014

fpletz commented Mar 20, 2016

kampfschlaefer commented Mar 21, 2016

kampfschlaefer commented Mar 29, 2016

bjornfor commented May 21, 2016

joachifm commented Aug 20, 2016

danbst commented Oct 4, 2016 • edited Loading

spacekitteh commented Nov 4, 2016

danbst commented Aug 22, 2017

disassembler commented Nov 1, 2017

danbst commented Nov 1, 2017

disassembler commented Nov 1, 2017

danbst commented Nov 1, 2017 • edited Loading

srghma commented Apr 22, 2018

arianvp commented Jan 3, 2019 • edited Loading

danbst commented Jan 3, 2019

nixos-discourse commented Jan 14, 2019

danbst commented Oct 4, 2016 •

edited

Loading

danbst commented Nov 1, 2017 •

edited

Loading

arianvp commented Jan 3, 2019 •

edited

Loading