-
Notifications
You must be signed in to change notification settings - Fork 956
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
/etc/docker/daemon.json is read-only in HA OS 9.0 #2135
Comments
This change has been on purpose, see #2116. The reason this folder was writeable is for Docker to store the What is your use case/what do you intend to change in
It is completely unsuited for debugging, IMHO 😅 . But that is the nature of a largely stateless, read-only system. 🤷♂️
So you use this altered config not just for debugging then? I am really curious what kind of changes you need. Ideally we make them "unnecessary" or configurable through a flag.
I have a HAOS build directory which I keep rather current. A rebuild is usually quite quick, so for me altering a system service for debugging purpose isn't that big of a deal. But I understand, that is a bit a hassle. Using a regular Debian & Supervised is probably the way to go if you need custom system service configurations which we can't ship by default. |
I add my own configuration options. It wouldn't make sense to force everyone to use these settings. It's not obvious how this could be handled after the migration to this setup. The change itself makes sense. I also understand the limitations around Docker's daemon.json configuration and the absence of config.d directories with multiple config files. Supplying multiple daemon.json config files doesn't seem to be possible. The one option would be to have a pre-start script which grabs daemon.json from two different locations and merges them into another one at /etc/docker/daemon.json. The options set in the second one would override the ones from the main daemon.json provided by HA OS. Of course, if the daemon doesn't start because of invalid configuration, that wouldn't be HA OS' problem.
The second Ethernet adapter froze on my live HA OS setup. I had to connect locally to the machine to figure out what's going on. It's great that there's not much to manage for HA OS and there's one less system to look after. Not being able to connect to it to debug the same way as with a regular distribution is a limitation.
Indeed, debugging is needed for my main HA instance. I've brought this up in the context of the daemon.json since it makes HA OS less configurable.
I've built my own testing image to learn how these images are built. The issue is with keeping up with HA OS releases. Perhaps it might be a good idea to build something similar to HA OS with SSH access and a few other changes. Maybe that would bring back some decent improvements to HA OS. |
Can confirm this is a super inconvenient change. My ISP router is on 172.17.x.x with no access to change, so moving the docker network to something else is required. I'm still unsure why docker needs a /16 (65k addresses per network), so using these default settings for mostly home use is a waste and inconvenient. Adding settings like below should be more than enough for most users, but most importantly, being able to update the address pool to something the user prefers, would be ideal.
I'm now at an impasse and won't be able to do connectivity checks to make sure my internet is up |
Seconding this, the ability to change the default docker network would be extremely helpful. |
We can use this command as a pre-start script for the systemd unit. I'll give it a shot. The plan is to try this:
Another thing to keep in mind is that we can check if we already have an up to date /etc/docker/daemon.json with the same content. That makes sure we don't write to the disk when that's not necessary. |
@agners: This should work. You'll probably want to modify this. The initial build of the images takes some time. |
@agners: Did you take a look at the posted patch? Is there something you want to have changed in that patch? |
There is SSH access, see SSH access to the host. Back to the topic:
HAOS has lots of hard coded configuration and decisions, and that is the intention: Limit the amount of variability, so it is manageable to get reliable updates etc. If maximum flexibility is our intention, we could just offer a pre-built Debian image with Supervisor pre-installed... There is always a trade-off. With OS 9.0 we decided to standardize on
I like the merging aspect of the implementation. What isn't super nice is that we need to copy the file on every boot (this has the potential of infamous 0 byte files when power cutting at the wrong point). That said, I like to understand the use cases we are trying to solve here, and rather prefer to bring "native" support for this use cases. @HangrilyJon @tristansgray bring up a good point: The hard coded network configuration. Now we have actually two networks: The To run Home Assistant Supervisor, and any of the plug-ins and add-ons the Is the hassio network in a problematic range for you? Maybe this should be configurable. |
In my case, it's the docker0 range. However, allowing modification of the hassio network would be useful as well. |
Thanks. I'll try that again. It didn't work for me the last time I tried it.
This is most likely valid for lots of things, including writing YAML by hand for Home Assistant. It can be broken in many ways. In my case, I don't expect the Home Assistant developers and the community to provide support for different settings. It's just that it doesn't make sense to have hardcoded and read-only settings because someone will break their system. It can also be perceived as saying that users need to not be allowed to make changes because they'll break something. This is a Linux system. People do expect to be able to make changes within reason.
The file isn't copied on every boot. This is the script:
If there's no local config, the default is copied. Otherwise a new daemon config is generated and stored in /tmp. Only the temporary file is generated on every startup. Does that address your concern? Is the fact that we copy the temporary file to /tmp a concern? I've attached the modified version of the patch which replaces diff with cmp. diff isn't present in the target image. I'd like to make an improvement to make the bind mount read-only in /etc/docker and operate directly on the files in /mnt/overlay/etc/docker.
Regarding the use case, Home Assistant Operating System is still a Linux system. Is it unreasonable to expect it to allow some configuration changes? I expect to be able to customize the running Docker daemon's configuration as I can on every other distribution. HA OS isn't the right solution for Home Assistant if the answer is no. That'd be a shame as it could save some time with maintaining yet another system.
|
Why does that not make sense? It does a lot to me, we use a read-only root file system to avoid any corruption, intentional or unintentional (mis-)configuration, and generally make the system more predictable and with that solid. We push that as much as possible, without taking away necessary features.
We remove a lot of features of a traditional distribution to make it largely stateless, single purpose operating system with a fixed set of packets "pre-installed", distributed in a monolithic image. Compared to a general purpose Linux operating system large parts are missing, e.g. no package manager. Is it "Linux"? I guess that depends on the definition 🤷♂️ is Andorid Linux? 🤷♂️ IMHO, it's a purpose built OS using Linux kernel and other OSS packages as its building blocks. Arch Linux is a proper Linux 😅 In the end, the question is, what is the exact definition of "without taking away necessary features". It seems that in your opinion we overstepping that line with making
No, that is not unreasonable. However, for HAOS, we try to limit the amount of configuration options to make the system manageable for us. For the set of core features we want to provide forward compatibility (e.g. network and host name configuration). We try to make those settings as much as possible accessible through the frontend as well. The limited feature set also allows us to advance the development of the system. E.g. there is the idea to move away from Docker entirely for HAOS and using containerd (or even runc) only. Docker "hides" some nice features available in containerd (like dynamic device permission updates for containers for hot-plugging). And it would make the distribution lean. These are just ideas at this point, and not sure if that will ever materialize. But "supporting" custom And that is also why I'd like to know what features are used in |
Private registries and registry mirrors are two such settings. I don't think they break your assumptions regarding the supported configuration. I keep it if I break it. The HA OS images can even be patched by hand if you prefer not to support this at all. |
Private registries are possible, in the add-on store page (top right corner): There is no support for registry mirrors afaik. That said, we are using ghcr.io mostly these days, afaik, the |
@agners: I don't see a way to open a simple ticket which isn't a bug report. The following patch would be helpful for HA OS:
You can try it out since you've mentioned you build HA OS often. It'd help to have it in HA OS. HA OS uses an unsupported version of Go from its fork of buildroot: https://github.com/home-assistant/buildroot/blob/2022.02.x-haos/package/go/go.mk#L7 The currently supported versions of Go are 1.18 and 1.19. I've put together a patch script to patch Docker's daemon.json config file:
Warning: use this only if you understand what needs to be modified to make this work. An unbootable HA OS and data loss are guaranteed. |
Feature requests should be submitted through the HA forum or Discussions here on GitHub.
We inherit this from the current LTS version of Buildroot, see https://github.com/buildroot/buildroot/blob/2022.02.x/package/go/go.mk#L7. Probably best is to complain in the Buildroot ML about this 😅 I don't feel its worth our time to backport a newer Go version to Buildroot 2022.02, hence Go version in 9.x release series will remain what it is today. I intend to move to the master branch of Buildroot soon, ultimately the intent is to use Buildroot 2023.02 LTS in OS 10. |
Any hope for a workaround, a dirty hack is also ok, on this issue? We are planning several instances of Homeassistant, and use the 172.17.0.0/16-range internally, so without the abiltiy to change the docker IP range, Homeassistant is basically useless 😳 |
Going back to modifiable daemon.json is the wrong solution, and also kinda pain at this point. What we discussed is to limit the scope of the default network to |
We use the entire range 172.17.0.0/16 over meny location, .234 is also in use. But it would be less catastrophic, only one location that would be unable to access HA-instances. Isn’t it possible just to have some config/override parameter for the docker bridge? |
It is not only the Docker bridge, it's also the custom network which uses a Ideally we make both configurable, but that is a bit more involved. |
*sigh* this poses a huge obstacle for our deployment.
We really hope you can come up with a solution to make networks configurable. The first net where we sat up a lab was actually using 172.17.0.0/24 and of course that was completly hopless, so I haven’t been abel to really look into our options yet. Deploying the pre-packaged virtual machine seemed to be perfect, since we will deploy many instances in diffrent locations
We have been using 172.17.0.0/16-ip range for almost 20-years, so we have had to change quite a few docker-bridges the last couple of years, since docker became a thing. This is the first time we meet an unconfigurable one.
|
I have skimmed quickly through this guide: https://community.home-assistant.io/t/installing-home-assistant-supervised-on-debian-11/200253 Will we get something that is functional equivalent to the virtual-machine if set up HA this way? I asume there won’t be any read-only restrictions or enforcement of 172.17-prefixes there? It would be very cumbersome to deploy and maintain all instances this way, but to get done some initial testing the next couple of months it would probably be doable. |
We have now installed and deployed several debian VMs with in the "Supervised" setup mentioned in the documentation, using the guide above, so we are helped for now. They can be both scripted and cloned, but pre-made Hassio-OS VMs would be a preffered choice. So I still cannot emphasis enough, how important we consider a solution for this issue. |
My use case for this is to be able to configure the shm-size for Frigate, and since it can't be configured per/for add-ons, I'd need to configure a global shm size for the whole system. Refs blakeblackshear/frigate#5123 But well, it would be perfect if it could be configured per add-on. :) |
Actually, never mind! It seems that by default all add-ons have access to the whole memory under |
HAOS 9.5 uses two smaller subnets in |
In China, GHCR.IO cannot be accessed, so I cannot update through the update button normally. I need to configure the docker registry mirror to normalize the Docker Pull command to download the latest image. |
There hasn't been any activity on this issue recently. To keep our backlog manageable we have to clean old issues, as many of them have already been resolved with the latest updates. |
Describe the issue you are experiencing
/etc/docker/daemon.json used to be writable before version 9.0. This facilitated the customisation of the Docker daemon startup flags.
This is no longer possible with HA OS 9.0. Please make this possible again. HA OS is already somewhat hostile when it comes to debugging.
The alternative is to fork HA OS and keep building new images. It doesn't really make sense to maintain a fork for one or two things which need to be modified. I could just migrate to the HA core container instead.
What operating system image do you use?
generic-x86-64 (Generic UEFI capable x86-64 systems)
What version of Home Assistant Operating System is installed?
9.0
Did you upgrade the Operating System.
Yes
Steps to reproduce the issue
...
Anything in the Supervisor logs that might be useful for us?
Anything in the Host logs that might be useful for us?
System Health information
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: