config lost after new build of stack

[KLelong]

Hi,
I'm new to IoTStack.
I installed it on an ubuntu server. Since I want to migrate from "native" applications to docker I'm going step by step.
One thing I discovered is that when I add a container (./menu.sh) and then run docker-compose up -d, I lose configurations. At least from mosquitto, which is my first container.
E.g. I configured mosquitto to use ssl-certificates and port 8883. This starts OK. But when I add e.g. InfluxDB and run docker-compose up -d again, my configuration of mosquitto is reverted to "out of the box" i.e. my modifications are lost.
Is this normal behaviour ? Now I backup my configuration before I run docker-compose up -d.
For some things, I use compose-override.yml, but do I need this for mosquitto.conf ?
TIA

[Paraphraser]

It is not normal behaviour.

It is not really stated anywhere as a hard-and-fast rule but IOTstack implicitly makes some assumptions, such as:

• You are using a Raspberry Pi (typically a 3B+ or 4B).
• You are running Raspberry Pi OS (aka Raspbian).
• You are using the default user account (pi).
• User pi has the ID 1000.
• You have installed at ~/IOTstack with that exact spelling.

This is not to say that IOTstack won't run on other platforms. It's just that it gets a lot more testing where the above assumptions hold so kinks tend to get ironed out.

I'm definitely not trying to say that anything about your platform is the cause of what you are seeing. You haven't really given me a lot to go on so I need to lay out the kinds of things you might need to think about.

Most containers use root ownership for their persistent storage (directories inside ~/IOTstack/volumes). When docker-compose brings up the stack, it looks through the volumes statements in docker-compose.yml and creates any missing external paths it sees as directories with root ownership.

Mosquitto is an exception. It assumes and relies upon:

• ~/IOTstack/volumes/mosquitto and everything inside being owned by user ID 1883 (an account that does not actually exist); and
• the two files it uses in ~/IOTstack/services/mosquitto being owned by "pi".

These presets are the result of running the menu. If something happens to undo the effect of running the menu then docker-compose will wade in the next time you run it. The Mosquitto container's volumes statements are:

    - ./volumes/mosquitto/data:/mosquitto/data
    - ./volumes/mosquitto/log:/mosquitto/log
    - ./volumes/mosquitto/pwfile:/mosquitto/pwfile
    - ./services/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf
    - ./services/mosquitto/filter.acl:/mosquitto/config/filter.acl

Those decompose about the colons into "externalPath:internalPath", and the leading "." on each external path is defined as "the directory containing docker-compose.yml".

Docker-compose assumes that everything is a directory so if, for example, the file at:

~/IOTstack/services/mosquitto/mosquitto.conf

is missing when you run docker-compose, then the result will be a directory at that path, and that directory will have root:root ownership.

I just tried this on my RPi and mosquitto got most upset:

ERROR: for mosquitto  Cannot start service mosquitto: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/home/pi/IOTstack/services/mosquitto/mosquitto.conf" to rootfs at "/var/lib/docker/overlay2/27ab10ba80fdee3bf63d67e9fa2046d6f5edfb18f7c2b70de2dca06b76f5f45c/merged/mosquitto/config/mosquitto.conf" caused: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

That's a world away from your report of the config being reset so, other than where I started (this is not normal behaviour; maybe it's a Ubuntu thing), I don't know what else to suggest.

If nothing here leads you to a solution, perhaps come back with a recursive listing of:

$ ls -alR ~/IOTstack/services/mosquitto ~/IOTstack/volumes/mosquitto

Maybe include your docker-compose.yml and override file too.

[Paraphraser]

I've been thinking about this some more.

Some background. I started using IOTstack late 2019 when it was gcgarner/IOTstack. I really had no idea what was going on but my adaptations as I kicked the tyres, broke things and learned how to fix them resulted in some rules-of-thumb:

• Use the menu to set up the initial set of containers.
• Learn how IOTstack hangs together so you can mimic what the menu would do.
• Stay away from the menu.

Fast forward to today and SensorsIot/IOTstack, and I added another rule-of-thumb:

• Don't so much as think the word "override".

When you first clone the repo, you don't have a services or volumes folder, or a docker-compose.yml. services, docker-compose.yml and some parts of volumes are built either by, or as side-effects of, running the menu, with the remainder of volumes initialised by docker-compose (as explained before).

There are two menu systems. The "old menu" (on the old-menu branch) has it origins in gcgarner/IOTstack. A bash script. Always good at first build but had a fair bit of "your mileage may vary" when trying to use it for ongoing maintenance.

The "new menu" (master branch) is a SensorsIot/IOTstack-only thing written in Python. I've used it for first build (just to see what it does; never in anger to produce a stack I will depend on) and my rules of thumb mean I'm unlikely to venture much further down that path.

You mentioning compose-override.yml makes me think two things. First, "new menu". Second, routinely re-running the menu for maintenance work.

I'm wondering whether re-running the menu to apply the override file is having the effect of re-initialising everything?

The way it works is that the menu starts with the template in:

~/IOTstack/.templates/mosquitto

and copies the elements to the service definition:

~/IOTstack/services/mosquitto

then sets up the folder and permission structure I mentioned before by running one of:

~/IOTstack/.templates/mosquitto/directoryfix.sh
~/IOTstack/services/mosquitto/directoryfix.sh

I'm not sure which one it actually runs but, as they're the same, it makes no difference. I know old-menu copied the former to the latter at setup time but I'm not sure whether new-menu does the same so it might be run from .templates. I'm just covering bases.

After that, all the little service.yml files in all the sub-directories of:

~/IOTstack/services

are concatenated along with a header and, if it's new menu, with ~/IOTstack/.templates/env.yml tacked onto the end, and that's docker-compose.yml.

I'm not sure at which point the override stuff is applied but probably after that.

In the old menu, if you re-chose a service that was already defined (ie existed in ~/IOTstack/services) you got some options. If, for example, you chose "pull full service from template" for mosquitto then whatever was in ~/IOTstack/services/mosquitto would be zapped, and that would definitely include the config file.

I'm wondering whether some side-effect of re-running new menu and/or the override processing is causing the same kind of thing to happen?

Hopefully, I've given you enough detail to ferret through and see what there is to be seen.

You could, for example, add a marker to:

~/IOTstack/.templates/mosquitto/mosquitto.conf

then do whatever it is that you've been doing and see if the marker turns up in:

~/IOTstack/services/mosquitto/mosquitto.conf

If it does, it will prove that something either intrinsic to the menu or the way you're using it is causing a "pull full service from template" effect. If not, at least it will eliminate that possibility.

Getting back to my rules of thumb, I just do everything in docker-compose.yml.

If I add a new service, I copy the template files myself, copy and paste the service.yml into docker-compose.yml.

I see no point to either the docker-compose-supported docker-compose.override.yml processing or the IOTstack-new-menu-supported compose-override.yml process. Why muck around when you can keep everything in a single file and see what's what?

But that's just me. Old dude. Grew up in the punched-card era. I like my CLI. I like in-ya-face. I hate opaque.

But I don't have this kind of trouble. Make of that what you will.

[MostHated]

my configuration of mosquitto is reverted to "out of the box" i.e. my modifications are lost.

After I saw this happen the first time, I decided I wanted to play it extra safe and I made a backup copy of the whole IOTStack folder. Then, added the new thing from the menu I wanted to add then opened up the docker-compose.yml and found the new entry it added, copied that portion and added it to my backup's compose file along with all my other modifications, and then made the adjustments I wanted and save it, then copied all of it back to the original compose file and fired it up.

This way, if for some reason you noticed anything else is goofy or overwritten, you can just grab it from your backup and put it back. If I ever need to add something else, I just refresh my backup copy with the current working IOTStack folder, rinse and repeat.

While it is ideal to have automatic scheduled backups of your system as well, if you don't for some reason, at least you have something to fall back on by doing this.

[Paraphraser]

An experiment

For @Slyke

TL;DR

Re-running the new menu resets mosquitto.conf

Details

Clean clone of master branch

$ git clone https://github.com/SensorsIot/IOTstack.git ~/IOTstack 
Cloning into '/home/pi/IOTstack'...
remote: Enumerating objects: 555, done.
remote: Counting objects: 100% (555/555), done.
remote: Compressing objects: 100% (314/314), done.
remote: Total 4368 (delta 213), reused 477 (delta 165), pack-reused 3813
Receiving objects: 100% (4368/4368), 1.86 MiB | 468.00 KiB/s, done.
Resolving deltas: 100% (2522/2522), done.

Run the menu:

$ ./menu.sh 
Checking for project update
From https://github.com/SensorsIot/IOTstack
 * branch            master     -> FETCH_HEAD
Project is up to date
Python Version: 'Python 3.7.3'. Python is up to date.
ruamel.yaml Version: 'ruamel.yaml 0.16.12'. ruamel.yaml is up to date.
Blessed Version: 'blessed 1.17.12'. Blessed is up to date.
Please enter sudo pasword if prompted
Command: docker version -f "{{.Server.Version}}"
Docker version 20.10.3 >= 18.2.0. Docker is good to go.
Project dependencies up to date

Choose Build Stack, select Mosquitto, and follow through to the end.

Display the last line of the mosquitto configuration file "as installed":

$ tail -1 services/mosquitto/mosquitto.conf
log_timestamp_format %Y-%m-%dT%H:%M:%S

Append a marker to that file:

$ echo "# Marker" >>services/mosquitto/mosquitto.conf

Prove the marker is there:

$ tail -1 services/mosquitto/mosquitto.conf 
# Marker

Bring up the stack:

$ docker-compose up -d
WARNING: Some networks were defined but are not used by any service: vpn_nw, nextcloud_internal, iotstack_nw_internal
Creating network "IOTstack_Net" with driver "bridge"
Creating mosquitto ... done

Show that this has not affected the marker:

$ tail -1 services/mosquitto/mosquitto.conf 
# Marker

Re-run the menu:

$ ./menu.sh 
Checking for project update
From https://github.com/SensorsIot/IOTstack
 * branch            master     -> FETCH_HEAD
Project is up to date
Python Version: 'Python 3.7.3'. Python is up to date.
ruamel.yaml Version: 'ruamel.yaml 0.16.12'. ruamel.yaml is up to date.
Blessed Version: 'blessed 1.17.12'. Blessed is up to date.
Please enter sudo pasword if prompted
Command: docker version -f "{{.Server.Version}}"
Docker version 20.10.3 >= 18.2.0. Docker is good to go.
Project dependencies up to date

Existing installation detected.

Choose Build Stack, select Grafana, and follow through to the end.

What's the story with the marker?

$ tail -1 services/mosquitto/mosquitto.conf
log_timestamp_format %Y-%m-%dT%H:%M:%S

Gone! Adding a new service re-copies mosquitto.conf from the template.

This should not happen!

[Paraphraser]

Probably a combination of pilot error and old Mosquitto behaviour before the self repair mods. Stale.