Added YAML merging Python script, with menu script modifications. Add…

[Slyke]

Added YAML merging Python script, with menu script modifications. Added minor changes and bug fixes to backup.

How this feature works:
If no compose-override.yml file is present in the main directory, the docker-compose.yml file is unmodified from the yaml builder.

If the compose-override.yml file is present in the main directory, it will merge in and override any properties that are set.

See screenshot for example:

• On the left is the override yaml file, stating to replace mosquitto's ports.
• The "tmp" (middle) docker compose file is the one originally generated by the menu.sh bash script (minus the custom services and volumes part - which has been removed due to being redundant).
• On the right is the final file, with mosquitto's ports replaced.

Do note that lists are replaced.

So for example, if You have the ports:

    ports:
      - 1883:1883
      - 9001:9001

And you set a port in the override:

    ports:
      - 3333:3333

Then the entire port list is replaced with what's specified in the override.

This PR removes the need for the custom services, and custom volume mounting since they can be specified in the override file.

The override file has also been added to the backup functionality, but I have not tested restoring it.

I need to update documentation for this in the wiki.

The override file is added to gitignore, so you don't need to worry about it being replaced or deleted when updating the project.

[Slyke]

Seems like pyyaml is not installed on the lite version of Raspberry Pi. Adding fixes now.

[DIYtechie]

Great idea to have an override that can be automatically backed up together with the rest of the configuration files and volumes. When does the override happen? Every time the docker-compose.yml is compiled?

What happens if some containers are deleted? - or not built when restoring certain containers from a backup? Certain parts of the compose-override would be relevant, but what about the parts relating to the containers that are not built?

Would it make sense to have an override for each container?

[Slyke]

The override is merged in with the compose file when the menu's build process is running. The menu basically generates the docker-compose.yml file in a temp directory, then if the override file exists, it merges it into a new docker-compose.yml file, using the old one as a basis.

I added it to the backup for convenience so that it doesn't need to be copied around manually when restoring (and you rebuild), but it technically doesn't need to exist after the build process is completed, you technically only need the dockler-compose.yml file and volumes to run docker-compose -d up.

Don't need an override for each container, it'd be kind of like asking for a docker-compose.yml file for each container. If you don't want to change any of the configs in for container, then just don't specify it in the override. It will only touch yml paths that are specified.

You can always send me your current docker-compose.yml file and what you want to override (and don't) and I'll write the compose-override.yml file that makes those changes.

[DIYtechie]

Ahh. I see. Guess I was confused by reading up on how docker handles docker-compose.override.yml files which is different from how your compose-override.yml feature works.

[Slyke]

Yup! The override file is just for IOTstack :). You could still use the official override if wanted, but I think for IOTstack's case, using the compose-override.yml is easier.

[Slyke]

This branch has been merged into the Experimental branch for PR: #40