Request for official Docker image

[ghost]

I'm currently running all of my web-facing applications behind a reverse proxy in separate Docker containers; it would be awesome if you could release an official Docker image for MeshCentral on the Docker Hub so that people like me can seamlessly integrate the software into their existing setup.

[Ylianst]

Your right, this is high on my priority list. I am a few trips coming up but as soon as I get a bit more familiar with Docker, I will be posting one.

[TheTechsTech]

I had created an Docker image base on my forked some time ago. https://hub.docker.com/r/technoexpress/meshcentral/

Here is another Docker image MeshCentral2. I updated to be just based on this project only. It will setup things automatically on docker run.

docker run --name meshcentral \
-p 25:25/tcp -p 80:80/tcp -p 443:443/tcp -p 4443:4443/tcp -p 27017-27018/tcp \
-v meshcentral-data:/home/meshserver/meshcentral-data \
-v meshcentral-files:/home/meshserver/meshcentral-files \
-e PORT=443 -e REDIRPORT=80 -e MPSPORT=4443 -e EMAIL=mail@host \
-e HOST=host.ltd -e SMTP=smtp.host.ltd \
-e USER=smtp@user -e PASS=smtppass! \
-e DB=netdb -e MONGODB="mongodb://127.0.0.1:27017/meshcentral" -e MONGODBCOL="meshcentral" \
--restart=always --hostname=mesh.host.name -d technoexpress/meshcentral2

(https://cloud.docker.com/repository/docker/technoexpress/meshcentral2)

[ghost]

I have also created a Docker image for MeshCentral which is quite a bit smaller and simpler than the one techno-express posted. The Docker image and be found here and the source repository here.

[TheTechsTech]

I don't know how you define simpler after reading your full description. How you setup monogb, and email, from your compose file? As far setting up proxy, there are docker images for that, which i use it has letsencrypt feature already, the image pointing to it, if it detect the volume mount, not using the in meshcentral, there is no need creating any network proxy.

The image is bigger cause mongodb is added, and will be used only when environment variable DB is changed. And to be able to move data around easier, sharing, backups, you better off with named volumes.

[ghost]

@techno-express Don't get me wrong, I wasn't criticizing your image or trying to say that it's worse. In fact, it currently has more features than mine like changing the config file by using environment variables. I was just thinking that since @Ylianst is not familiar with Docker right now, my simple Dockerfile would offer him a decent starting point once he starts working on making an official MeshCentral image.

That being said, the reason why your image is bigger than mine is not because of MongoDB but because you are using the standard version of node rather than the alpine version. If you're interested in reducing the size of your image, you should be able to do so by switching from node:stretch to node:alpine and by replacing RUN apt-get update && apt-get -y install mongodb sudo with RUN apk update && apk add --no-cache mongodb sudo in your Dockerfile.

[TheTechsTech]

ok, giving @Ylianst that is understandable.

You also reminder me of something i glossed over. I intentionally choose node:stretch, i looked at alpine and seen how/why it's smaller. Stuff has been removed to the point where user will need to use different commands. Which is something, i currently don't use normally. I personalty don't use apline in anything for that reason, not interested in the size, more on versatile image.

[ghost]

@Ylianst In light of recent improvements to MeshCentral, I've rewritten the description of my Docker image from scratch and included a list of easy-to-follow steps for people who are looking to get started with MeshCentral. Thoughts?

[Ylianst]

That looks really nice! I really should be working on getting an official Docker image ready soon. I feel I am swimming in more critical issues right now. One question I had is how to best configure a MeshCentral instance in a container. One option would be for me to build a web based "MeshCentral Configuration Editor" that would be listening on a odd port. You run the container, hit the configuration editor port using a browser and setup everything else. If not this, what is the state of the art for the easiest way to configure a container? - Any suggestion appreciated for how to best do this.

[ghost]

@Ylianst Thanks for your feedback. The web based configuration editor sounds like a good idea to me. Another option, one that is commonly used to configure containers, is to define the configuration via environment variables. Naturally, the application has to be able to fetch its configuration from environment variables, so you'd first have to adapt MeshCentral accordingly. I think they're both sound options, though I think that a web based editor would be more convenient for newcomers to Docker. It's ultimately up to you.

[ghost]

@Ylianst I have made another major update to my Docker setup just now. The Docker image is now being automatically built from the MeshCentral NPM repository and the updated image then subsequently pushed to the Docker Hub every 24 hours. This way, it always stays up-to-date, even when I'm away on vacation or business trips.

I have also expanded the documentation to include information about the only (known) issue with it, which is that data you upload to MeshCentral server through the website doesn't persist across Docker container restarts. The way you would normally solve this issue is by mapping the internal container folder to a folder on the server, but when I do that, files can't be uploaded to the MeshCentral server at all. It's quite strange because I've never had this problem when Dockerizing any other applications.

I'm not yet sure what the cause of the issue is, but since file sharing is only a minor functionality, I consider the Docker image perfectly usable in its current state. I hope that it will serve you as a helpful and useful starting point once you start working on an official Docker image.

[Ylianst]

Thanks for doing this. Getting to know Docker more has been on my list for months now, but I keep being sucked into more features & bug fixes. Lately I have been doing more code scans and making sure the security of the software is as good as possible. I will certainly move on to Docker at some point, hopefully soon. I really appreciate your work on this. A few questions:

Do you have a server of your own that builds the latest docket images? Or is the auto-rebuild something that is part of DocketHub? Do you have software/script that interacts with Docker hub to auto-build the new image? Also, why not "npm install" at the launch of the container? Is that not proper for the container usage? Would it not be proper for a container to self-update?

I certainly want to take a look at a better way to setup the server than with environment variables. If there is a typical container image that exists that is considered the state of the art for something with the same needs/config as MeshCentral, I would really like to know what it would be and to give it a try myself Not for the usage, but for how the configuration works, database, files, etc.

Thanks - Ylian

[ghost]

@Ylianst

Do you have a server of your own that builds the latest docket images?

Yes.

Or is the auto-rebuild something that is part of DocketHub?

That functionality is also part of the DockerHub, but I don't use it since I don't own the repository that contains the source code. In order to make it work, you basically put a Dockerfile in the root of your repository which tells Docker how to build the image and point the DockerHub at your repository. The DockerHub then monitors your repository and automatically builds a new image each time a new commit is made so that it is always up-to-date. This means that is does not build the images on a fixed schedule like I do. Naturally, you can specify different tags and build jobs so that you can have separate images for stable and development versions of your software.

Do you have software/script that interacts with Docker hub to auto-build the new image?

Yes, I use the following shell script to automatically build and push MeshCentral images to the DockerHub.

You don't need any software to run it apart from Docker itself.

docker login -u graywhale -p "<password>" && \
docker build --compress -t graywhale/meshcentral /root/docker-compose/build/meshcentral && \
docker push graywhale/meshcentral

Also, why not "npm install" at the launch of the container? Is that not proper for the container usage? Would it not be proper for a container to self-update?

Huh, I hadn't considered that possibility. Interesting. I will give this a try to see if it works as expected.

I certainly want to take a look at a better way to setup the server than with environment variables. If there is a typical container image that exists that is considered the state of the art for something with the same needs/config as MeshCentral, I would really like to know what it would be and to give it a try myself Not for the usage, but for how the configuration works, database, files, etc.

I'm unable to give you an example of the top off my head, but I'm sure there are similar projects out there which have been dockerized.

P.S.: I increased the build frequency from once every 24 hours to once every hour since MeshCentral is often updated several times during the day.

[ghost]

@Ylianst Heureka! I rewrote my Dockerfile from scratch based on your idea and I can happily report that it works flawlessly

FROM node:alpine

LABEL maintainer="whalehub/graywhale"

VOLUME /meshcentral/meshcentral-data
VOLUME /meshcentral/meshcentral-files

EXPOSE 80 443 4433

WORKDIR /meshcentral
CMD ["sh", "-c", "npm install meshcentral ; cd ./node_modules ; node meshcentral"]

The resulting Docker image never has to be updated because the container updates MeshCentral to the latest version at runtime.

[ghost]

@Ylianst I can now confirm that both manual and automatic updates work flawlessly with the new docker image.

[luckydonald]

Also, why not "npm install" at the launch of the container? Is that not proper for the container usage?

It has a some drawbacks:

• increases startup time,
• it can't start if npm is down,
• starting two containers can end you up two different versions.
• I can't prevent it from upgrading

Having a version 'baked in' the container feels like the better way.

If that DockerHub approach doesn't work, maybe the docker build & push can integrated into the normal npm deployment cycle?

Googling lead me to https://github.com/nvellon/meshcentral-docker/blob/master/Dockerfile, which looks quite nice.
Starting with the official docker image and installing the project into it. Config files could be either added with --volume settings or by extending the file

FROM ylianst/MeshCentra:latest

COPY ./config.json:node-module/meshcentral-data/config.json

[ghost]

@luckydonald Those are some good observations. I will have to give this some thought.

[Ylianst]

I have been doing some thinking on the MeshCentral Docker question in light of this thread and what I have been reading. Note that I am no Docker expert at all, so I need input on this.

First, for the MeshCentral version in the container problem. I am thinking that I will add a feature so that instead of saying "SelfUpdate: true" in the config.json, you can say "SelfUpdate: "@0.2.5-a"" or something like this. The server will seek to update to a specific version and do nothing if it's at that version. That is, you can build a container with one version, but have the container try to self-update to a different specified version if needed. This way, you can launch a set of containers with identical versions.

Second. I have been seeing that the model for containers is that they should be, in fact, stateless. That is, you should be able to just launch a container with database connection information and have the server pull it's config, certs and everything else from the DB. Even the server side user files could/should be stored in the DB. The container's file system should really contains nothing of any importance.

In this stateless model, built-in Let's Encrypt support in MeshCentral would be disabled, but that is probably correct since a reverse-proxy in front of the container would likely do TLS offload + certificate handling. On start, the config would come from the database and you would, for example, set the "SelfUpdate" to a specific version and have the container update itself. This also has the benefit that everything in the database could be "encrypted at rest" for regulation compliance (MongoDB enterprise or something). You could quickly launch a load-balanced MeshCentral setup this way.

Feedback appreciated...

[ghost]

@Ylianst You are correct on all accounts. Docker containers are stateless by nature, so all configuration files and databases should be stored outside of the container itself. Only the executable of the application itself should be in the container. This is what allows us to have Docker install Meshcentral's latest version at runtime, because the moment the container is stopped, the previous version of Meshcentral is discarded.

[Ylianst]

I think it makes sense. Now I need to offer a easy way to load/save files in the DB and edit the config in the db. Also, is it common to save certificates in a db? I probably need to encrypt all these things in the db and have the docker image launch with DB connection information and decryption key...

[ghost]

@Ylianst To be honest, this is the first time I've heard of someone saving certificates in a db. I'm sure it's possible though.

[jsastriawan]

Chiming to this discussion. Definitely there are many ways of doing persistent storage of configuration from simple volume to db. There is also alternative like etcd. https://github.com/etcd-io/etcd

[TheTechsTech]

How is configuration setup currently being done?
That should be tookin care of first, beyond the current implementation, which is configuration file.

Docker is nothing special besides having an application/service self contained. Wherever the the config is stored or setup, that location can be mounted outside of Docker build image, and is available before Docker is even started.

I'm using environment variables and json configure file, because that's how you currently have it working.

If things was changed to be stored in an DB, that database location would be name volume mount outside the actual image for persistent storage. There are many different Docker storage plugins available to just to manage persistent storage concerns if need be.

You my want to play around on setting up some other favorite application you use in Docker, to get the hang of things. I think you trying to setup something special for some kinda Docker environment that doesn't really exits.

[Ylianst]

So, I don't plan on changing the existing setup. The "meshcentral-data" folder and config.json file works for most people and is very easy, I will not be changing that. I am exploring an optional way to load the config from the database. So, you would still have a very small config.json that indicated where the database is and that the config should be loaded from the db... and the rest would all be in the database. Something like:

{ "settings": { "MongoDb": "mongodb://127.0.0.1:27017/meshcentral", "LoadConfigFromDb": true } }

Since the "settings" portion of the config.json is actually the same as placing options in the commands line. You could have no config.json at all and just run:

node node_modules/meshcentral --MongoDb "mongodb://127.0.0.1:27017/meshcentral" --LoadConfigFromDb

It would only be used by advanced users that need all data to be "encrypted at rest" in the database, or want the container to be completely stateless. Would not change anything for existing users. Good or bad feedback welcome on this.

[ghost]

@Ylianst I love that idea. It would be great to have the database encrypted at rest for compliance reasons.

[luckydonald]

For readability i would put LoadConfigFromDb first, before that mongodb one.
Also encryption should be a secondary step.

[TheTechsTech]

Yes, including something like --LoadConfigFromDb would be better implementation and fix some security concerns too.

[luckydonald]

Is the config file getting written to? I was under the impression it is read only?

[Ylianst]

The config.json file is read only. There are situations where administrators would like to have this and other files read from the database instead of the "meshcentral-data" folder. This way, everything is encrypted in the database and you don't need to configure each server individually.

[ghost]

@Ylianst I have some ideas about how to enhance my Docker container even further. I'm going to have MongoDB installed in it and used in the config.json file by default, for one. I also want to make it possible to conveniently store the config file and certs in the MongoDB database in encrypted form within the container.

Furthermore, I want to switch from the current "build at runtime" system to using automated builds so that every time you push a new commit (i.e. release a new version of MeshCentral), a new Docker image is built from it. That way, people can update to the latest version (or go back to a previous version) at their convenience.

I'll keep you posted with my progress.

[Ylianst]

Nice! Thanks. I got so much going on, I have not looked at Docker at all.

[luckydonald]

@whalehub I hope MongoDB wouldn't be running inside the same container, but be a separate container?

[ghost]

@luckydonald Correct.

[GnaXi]

Any news on this? =D

[Ylianst]

I have made zero progress on this. I am completely swamped as it is and about to do a bunch of traveling the first half of November. There are a bunch of unofficial images, I would have to get more familiar with Docker to make a professional image. This is still on the TODO list, but probably sometimes next year if things calm down.

[GnaXi]

No worries! Thanks for the response! =)

[olberger]

I have also created a Docker image for MeshCentral which is quite a bit smaller and simpler than the one techno-express posted. The Docker image and be found here and the source repository here.

These seem to have vanished, @whalehub ... Is there any new location to check ?

[ghost]

@olberger I gave up on this last year because MeshCentral's file uploading feature just refused to work inside of a container. I might give it another shot soon since I've learned a lot about Docker since then.

[edlins]

Very interested in this. Wondering about running containers behind Elastic Beanstalk for load balancing, auto scaling, health checks, etc. Seems that @whalehub and their containers have disappeared. Seems that https://hub.docker.com/r/bsmout/meshcentral is the most popular image at 50k+ downloads and is the only image with a star.

[uldiseihenbergs]

if anyone have interest, i built new docker image myself last week - https://hub.docker.com/r/uldiseihenbergs/meshcentral
It's based on node8.1-slim base-image, cos with newer node i have some issues on my not so new Docker engine. it's plain MC, if you wanna MongoDB, just deploy it on it's own container.

[Jamesits]

I'm currently maintaining a custom docker build (based on Debian & node LTS) and a reference docker-compose config (with fully-working MongoDB setup). Prebuilt images are available on Docker Hub.
https://github.com/Jamesits/docker-meshcentral2

@Ylianst I'm currently tracking updates manually. Any idea to integrate one of the Dockerfiles in this thread and make this official?

[nhalstead]

@Jamesits Ideally this is tied into the Pushing steps of the Repo and builds are run automatically where they can be pushed with a build number (or in this case, release numbers). So to accomplish this you will need the helping of the repo owner to add the webhook for new releases or builds etc to trigger the build and tag process in Docker Hub.

I've configured things like this at my work and personal project but we use Build Numbers from CircleCI then push to DockerHub using that build number. Easy stuff so long as you have access to adding the webhook for GitHub to trigger the CircleCI builds, then CircleCI does the build steps (using a CircleCI Config) to push to DockerHub.

[Jamesits]

@nhalstead I think there is a big blocker that the MeshCentral code is not self-contained, it installs 3rd party npm modules during each launch which kind of defeated the idea of Docker packaging... I think I need a method to generate a list of what might be installed so I can package them into my Docker image but I've not found any.

[nhalstead]

Gotcha, I wouldn't mind taking a crack at it. I've been looking at this repo to see where I could help make it better and more immutable for things like containerized environments.

[JSuenram]

@Ylianst
You can close this, cause no more updates since over a year....

[Jamesits]

Meanwhile I'm still maintaining a separate Docker container repo, tracking upstream releases & dependencies, and verify it for every update, which consumes some of my mental capacity constantly... Please do investigate the possibility of an official Docker image.

[luckydonald]

The fact that the docker image built in this repo can be automated is a huge argument for having it be built here. It could still be labeled "Experimental" or something.

[stavros-k]

@Ylianst I little feedback that could make configuration with env vars available, and probably with little coding from your side.

I've seen projects with a lot of configuration fields, They usually read all the envs in the container. And start by filtering the ones that start with the project name or a specific prefix.

Then they loop on those variables and load the values in the code.
They usually don't store the config anywhere persistent, as env's should be there the next time the container start again.

Each _ means a level deep.

Example:
MESH_SETTINGS_MONGODB="mongodb://...
Would be equal to

{ 
  "settings": {
    "mongodb": "mongodb://..."
    }
}

MESH_SETTINGS_LEVEL1_LEVEL2="something"
Would be equal to

{
  "settings": {
     "level1": {
        "level2": "something"
        }
     }
}

There might be some sharp edges on how to deal with this, as you can't just convert the env names to lowercase and call it a day. Because the config.json contains case sensitive config keys (or is it not case sensitive?)

But you can just require the user to set the env vars with the correct case sensitivity. eg MESH_settings_mongoDb
It just that by convention vars are all capital. But it's not a hard rule.

I believe once the parsing code is in place, it will make container creation a lot easier.
And won't require any kind of scripting to convert env vars into config.json.

I'll see if I can recall any of those projects that did what I mentioned here. So I can link the code snippet for you!

---

A reason requesting something like this, Is that I'm a core maintainer for an App/Chart catalog (TrueCharts.org) focused on creating apps for truenas scale and native helm charts.

We try to aim to make configuration of the installation via a UI in truenas scale.
You can add infinite amount of env vars easily. So having the above method user can do whatever he wants without limits!
But there is no easy way to edit the config file.
You can still of course jump in the command line and edit the config file, but eliminates the point of having "easy installers"

Native helm users don't have that problem as they deal with the chart files directly and can easily edit them.

With current config implementation of Meshcentral is hard to have a way to give the user the ability to add whatever option the user wants in the gui. Apart that it would just be a nightmare of a shell script to translate gui configs to config.json keys, you still won't cover options like arrays of objects where the user can add infinite things there.
Also the Truenas scale UI is limited as we have only handful of options to display (configured via yaml).

It's doable, but will result in a UI that is not usable at all. Plus for every new option added in mesh, it would require also a modification to add it to the gui

---

As always thanks for this awesome project, I discovering new features everyday and realizing how powerful this tool is.

[si458]

@Ylianst I little feedback that could make configuration with env vars available, and probably with little coding from your side.

I've seen projects with a lot of configuration fields, They usually read all the envs in the container. And start by filtering the ones that start with the project name or a specific prefix.

Then they loop on those variables and load the values in the code.
They usually don't store the config anywhere persistent, as env's should be there the next time the container start again.

Each _ means a level deep.

Example:
MESH_SETTINGS_MONGODB="mongodb://...
Would be equal to

{ 
  "settings": {
    "mongodb": "mongodb://..."
    }
}

MESH_SETTINGS_LEVEL1_LEVEL2="something"
Would be equal to

{
  "settings": {
     "level1": {
        "level2": "something"
        }
     }
}

There might be some sharp edges on how to deal with this, as you can't just convert the env names to lowercase and call it a day. Because the config.json contains case sensitive config keys (or is it not case sensitive?)

But you can just require the user to set the env vars with the correct case sensitivity. eg MESH_settings_mongoDb
It just that by convention vars are all capital. But it's not a hard rule.

I believe once the parsing code is in place, it will make container creation a lot easier.
And won't require any kind of scripting to convert env vars into config.json.

I'll see if I can recall any of those projects that did what I mentioned here. So I can link the code snippet for you!

---

A reason requesting something like this, Is that I'm a core maintainer for an App/Chart catalog (TrueCharts.org) focused on creating apps for truenas scale and native helm charts.

We try to aim to make configuration of the installation via a UI in truenas scale.
You can add infinite amount of env vars easily. So having the above method user can do whatever he wants without limits!
But there is no easy way to edit the config file.
You can still of course jump in the command line and edit the config file, but eliminates the point of having "easy installers"

Native helm users don't have that problem as they deal with the chart files directly and can easily edit them.

With current config implementation of Meshcentral is hard to have a way to give the user the ability to add whatever option the user wants in the gui. Apart that it would just be a nightmare of a shell script to translate gui configs to config.json keys, you still won't cover options like arrays of objects where the user can add infinite things there.
Also the Truenas scale UI is limited as we have only handful of options to display (configured via yaml).

It's doable, but will result in a UI that is not usable at all. Plus for every new option added in mesh, it would require also a modification to add it to the gui

---

As always thanks for this awesome project, I discovering new features everyday and realizing how powerful this tool is.

The current docker image has some variables already but not all of them

If you can automate the variables into the current docker image we will happily accept a PR!

Note: the is an official docker image release now on github but not dockerhub

[stavros-k]

If you can automate the variables into the current docker image we will happily accept a PR!

While I don't have the in-depth js skills to implement this. I found 2 small projects that does exactly that,
automate the variables -> json conversion.

They are not-updated in some time now. But for someone with a good js/ts knowledge should be relatively easy to get an idea and implement it in MeshCentral. And probably won't need to be that modular as it will be built for a specific use case.

The two simillar projects:

https://github.com/Ortham/json-schema-env-config
https://github.com/yatki/read-env

Just to be clear:
I know that there are a lot more issues that will have priority
and not asking to move the priority of this, just because I gave 2 links.

Thanks

[si458]

this has been completed so now closing issue
https://github.com/Ylianst/MeshCentral/pkgs/container/meshcentral
docker pull ghcr.io/ylianst/meshcentral:1.1.11