tags | |
---|---|
|
A short introduction into the infrastructure of Mint System.
Early on we wanted to be able to self-host web application.
ERP-Systems are operating system for companies
- They have to be hosted reliably
- Companies want know where their data is
Infrastructure as Code:
- Every system / server can be rebuilt from code
- The deployment of configuration must be automated
Run apps with Docker containers:
- Manage containers not apps
- Every decent web app provides Docker images
We don't want to manage hardware. We use these providers:
- Hetzner
- Exoscale
- Infomaniak
- Ungleich
And service partners.
Everything is done with Ansible. There is a role for that: https://ansible.build/#roles
- Install and configure the OS
- Manage firewalls and access rights
- Deploy Docker containers
- Manage backups and cron jobs
- Build Wireguard networks
We provide a managed server service.
Every customer has a server. No shared environments.
Every application is a hosting offer.
For things that cannot be automated there are scripts: https://ansible.build/scripts.html
Ansible roles rely on these helper scripts. Helper scripts work independent of Ansible.
To monitor servers, containers and applications we use Prometheus/Grafana.
Prometheus provides "Exporters" for many applications. Data collection and visualization is simple.
Backups are defined in Ansible and are done using [[restic]].
- In Ansible inventory a backup set is defined
- Ansible creates a cron job to run the backup
- The backup job uses helper scripts to create local backups
- The local backups are snapshoted with restic to the backup server
- All backup data on the backup server is mirrored to an S3 bucket
On every server there is Nginx-instance running.
The Nginx config is generated by Ansible: https://ansible.build/roles/nginx/
Some roles require specific configs: https://ansible.build/roles/collabora_code/#nginx-config
Certificates are managed with certbot and let's encrypt.
We a running Keycloak and integrate it using OAuth/OpenID Connect.
It is possible to manage all Keycloak config in Ansible.
We try to apply best practices:
- Update our applications / Docker images
- Linux server patching
- Basics: fail2ban, ssh pubkey, named users
Nginx WAF with OWASP has been tried. It was too much effort to train.
Configuration drift
One an Ansible role or a config is update we have to apply this change to all hosts. This rarely done.
Proxy config not integrated
The proxy config is not provided by the Ansible role. It would be better if the application provides the config.
Backup verification
We cannot be sure that all backups are working. So far they did.