k8s-home

This repository builds & stores k8s configuration.

Thanks to cdk8s, it builds kubernetes charts for all helm charts & apps prior to store them in the generated branch. This branch is then used by Flux CD to reconcile the k8s state with the built charts.

Contents

I install and manage several apps for both infra and endusers. It includes secure encrypted storage with longhorn, secrets management, several databases, and some apps.

Initial installation with flux

Initial installation of Flux CD comes with the flux binary. The whole installation procedure is available on the official website.

Please start fluxcd:

flux bootstrap git \
  --url=ssh://git@git.mkz.me/mycroft/k8s-home.git \
  --private-key-file=/tmp/rsa \
  --branch=generated \
  --path=generated/

Upgrading flux

Dump key file:

k get secret -n flux-system -o yaml flux-system | yq .data.identity -r | base64 -d > /tmp/rsa

Then re-run the bootstrap command as seen in the Installation section.

Upgrade services

To check outdated Helm charts, use:

go build && ./k8s-home -check-version

Services must be updated by modifying versions in source code.

Security

Secrets can be either stored in source code thanks to sealed-secrets or in vault and deployed as a Secret in Kubernetes with external-secrets.

Ingresses can be protected thanks to traefik-forward-auth which perform an Oauth process using my own private and external gitea instance.

Services

Traefik tweaks

traefik should/must be reconfigured to redirect all http:// requests to https://.

On master node, edit /var/lib/rancher/k3s/server/manifests/traefik-config.yaml and add the following snippet in valuesContent section:

  valuesContent: |-
    ports:
      web:
        redirectTo: websecure

k3s will automatically redeploy the helm chart.

Also, it should allow IngressRoutes to use middlewares from another namespaces the ingressroutes is in (disabled by default)

    providers:
      kubernetesCRD:
        allowCrossNamespace: true

Note: Do not edit traefik.yaml. It is overwritten on restart.

Viewing traefik dashboard

port-forward the 9000 port and reach http://localhost:9000/dashboard/:

> k port-forward -n kube-system (k get pods -n kube-system | grep ^traefik | cut -d' ' -f1) 9000:9000
Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from [::1]:9000 -> 9000

Kubernetes-Dashboard

Generate a login and use it on https://kubernetes-dashboard.services.mkz.me/

kubectl -n kubernetes-dashboard create token admin

You need a recent kubectl binary to be able to do this!

This is fully described on https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

See https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/README.md#login-view

Other services

• Grafana
• Vault

Notes

Force a HelmRelease reconcile after an undetected change

Add/edit the annotation:

kubectl annotate --overwrite -n monitoring helmrelease/prometheus reconcile.fluxcd.io/requestedAt="$(date +%s)"

On HelmRelease 'install retries exhausted'

After fixing the issue, suspend & resume the installation:

> flux suspend hr -n kubernetes-dashboard kubernetes-dashboard
► suspending helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease suspended

> flux resume hr -n kubernetes-dashboard kubernetes-dashboard
► resuming helmrelease kubernetes-dashboard in kubernetes-dashboard namespace
✔ helmrelease resumed
◎ waiting for HelmRelease reconciliation
✔ HelmRelease reconciliation completed
✔ applied revision 5.11.0

Install whatismyip

It is required to not use the default externalTrafficPolicy to be able to get real-ip.

Please use the following configuration:

# cat /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--serverstransport.insecureskipverify=true"
    service:
      spec:
        externalTrafficPolicy: Local

# kubectl apply -f /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
# kubectl get svc -o yaml -n kube-system traefik | grep -i policy
  externalTrafficPolicy: Local
  internalTrafficPolicy: Cluster

See https://github.com/traefik/traefik-helm-chart/blob/master/traefik/values.yaml for more values.

Creating a secret using Sealed-Secrets

See https://github.com/bitnami-labs/sealed-secrets:

# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json >mysecret.json

# Warning! Do not forget the namespace & the secret name. By default SealedSecret is very strict about those.

# This is the important bit:
# (note default format is json!)
kubeseal <mysecret.json >mysealedsecret.json

# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.

# Eventually:
kubectl create -f mysealedsecret.json

Creating PostgreSQL storage

To create and be able to use a PostgreSQL database, a few steps are required:

• In postgresql.go, add a record in the list. Commit and push the file; It will spawn the database and create secrets in the postgres namespace;
• When database is ready, connect to database with psql to allow user to create tables:

$ k exec -ti -n postgres postgres-instance-0 -- psql -U dex-admin dex
dex=# GRANT CREATE ON SCHEMA public TO PUBLIC;
GRANT

• Retrieve username & password from Secret in the postgres namespace:

$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.username | base64 -d
...
$ k get secret -n postgres dex-admin.postgres-instance.credentials.postgresql.acid.zalan.do -o yaml | yq -r .data.password | base64 -d
...

• Create a record in vault (ie: namespaces/<namespace>/postgresql) and create an username and a password record with the values retrieved in the Secret.

• In the target namespace, create an ExternalSecret