loadingdock: Updates for k8s, process improvements

* Container now probes ssh public keys from a gerrit group via
  rest API, and gives all users access.
* Affinity to buildmaster node so forklift can access sftp RWO
  mount and the RWO buildmaster package volumes.

containers/loadingdock/Makefile

@@ -1,4 +1,4 @@
-VERSION = 0.1
+VERSION = 0.2
 default:
 	docker build --no-cache --tag docker.io/haiku/loadingdock:$(VERSION) .
 test:

containers/loadingdock/README.md

@@ -6,19 +6,16 @@ build-packages are software packages which are actively used during the build of
 ## Requirements
 
 Environment:
-  * GERRIT_UIDS: List of Gerrit user ID's (number, seen on profile) who can access this service.
+  * GERRIT_SA: Service account in Gerrit to access API (username:password)
+  * ACCESS_GROUP_ID: Group to allow access to via ssh public keys.
 
 Volumes:
   * /sftp: A volume to store incoming packages
-  * /gerrit: A read-only mount of gerrit for access of "All-Users.git"
 
 ## Process
 
-On startup, this container examines the provided GERRIT_UIDS and pulls the public keys for the
-users from Gerrit.
-
-These public keys are allowed access to the service.  Users can submit haiku package files
-for their desired architecture.
+On startup, this container scans the members of ACCESS_GROUP_ID for ssh public
+keys and sets up an sftp server.
 
 After uploaded packages have been modified > 15 minutes ago, they are picked up by forklift
 which moves them to the build-packages packages repository.
@@ -29,14 +26,11 @@ which moves them to the build-packages packages repository.
 
 ## Onboarding Users
 
-Any users in Gerrit can be added to the loadingdock service.
-
-  * User obtains their Gerrit ID from https://review.haiku-os.org/settings/#Profile and provides this ID to the sysadmin team
-  * User confirms their Gerrit account has SSH keys configured
-
-The sysadmins will add the GERRIT_UIDS to the "GERRIT_UIDS" environment variable and
-restart the container.
+* Add any users who need access to the group matching ACCESS_GROUP_ID. (Generally "Loading Dock")
+* Restart this container.
 
-> Eventually the goal is to make this access Gerrit group based
+# TODO
 
-Users will then have access to submit@limerick.ams3.haiku-os.org:1099 to upload packages
+* We might want to periodically rescan the memberships of ACCESS_GROUP_ID and reform the user list.
+  for now the container needs restarted to pickup changes
+* Static SSH host keys

containers/loadingdock/entry.sh

@@ -1,26 +1,30 @@
 #!/bin/bash
 
-GERRIT_SERVER="https://review.haiku-os.org"
+if [ -z "${GERRIT_SERVER}" ]; then
+	echo "This tool needs provided a Gerrit server hostname as GERRIT_SERVER!"
+	exit 1
+fi
 
-if [ -z "${GERRIT_EMAILS}" ]; then
-	echo "This tool needs at least one Gerrit uid in GERRIT_EMAILS to allow access to!"
+if [ -z "${ACCESS_GROUP_ID}" ]; then
+	echo "This tool needs provided a Gerrit group id as ACCESS_GROUP_ID!"
 	exit 1
 fi
 
 if [ -z "${GERRIT_SA}" ]; then
-	echo "This tool need provided a Gerrit service account as GERRIT_SA!"
+	echo "This tool needs provided a Gerrit service account as GERRIT_SA!"
 	exit 1
 fi
 
-lookup_gerrit_id() {
-        curl -s --header "Content-Type: application/json" \
-		--user ${GERRIT_SA} \
-                ${GERRIT_SERVER}/a/accounts/?q=name:$1 | egrep -v "^)]}'$" | jq ".[]._account_id"
+collect_users() {
+	curl -s --header "Content-Type: application/json" \
+		--user ${GERRIT_SA} --insecure \
+		${GERRIT_SERVER}/a/groups/${ACCESS_GROUP_ID}/members \
+		| egrep -v "^)]}'$" | jq -r ".[]._account_id" | tr '\n' ' '
 }
 
 get_ssh_keys() {
 	curl -s --header "Content-Type: application/json" \
-		--user ${GERRIT_SA} \
+		--user ${GERRIT_SA} --insecure \
 		${GERRIT_SERVER}/a/accounts/$1/sshkeys | egrep -v "^)]}'$" | jq -r '.[].ssh_public_key'
 }
 
@@ -35,9 +39,8 @@ if [ ! -f "/etc/ssh/ssh_host_rsa_key" ]; then
 fi
 
 # Collect ssh public keys from users in gerrit
-for email in ${GERRIT_EMAILS}; do
-	GERRIT_UID=$(lookup_gerrit_id ${email})
-	get_ssh_keys $GERRIT_UID >> /etc/authorized_keys/submit
+for id in $(collect_users); do
+	get_ssh_keys $id >> /etc/authorized_keys/submit
 done
 chown -R submit:users /etc/authorized_keys/submit
 chmod 600 /etc/authorized_keys/submit

deployments/loadingdock.yml

@@ -0,0 +1,101 @@
+# Accepts packages from developers for things like build-packages
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: hpkg-loadingdock
+  labels:
+    app: hpkg-loadingdock
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: hpkg-loadingdock
+  template:
+    metadata:
+      labels:
+        app: hpkg-loadingdock
+    spec:
+      # Run on the buildmaster node for shared access to packages pvc by forklift
+      affinity:
+        podAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+              - key: app
+                operator: In
+                values:
+                - buildmaster
+            topologyKey: kubernetes.io/hostname
+      containers:
+      - name: loadingdock
+        image: docker.io/haiku/loadingdock:0.2
+        volumeMounts:
+        - name: sftp
+          mountPath: "/sftp"
+        env:
+        - name: GERRIT_SERVER
+          value: "http://review"
+        - name: GERRIT_SA
+          valueFrom:
+            secretKeyRef:
+              name: gerrit-automation-sa
+              key: key
+        - name: ACCESS_GROUP_ID
+          value: "3c545e4a0e1566910a7be0a13b856de27616be10"
+      volumes:
+      - name: sftp
+        persistentVolumeClaim:
+          claimName: loadingdock-data-pv
+---
+# incoming packages
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: loadingdock-data-pv
+spec:
+  storageClassName: do-block-storage
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: loadingdock
+spec:
+  selector:
+    app: hpkg-loadingdock
+  ports:
+  - name: sftp
+    port: 22
+    targetPort: 22
+---
+apiVersion: traefik.containo.us/v1alpha1
+kind: MiddlewareTCP
+metadata:
+  name: loadingdock-middle
+spec:
+  inFlightConn:
+    amount: 25
+---
+apiVersion: traefik.containo.us/v1alpha1
+kind: IngressRouteTCP
+metadata:
+  name: loadingdock-sftp
+spec:
+  entryPoints:
+  - sftp
+  routes:
+  - match: HostSNI(`*`)
+    services:
+    - name: loadingdock
+      port: 22
+      weight: 10
+      terminationDelay: 900000
+      # breaks sshd, didn't see a config option either
+      #proxyProtocol:
+      #  version: 1
+    middlewares:
+    - name: loadingdock-middle