improve setup for a more production ready workload #2

anubisg1 · 2022-03-31T08:01:38Z

split gui, poller and rest into separate pods
use env variables where needed
poller configured as a statefulset
use ingress to expose suzieq webui and rest server
updated configuration files to match current suzieq requirements and showing multiple namespaces
use official suzieq image

hyposcaler-bot · 2022-03-31T19:45:37Z

Hey! Thanks for the PR, more than happy to merge it, just want to take some time this weekend to try a test deployment first..

bitcollector1 · 2022-04-01T04:47:27Z

Thanks folks, this has been a perfect project to get me introduced to K8, really appreciate the updates to support the newest version of suzieq!

added fsgroup to ensure it works for the non root user

bitcollector1 · 2022-04-07T18:46:26Z

I did manage to find a couple of typos, one of them in is gui.yml

 image: docker pull netenglabs/suzieq:0.17.0

bitcollector1 · 2022-04-08T07:56:47Z

Using manual storage currently

[root@kube4 suzieq-0.17.0]# kubectl get storageclass
No resources found

[root@kube4 suzieq-0.17.0]# kubectl get pv parquet
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
parquet   1000Gi     RWX            Retain           Bound    default/parquet-pvc   manual                  2m19s

[root@kube4 suzieq-0.17.0]# kubectl get pvc parquet-pvc
NAME          STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
parquet-pvc   Bound    parquet   1000Gi     RWX            manual         2m43s

bitcollector1 · 2022-04-08T07:57:24Z

[root@kube4 suzieq-0.17.0]# kubectl describe pvc parquet-pvc
Name:          parquet-pvc
Namespace:     default
StorageClass:  manual
Status:        Bound
Volume:        parquet
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1000Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       sq-gui-deploy-674f644bc6-dlfmf
               sq-poller-0
               sq-rest-deploy-7f485f66b5-5zklc
Events:        <none>

bitcollector1 · 2022-04-08T07:57:58Z

[root@kube4 suzieq-0.17.0]# kubectl describe pv parquet
Name:            parquet
Labels:          type=local
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    manual
Status:          Bound
Claim:           default/parquet-pvc
Reclaim Policy:  Retain
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        1000Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /home/suzieq
    HostPathType:  
Events:            <none>

anubisg1 · 2022-04-08T08:10:08Z

ok, that is a weird setup you have there...
i fully tested the manifests now and i know they work.

in my example:

i have two storage classes definied. one is NFS shared storage class, and the other is a single node disk "microk8s-hostpath"
i know i have permission issues on the NFS side so i use microk8ts-hostpath in this example and i do that by editing the pvc manifest

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: parquet-pvc
spec:
  storageClassName: microk8s-hostpath    <<<<--- HERE 
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

i then apply all manifests:

admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl create namespace suzieq
namespace/suzieq created
admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl apply -f . -n suzieq
configmap/sq-inventory created
configmap/sq-conf created
deployment.apps/sq-gui-deploy created
service/sq-gui-service created
ingress.networking.k8s.io/suzieq-ingress created
service/sq-poller-headless created
statefulset.apps/sq-poller created
persistentvolumeclaim/parquet-pvc created
deployment.apps/sq-rest-deploy created
service/sq-rest-service created
secret/sq-credentials created

everything works:

admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl get all -n suzieq
NAME                                  READY   STATUS    RESTARTS   AGE
pod/sq-gui-deploy-674f644bc6-8r2m5    1/1     Running   0          43s
pod/sq-poller-0                       1/1     Running   0          43s
pod/sq-rest-deploy-7f485f66b5-78cjm   1/1     Running   0          42s

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/sq-gui-service       NodePort    10.152.183.23    <none>        80:30080/TCP   43s
service/sq-poller-headless   ClusterIP   None             <none>        <none>         43s
service/sq-rest-service      NodePort    10.152.183.140   <none>        80:30081/TCP   42s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sq-gui-deploy    1/1     1            1           43s
deployment.apps/sq-rest-deploy   1/1     1            1           43s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/sq-gui-deploy-674f644bc6    1         1         1       43s
replicaset.apps/sq-rest-deploy-7f485f66b5   1         1         1       43s

NAME                         READY   AGE
statefulset.apps/sq-poller   1/1     43s


admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl logs sq-poller-0  -n suzieq
2022-04-08 08:08:36,665 - suzieq.poller.controller - WARNING - log level WARNING


admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl logs sq-gui-deploy-674f644bc6-8r2m5  -n suzieq

  Starting Suzieq GUI
2022-04-08 08:08:46.640 INFO    matplotlib.font_manager: generated new fontManager

  You can now view your Streamlit app in your browser.

  Network URL: http://10.1.7.231:8501
  External URL: http://xx.xx.xx.xx:8501


admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl logs sq-rest-deploy-7f485f66b5-78cjm  -n suzieq
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

bitcollector1 · 2022-04-08T08:14:51Z

nice work! I appreciate all your efforts.....yeah, I inherited this cluster#*ck from the guy we fired ;) Trying to come up to speed but there is so much depth and I'm just getting started.

anubisg1 · 2022-04-08T08:15:48Z

after fixing my NFS share, it works over there too without problems

admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl get pvc -n suzieq
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
parquet-pvc   Bound    pvc-ef00660c-a31e-4208-911c-a6dbcd30c802   5Gi        RWO            nfs-csi        61s


admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl get pod -n suzieq
NAME                              READY   STATUS    RESTARTS   AGE
sq-gui-deploy-674f644bc6-dd8tb    1/1     Running   0          64s
sq-poller-0                       1/1     Running   0          63s
sq-rest-deploy-7f485f66b5-576cn   1/1     Running   0          63s


admin@aks-node-0:~/suzieq-on-k8s/manifest$ kubectl logs sq-poller-0 -n suzieq
2022-04-08 08:13:18,144 - suzieq.poller.controller - WARNING - log level WARNING

therefore i would suggest:

use the manifest as they are, you only need change the configmap if there is anything you need to change, but for testing you don't need to change anything at all
ensure that you kubernetes cluster is configured appropriately in regarding of storage and storageclasses

bitcollector1 · 2022-04-08T08:52:59Z

Looks like the storage is the issue now, no more pods crashing

<invalid>   Warning   FailedMount         pod/sq-gui-deploy-6dd9fd9f89-ck5jr     MountVolume.SetUp failed for volume "sq-conf" : configmap "sq-conf" not found
<invalid>   Warning   FailedMount         pod/sq-poller-0                        MountVolume.SetUp failed for volume "inventory" : configmap "sq-inventory" not found
<invalid>   Warning   FailedMount         pod/sq-poller-0                        MountVolume.SetUp failed for volume "sq-conf" : configmap "sq-conf" not found
<invalid>   Warning   FailedMount         pod/sq-rest-deploy-7f485f66b5-2fd8d    MountVolume.SetUp failed for volume "sq-conf" : configmap "sq-conf" not found

[root@kube4 manifest]# kubectl get all -n suzieq
NAME                                  READY   STATUS              RESTARTS   AGE
pod/sq-gui-deploy-6dd9fd9f89-ck5jr    0/1     ContainerCreating   0          4m3s
pod/sq-poller-0                       0/1     ContainerCreating   0          4m3s
pod/sq-rest-deploy-7f485f66b5-2fd8d   0/1     ContainerCreating   0          4m2s

NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/sq-gui-service       NodePort    10.98.85.55      <none>        80:30080/TCP   4m3s
service/sq-poller-headless   ClusterIP   None             <none>        <none>         4m3s
service/sq-rest-service      NodePort    10.105.205.132   <none>        80:30081/TCP   4m2s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sq-gui-deploy    0/1     1            0           4m3s
deployment.apps/sq-rest-deploy   0/1     1            0           4m3s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/sq-gui-deploy-6dd9fd9f89    1         1         0       4m3s
replicaset.apps/sq-rest-deploy-7f485f66b5   1         1         0       4m3s

NAME                         READY   AGE
statefulset.apps/sq-poller   0/1     4m3s

anubisg1 · 2022-04-08T08:54:22Z

did you apply the configmaps??

kubectl get configmap -n suzieq

bitcollector1 · 2022-04-08T08:54:35Z

Thanks again for everything, pretty sure I can get this last issue figured out. I learned sooooo much today!

bitcollector1 · 2022-04-08T08:58:49Z

apparently I did not....thought it would be built with the one command if I chucked it into the manifest folder but it appears it did not.....so thanks again.....I owe you a few beers ;)

[root@kube4 manifest]# kubectl get configmap
NAME               DATA   AGE
kube-root-ca.crt   1      106d
sq-conf            1      55s
sq-inventory       1      55s

[root@kube4 manifest]# kubectl get configmap -n suzieq
NAME               DATA   AGE
kube-root-ca.crt   1      37d

anubisg1 · 2022-04-08T08:59:46Z

Thanks again for everything, pretty sure I can get this last issue figured out. I learned sooooo much today!

don't worry...

if you just want to test that everything works, follow the "TLDR" section in the readme.
in the issue you just showed me, you didn't apply "configmap.yaml" (or maybe you did in a different namespace).

you can apply all manifests at once (and you should ;) ) .. again, check the quick instructions below

create a namespace

kubectl create namespace suzieq

provide your own certificates or generate a self signed ones

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
kubectl -n suzieq create secret tls suzieq-tls-secret \
  --cert=cert.pem \
  --key=key.pem

deploy suzieq

kubectl -n suzieq apply -f manifest/.
kubectl -n suzieq get all

anubisg1 · 2022-04-08T09:01:09Z

@hyposcaler-bot

all issues are solved, what's left is just due bitcollector knowledge gap.
please review and merge :)

bitcollector1 · 2022-04-08T17:57:44Z

Okay, so the issue was permissions on the worker node where these files were created! It turns out there was no write permissions by default. Man I'm ready for the weekend now! Fun times, LOL.

update to 0.17.1

bitcollector1 · 2022-04-14T20:36:25Z

I ran into an issue where the poller kept crashing and also seemed to take awhile to get results so I added a second worker and bumped up the resources and things are looking much better now!

Here was the error I managed to catch

root@kube4 suzieq-0.17.0]# kubectl get all -n suzieq
NAME                                  READY   STATUS      RESTARTS   AGE
pod/haproxy                           1/1     Running     0          18h
pod/sq-gui-deploy-b9dd5f6bd-vd9rl     1/1     Running     0          2m53s
pod/sq-poller-0                       0/1     OOMKilled   1          2m52s
pod/sq-rest-deploy-6dc7c876f4-bxcpk   1/1     Running     0          2m52s

containers:
        - name: poller
          image: netenglabs/suzieq:0.17.1
          command: ["sq-poller", "-I", "inventory.yml", "-w 2"]
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "1000Mi"
              cpu: "1000m"

bitcollector1 · 2022-04-14T20:38:05Z

Last step is to figure out the HA proxy for my bare metal setup, this has been kicking my butt pretty hard. It's so much easier to just use a cloud provider it seems.

anubisg1 · 2022-04-15T16:56:47Z

Last step is to figure out the HA proxy for my bare metal setup, this has been kicking my butt pretty hard. It's so much easier to just use a cloud provider it seems.

That's easy, just install metallb and configure your ingress svc with service type load balancer

anubisg1 · 2022-05-23T11:09:45Z

hello @hyposcaler-bot is there any expectation to merge?

hyposcaler-bot · 2022-05-24T12:18:00Z

Feel free to fork, I’m overwhelmed with work, and have unfortunately been neglecting this repo. From: Andrea Florio ***@***.***> Sent: Monday, May 23, 2022 7:10 AM To: hyposcaler/suzieq-on-k8s ***@***.***> Cc: Dennis Fanshaw ***@***.***>; Mention ***@***.***> Subject: Re: [hyposcaler/suzieq-on-k8s] improve setup for a more production ready workload (PR #2) hello @hyposcaler-bot<https://github.com/hyposcaler-bot> is there any expectation to merge? — Reply to this email directly, view it on GitHub<#2 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APMPDAQHVFZ63D7T53V7MHLVLNRQNANCNFSM5SEQD5YQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

Adjusted the prefix in ingress

anubisg1 added 23 commits March 31, 2022 09:29

Update configmap.yml

be12731

Delete deployment.yml

c2591aa

Update pvc.yml

728f533

Delete suzieq-cfg.yml

7872560

Delete service.yml

3e504a8

Create gui.yml

ce836be

Create rest.yml

407e83f

Create poller.yml

d884eb5

Create ingress.yml

0201481

Create secrets.yml

a3a8ac2

Update ingress.yml

0768d8e

Update README.md

a6429dc

Update configmap.yml

c74dac2

Update poller.yml

7531431

Update secrets.yml

e853d59

Update README.md

77b7a83

Update rest.yml

ccb600d

Update gui.yml

c066894

Update poller.yml

85ffc6f

Update rest.yml

793a960

Update poller.yml

164ecdc

Update gui.yml

3d5b25d

Update gui.yml

3a61d30

anubisg1 added 4 commits April 4, 2022 17:23

Update gui.yml

ca2c050

added fsgroup to ensure it works for the non root user

Update poller.yml

c62539e

Update gui.yml

d1ece8e

Update rest.yml

a9463bf

anubisg1 added 2 commits April 8, 2022 08:00

organize into manifest folder

698191f

Merge branch 'main' of https://github.com/anubisg1/suzieq-on-k8s

1eca739

anubisg1 added 3 commits April 8, 2022 08:24

add TLDR to readme

c553f34

Update README.md

36bfa5d

Update README.md

06e648a

anubisg1 added 3 commits April 14, 2022 18:00

Update gui.yml

52e4343

update to 0.17.1

Update poller.yml

b96c8e9

update to 0.17.1

Update rest.yml

f519378

update to 0.17.1

anubisg1 added 3 commits June 2, 2022 22:48

Update ingress.yml

827ad8f

Adjusted the prefix in ingress

1pass test

da039a7

test 1pass signature

3d829e0

anubisg1 closed this by deleting the head repository Mar 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve setup for a more production ready workload #2

improve setup for a more production ready workload #2

anubisg1 commented Mar 31, 2022

hyposcaler-bot commented Mar 31, 2022

bitcollector1 commented Apr 1, 2022

bitcollector1 commented Apr 7, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022 •

edited

Loading

bitcollector1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022 •

edited

Loading

bitcollector1 commented Apr 8, 2022 •

edited

Loading

anubisg1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022 •

edited

Loading

anubisg1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 14, 2022

bitcollector1 commented Apr 14, 2022

anubisg1 commented Apr 15, 2022

anubisg1 commented May 23, 2022

hyposcaler-bot commented May 24, 2022 via email

improve setup for a more production ready workload #2

improve setup for a more production ready workload #2

Conversation

anubisg1 commented Mar 31, 2022

hyposcaler-bot commented Mar 31, 2022

bitcollector1 commented Apr 1, 2022

bitcollector1 commented Apr 7, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022 • edited Loading

bitcollector1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022 • edited Loading

bitcollector1 commented Apr 8, 2022 • edited Loading

anubisg1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022 • edited Loading

anubisg1 commented Apr 8, 2022

anubisg1 commented Apr 8, 2022

bitcollector1 commented Apr 8, 2022

bitcollector1 commented Apr 14, 2022

bitcollector1 commented Apr 14, 2022

anubisg1 commented Apr 15, 2022

anubisg1 commented May 23, 2022

hyposcaler-bot commented May 24, 2022 via email

anubisg1 commented Apr 8, 2022 •

edited

Loading

anubisg1 commented Apr 8, 2022 •

edited

Loading

bitcollector1 commented Apr 8, 2022 •

edited

Loading

bitcollector1 commented Apr 8, 2022 •

edited

Loading