Feat/cluster status #605

Despire · 2023-02-23T15:27:21Z

Closes #440

The ErrorMessage field in the mongo document was replaced by a map<string, Workflow> which holds. the information of the current state for each cluster of the config.

This is also used when the config is being processed by scheduler mainly to indicate where it failed (before it will be picked up by builder to store the states of each cluster)

Frontend polls contextbox every ~40 seconds to get the state of each cluster to display it more readable in the logs

onfig: TestSet-2 - cluster ts2-c-2 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-3 - cluster ts3-c-1 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-hetzner currently in stage ANSIBLER with status IN_PROGRESS Processing stage [1/2] module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-aws currently in stage TERRAFORMER with status IN_PROGRESS Processing stage [1/2] module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-azure currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-oci currently in stage TERRAFORMER with status IN_PROGRESS Processing stage [1/2] module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-gcp currently in stage TERRAFORMER with status IN_PROGRESS Processing stage [1/2] module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
7:31PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend

...

44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
7:44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
7:44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-gcp currently in stage DELETE_NODES with status IN_PROGRESS Processing stage [1/2] module=frontend
7:44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-aws currently in stage ANSIBLER with status IN_PROGRESS Processing stage [2/2] module=frontend
7:44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-azure currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
7:44PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-oci currently in stage ANSIBLER with status IN_PROGRESS Processing stage [2/2] module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage KUBER with status IN_PROGRESS module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage KUBER with status IN_PROGRESS module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-oci currently in stage KUBE_ELEVEN with status IN_PROGRESS Processing stage [2/2] module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-gcp currently in stage DELETE_NODES with status IN_PROGRESS Processing stage [1/2] module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-aws currently in stage ANSIBLER with status IN_PROGRESS Processing stage [2/2] module=frontend
7:45PM INF ../go/services/frontend/server.go:227 > Config: TestSet-1 - cluster ts1-azure currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend

...

8:03PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:04PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:04PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:05PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:05PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:05PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:05PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:06PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:06PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:07PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-1 currently in stage DESTROY_KUBER with status IN_PROGRESS module=frontend
8:07PM INF ../go/services/frontend/server.go:227 > Config: TestSet-2 - cluster ts2-c-2 currently in stage DESTROY_TERRAFORMER with status IN_PROGRESS module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-1 - cluster ts1-gcp has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-2 - cluster ts2-c-2 has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-2 - cluster ts2-c-1 has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-1 - cluster ts1-azure has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-1 - cluster ts1-aws has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-1 - cluster ts1-hetzner has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-1 - cluster ts1-oci has been deleted module=frontend
8:07PM INF ../go/services/frontend/server.go:197 > Config: TestSet-3 - cluster ts3-c-1 has been deleted module=frontend
...

MiroslavRepka

I left some comments, and questions on the PR, otherwise good job.

One more thing I want to mention is that the 40s might be a bit long IMO. I would prefer to have it every ~10s, but let's see what others will say.

services/context-box/server/claudieDB/claudieMongo.go

services/frontend/process_configs.go

services/scheduler/scheduler.go

services/builder/serviceCaller.go

services/frontend/server.go

Despire · 2023-02-24T10:35:39Z

made the time to check for the state 10 seconds
added more granular descriptions for ansibler and kuber

11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:25AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage TERRAFORMER with status IN_PROGRESS module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:26AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:27AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing VPN module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS installing node requirments module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS setting up loadbalancers module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage ANSIBLER with status IN_PROGRESS setting up loadbalancers module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:28AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:29AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:30AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:31AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:32AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:32AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:32AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBE_ELEVEN with status IN_PROGRESS module=frontend
11:32AM INF services/frontend/server.go:227 > Config: TestSet-3 - cluster matus-ts3 currently in stage KUBER with status IN_PROGRESS setting up storage module=frontend
11:32AM INF services/frontend/server.go:215 > workflow finished for cluster matus-ts3 module=frontend

# Conflicts: # manifests/claudie/kustomization.yaml # manifests/testing-framework/kustomization.yaml

MiroslavRepka

LGTM

Despire and others added 8 commits February 23, 2023 16:06

frontend workflow logs

9a58395

fix typo

cb6ec6c

add initial check

ff97408

Merge master into feat/cluster-status

016f1dd

Auto commit - update kustomization.yaml

5f8cc58

add missing update

cbe5004

Merge branch 'master' into feat/cluster-status

65779d3

Auto commit - update kustomization.yaml

2d446bc

Despire requested a review from MiroslavRepka February 24, 2023 07:14

MiroslavRepka reviewed Feb 24, 2023

View reviewed changes

add more granular descriptions

43cc555

MiroslavRepka approved these changes Feb 28, 2023

View reviewed changes

MiroslavRepka added 3 commits March 8, 2023 11:03

Add better progress descriptions

da9fcba

Merge branch 'master' into feat/cluster-status

f5c3755

Merge conflict

6c0ff1d

MiroslavRepka marked this pull request as draft April 11, 2023 11:27

MiroslavRepka and others added 3 commits April 11, 2023 13:53

Fix merge error

d50efc5

Fix errors from merge

f851ca8

correctly handle picked up manifests

83937ee

Despire marked this pull request as ready for review April 13, 2023 16:03

Despire and others added 3 commits April 13, 2023 18:04

Merge branch 'master' into feat/cluster-status

abde7aa

# Conflicts: # manifests/claudie/kustomization.yaml # manifests/testing-framework/kustomization.yaml

trigger ci

43dd714

Auto commit - update kustomization.yaml

beaa89a

MiroslavRepka approved these changes Apr 14, 2023

View reviewed changes

Despire added this pull request to the merge queue Apr 14, 2023

Merged via the queue into master with commit 026d7ce Apr 14, 2023

Despire deleted the feat/cluster-status branch April 14, 2023 13:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/cluster status #605

Feat/cluster status #605

Despire commented Feb 23, 2023 •

edited

MiroslavRepka left a comment

Despire commented Feb 24, 2023

MiroslavRepka left a comment

Feat/cluster status #605

Feat/cluster status #605

Conversation

Despire commented Feb 23, 2023 • edited

MiroslavRepka left a comment

Choose a reason for hiding this comment

Despire commented Feb 24, 2023

MiroslavRepka left a comment

Choose a reason for hiding this comment

Despire commented Feb 23, 2023 •

edited