New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add a minimal prometheus server manifest #4687
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Could you attach some screenshots? E.g.
- argo_workflows_count changing over time
- argo_workflows_queue_latency_bucket showing latency for workflows?
- otherL
- Could you do a hands-on with me and Bala?
Would be happy to. Today at standup? |
Stand-up demo - yes please. What would be neat? |
Will have this ready by standup time |
Signed-off-by: Simon Behar <simbeh7@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we see a demo at stand-up please?
Makefile
Outdated
ifeq ($(PROFILE),prometheus) | ||
kubectl -n $(KUBE_NAMESPACE) scale deploy/prometheus --replicas 1 | ||
endif | ||
kubectl -n $(KUBE_NAMESPACE) wait --for=condition=Ready pod --all -l app --timeout 1m || true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my experience kubectl wait
hangs/gets stuck even after all pods are ready (which happens after around 30 seconds). Confirmed this by running the same command in another terminal window while it is stuck, and in that case the command completes successfully.
We still want to have a wait before we do port-forwarding
, but we don't want this flakey command to kill start up, so I lowered the timeout and added || true
to ensure this flakey command doesn't stop the build. If the pods actually aren't ready, the next command will fail anyway (which in my experience does not happen).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've noted this is flakey too. Could we make the wait specific? E.g.
kubectl -n $(KUBE_NAMESPACE) wait --for=condition=Ready pod -l app=prometheus
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
might need a short sleep before hand
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added sleep
@@ -3,12 +3,14 @@ kind: Kustomization | |||
|
|||
resources: | |||
- ../../../../manifests/quick-start/mysql | |||
- ../../../../manifests/quick-start/base/prometheus |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this will work, because RUN!=kubernetes typically , I think you only need to add to PROFILE=prometheus
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure thing
@alexec Ready again |
Signed-off-by: Simon Behar simbeh7@gmail.com
Checklist: