New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Kubernetes performance dashboard #3118

Closed
roberthbailey opened this Issue Dec 23, 2014 · 26 comments

Comments

Projects
None yet
10 participants
@roberthbailey
Member

roberthbailey commented Dec 23, 2014

Build/find a dashboard to record performance metrics. The dashboard should have graphs of metrics over time and be queriable.

@jeremyeder

This comment has been minimized.

Show comment
Hide comment
@jeremyeder

jeremyeder Jan 8, 2015

Member

Has there been anything proposed for how to collect system metrics while a test is running? What is the Google team familiar with and what do they use now?

Member

jeremyeder commented Jan 8, 2015

Has there been anything proposed for how to collect system metrics while a test is running? What is the Google team familiar with and what do they use now?

@dchen1107

This comment has been minimized.

Show comment
Hide comment
@dchen1107
Member

dchen1107 commented Jan 9, 2015

#415 too

@roberthbailey

This comment has been minimized.

Show comment
Hide comment
@roberthbailey

roberthbailey Jan 9, 2015

Member

@jeremyeder nothing has been proposed yet and this issue is to collect/present such proposals.

I think the Google folks may actually be at a disadvantage is presenting proposals for a dashboard since we are generally used to our internal tools that handle this sort of thing for us.

Member

roberthbailey commented Jan 9, 2015

@jeremyeder nothing has been proposed yet and this issue is to collect/present such proposals.

I think the Google folks may actually be at a disadvantage is presenting proposals for a dashboard since we are generally used to our internal tools that handle this sort of thing for us.

@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc

timothysc Feb 18, 2015

Member

@jeremyeder is there any format that would be easier to consume data?

Member

timothysc commented Feb 18, 2015

@jeremyeder is there any format that would be easier to consume data?

@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc

timothysc Apr 17, 2015

Member

@quinton-hoole perhaps we can write the proposals against this issue, happy to help on this one.

Member

timothysc commented Apr 17, 2015

@quinton-hoole perhaps we can write the proposals against this issue, happy to help on this one.

@fgrzadkowski

This comment has been minimized.

Show comment
Hide comment
@fgrzadkowski

fgrzadkowski Apr 20, 2015

Member

@mwielgus

See #6981, which is closely related. I think that we should use Prometheus server for this to be consistent with the current mechanism to expose metrics.

Member

fgrzadkowski commented Apr 20, 2015

@mwielgus

See #6981, which is closely related. I think that we should use Prometheus server for this to be consistent with the current mechanism to expose metrics.

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 5, 2015

Member

I'm working on this right now. the plan is to add a prometheus introspector to examples/ . per conversation with @timothysc

Member

jayunit100 commented May 5, 2015

I'm working on this right now. the plan is to add a prometheus introspector to examples/ . per conversation with @timothysc

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 5, 2015

Contributor

@jayunit100, what is your plan regarding Prometheus config?

Contributor

mwielgus commented May 5, 2015

@jayunit100, what is your plan regarding Prometheus config?

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 5, 2015

Member

hi @mwielgus I'll put a rough PR or description together and update in this thread.

Member

jayunit100 commented May 5, 2015

hi @mwielgus I'll put a rough PR or description together and update in this thread.

@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc

timothysc May 5, 2015

Member

@mwielgus Tentatively we'll create an examples for prom to spin a Pod as an introspective service. The whole round-trip integration with Heapster #5637 looks to be TBD still at this time.

/cc @vishh @vmarmol

Member

timothysc commented May 5, 2015

@mwielgus Tentatively we'll create an examples for prom to spin a Pod as an introspective service. The whole round-trip integration with Heapster #5637 looks to be TBD still at this time.

/cc @vishh @vmarmol

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 5, 2015

Contributor

Recently I did some experiments with prometheus on kubernetes being configured in a kindof introspective manner using env variables and command line attributes - you can take a look at a draft here master...mwielgus:prometheus. If it is anyhow in line with your plan I can modify/polish it and submit.

Contributor

mwielgus commented May 5, 2015

Recently I did some experiments with prometheus on kubernetes being configured in a kindof introspective manner using env variables and command line attributes - you can take a look at a draft here master...mwielgus:prometheus. If it is anyhow in line with your plan I can modify/polish it and submit.

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 5, 2015

Member

@mwielgus are you working on this? I dont want to steal it from you.

BUT I'm happy to pollish and submit a PR if you want to hand it off !

Just let me know ! THANKS !

Member

jayunit100 commented May 5, 2015

@mwielgus are you working on this? I dont want to steal it from you.

BUT I'm happy to pollish and submit a PR if you want to hand it off !

Just let me know ! THANKS !

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 5, 2015

Contributor

@jayunit100 It depends whether this draft is in line with your plan and makes any sense at all (I'm very new to Kubernetes). If yes and this will save you some time then sure, I would like submit it. If not really and/or you need it ASAP then I'm fine with you taking over this.

Contributor

mwielgus commented May 5, 2015

@jayunit100 It depends whether this draft is in line with your plan and makes any sense at all (I'm very new to Kubernetes). If yes and this will save you some time then sure, I would like submit it. If not really and/or you need it ASAP then I'm fine with you taking over this.

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 5, 2015

Member

okay i see how your doing it - you are using kubernetes-ro as the entrypoint to find the prometheus metrics target exported from api-server. thats a good idea i didnt realize the kube-ro server worked that way- i always wondered what it was for.... im still learning kube internals also :).

two things i think might need to change

  • where is your DATABASE_URL ? I alway set that wehn running. without it i thought prometheus will not run at all ? (ACTUALLY nevermind - I see the dashboard is unimplemented here. so i will ad that on)
  • its not HA - if this pod fails, then you lose all data when you relaunch on a new minion.

Let me know thoughts on those two ^ @mwielgus , then we can decide next steps ! thanks again for sharing this!

Member

jayunit100 commented May 5, 2015

okay i see how your doing it - you are using kubernetes-ro as the entrypoint to find the prometheus metrics target exported from api-server. thats a good idea i didnt realize the kube-ro server worked that way- i always wondered what it was for.... im still learning kube internals also :).

two things i think might need to change

  • where is your DATABASE_URL ? I alway set that wehn running. without it i thought prometheus will not run at all ? (ACTUALLY nevermind - I see the dashboard is unimplemented here. so i will ad that on)
  • its not HA - if this pod fails, then you lose all data when you relaunch on a new minion.

Let me know thoughts on those two ^ @mwielgus , then we can decide next steps ! thanks again for sharing this!

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 5, 2015

Contributor

Database url/HA - I use local storage (see master...mwielgus:prometheusdiff-6fb454b575d1bd04454e4ff4e3048b32R95) which I guess can be placed on GCE persistent disk. But if your preference is to write to InfluxDB then i can modify it accordingly.

Contributor

mwielgus commented May 5, 2015

Database url/HA - I use local storage (see master...mwielgus:prometheusdiff-6fb454b575d1bd04454e4ff4e3048b32R95) which I guess can be placed on GCE persistent disk. But if your preference is to write to InfluxDB then i can modify it accordingly.

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 5, 2015

Member

@mwielgus sounds great. how about this - just tell me what you want to do so we can keep pressing forward.

Either :

  • I can adopt your patch, add some commits on top, and then we can just submit it as a joint PR by layering the commits ? That way you get credit for your hard (and awesome) work :)

OR :

  • You can finish this patch off.

We really need this so I'm happy to take over if you're busy with other tasks. Either way you will get credit for the work thus far - so its just a matter of figuring out who is able to work on it and finish it

Member

jayunit100 commented May 5, 2015

@mwielgus sounds great. how about this - just tell me what you want to do so we can keep pressing forward.

Either :

  • I can adopt your patch, add some commits on top, and then we can just submit it as a joint PR by layering the commits ? That way you get credit for your hard (and awesome) work :)

OR :

  • You can finish this patch off.

We really need this so I'm happy to take over if you're busy with other tasks. Either way you will get credit for the work thus far - so its just a matter of figuring out who is able to work on it and finish it

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 5, 2015

Contributor

@jayunit100 let's go with the first option.

Contributor

mwielgus commented May 5, 2015

@jayunit100 let's go with the first option.

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 6, 2015

Member

okay ! TL;DR summary for this guy...

  • we'll add prometheus replication controller to add-ons
  • it will use kubernetes-ro service endpoint
  • it will be based on mwieigus initial patch above
  • i'll layer prom dash into it as well, so that the whole thing is spun up on the fly
Member

jayunit100 commented May 6, 2015

okay ! TL;DR summary for this guy...

  • we'll add prometheus replication controller to add-ons
  • it will use kubernetes-ro service endpoint
  • it will be based on mwieigus initial patch above
  • i'll layer prom dash into it as well, so that the whole thing is spun up on the fly
@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc

timothysc May 6, 2015

Member

👍

Member

timothysc commented May 6, 2015

👍

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 6, 2015

Member
Member

jayunit100 commented May 6, 2015

@mwielgus

This comment has been minimized.

Show comment
Hide comment
@mwielgus

mwielgus May 6, 2015

Contributor
  • Yes.
  • Sure.
Contributor

mwielgus commented May 6, 2015

  • Yes.
  • Sure.
@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 7, 2015

Member

@mwielgus okay, about to put in the PR, tested a few times and refined a few minor things.... I have one question - why did you use a volumeMount ? It seems unnecessary unless you're planning on having other containers in the pod?

Member

jayunit100 commented May 7, 2015

@mwielgus okay, about to put in the PR, tested a few times and refined a few minor things.... I have one question - why did you use a volumeMount ? It seems unnecessary unless you're planning on having other containers in the pod?

@jayunit100

This comment has been minimized.

Show comment
Hide comment
@jayunit100

jayunit100 May 7, 2015

Member

okay, heres the PR ! #7878

Member

jayunit100 commented May 7, 2015

okay, heres the PR ! #7878

@timothysc

This comment has been minimized.

Show comment
Hide comment
@timothysc

timothysc May 7, 2015

Member

awesome work guys!

Member

timothysc commented May 7, 2015

awesome work guys!

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp May 13, 2015

Member

@bprashanth was interested in this possibly as a way to store the results from his micro-benchmarks

Member

davidopp commented May 13, 2015

@bprashanth was interested in this possibly as a way to store the results from his micro-benchmarks

zmerlynn added a commit that referenced this issue May 14, 2015

Merge pull request #7878 from jayunit100/prometheus
Prometheus dashboard add-on kube application ( #3118 )
@borg286

This comment has been minimized.

Show comment
Hide comment
@borg286

borg286 May 19, 2015

I apologize if this isn't the right place to comment on the above PR but I'm new to github.
On line 35 of your run-prometheus.sh script, why not have the user simply pass in the targets in the form of hostname1:port1,hostname2:port2,hostname3:port3.... instead of requiring the user to define a slew of environment variables?
This would greatly simplify and clarify the description on line 28.
Would it be clearer to have on line 21 instead of KUBERNETES_RO to be instead localhost:9090 ? And make a comment that this configures prometheus to monitor itself.

borg286 commented May 19, 2015

I apologize if this isn't the right place to comment on the above PR but I'm new to github.
On line 35 of your run-prometheus.sh script, why not have the user simply pass in the targets in the form of hostname1:port1,hostname2:port2,hostname3:port3.... instead of requiring the user to define a slew of environment variables?
This would greatly simplify and clarify the description on line 28.
Would it be clearer to have on line 21 instead of KUBERNETES_RO to be instead localhost:9090 ? And make a comment that this configures prometheus to monitor itself.

@fejta fejta closed this Mar 17, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment