Add allocations metrics #963

cyriltovena · 2019-07-30T17:09:06Z

This adds gameserver allocations metrics with a dashboard. The metrics tracks allocations count, rates and latency per nodes, fleet and status.

I've also added everything required to handle multi cluster allocations but nothing used in the dashboard yet.

I've fixed some issues with kind cluster and also took the time to add metrics and healthcheck for the allocator service. (with a grafana dashboard)

Documentation has been updated.

Closes #144

agones-bot · 2019-07-30T19:42:55Z

Build Failed 😱

Build Id: c5aca92e-cc39-4d57-a560-5aff291d35c9

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

aLekSer · 2019-07-30T20:05:38Z

Issue in make test-install-yaml
@cyriltovena, you need to run make gen-install and commit changes.

agones-bot · 2019-07-30T20:45:48Z

Build Failed 😱

Build Id: c72a940e-9e15-4b60-b154-eb6cdb765116

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2019-07-31T00:52:42Z

Hitting retry!

agones-bot · 2019-07-31T01:25:47Z

Build Succeeded 👏

Build Id: f81b5185-8591-4b53-b5ff-ad8f6cb93574

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.12.0-8866ae8
image: gcr.io/agones-images/agones-sdk:0.12.0-8866ae8
image: gcr.io/agones-images/agones-ping:0.12.0-8866ae8
Linux C++ SDK (build): agonessdk-0.12.0-8866ae8-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.12.0-8866ae8.zip

A preview of the website (the last 30 builds are retained):

https://8866ae8-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/963/head:pr_963 && git checkout pr_963
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.12.0-8866ae8

roberthbailey · 2019-07-31T20:55:01Z

@markmandel - have you had time to review this? Are you still hoping to get it in before the 0.12.0 RC?

agones-bot · 2019-07-31T21:28:30Z

Build Succeeded 👏

Build Id: b80982f4-2e2e-46ad-b8a7-f69cd755796e

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.12.0-2034569
image: gcr.io/agones-images/agones-sdk:0.12.0-2034569
image: gcr.io/agones-images/agones-ping:0.12.0-2034569
Linux C++ SDK (build): agonessdk-0.12.0-2034569-linux-arch_64.tar.gz
SDK Server: agonessdk-server-0.12.0-2034569.zip

A preview of the website (the last 30 builds are retained):

https://2034569-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/963/head:pr_963 && git checkout pr_963
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.12.0-2034569

cyriltovena · 2019-07-31T21:44:47Z

You can go ahead Robert, so I can start fixing if you find things, I created a structure to embed the tagging/record logic because I didn't wanted to add a lot of nested code in the current allocation implementation. I think that was a good idea but I would love feedback

pooneh-m

Thanks for adding metrics for allocation and liveliness check.

cmd/allocator/main.go

pooneh-m · 2019-07-31T21:57:53Z

cmd/allocator/main.go

+		var err error
+		lock := sync.Mutex{}
+		// force a pod restart if the https server exits.
+		health.AddLivenessCheck("allocator-https", func() error {


Can we instead having two place for liveliness check and healthz only use healthz and leverage wait group to end the process if ListenAndServeTLS returns?

yes good idea, I'm doing it, way simpler :)

markmandel · 2019-07-31T22:23:00Z

@markmandel - have you had time to review this? Are you still hoping to get it in before the 0.12.0 RC?

I only just woke up 😁- so if there is a chance to get this in before I do the RC release in a few hours, that would be awesome, I think.

cyriltovena · 2019-08-05T18:01:48Z

Thanks @pooneh-m, I'll look into it.

markmandel · 2019-08-14T16:49:21Z

@cyriltovena - was going to do a review, but looks like you have some outstanding review items from @pooneh-m , so I'll hold off until they are complete.

agones-bot · 2019-08-23T15:08:18Z

Build Failed 😱

Build Id: 6c91761c-f24f-4c4c-b279-8c7d74367b46

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

cyriltovena · 2019-08-23T15:53:54Z

@pooneh-m I fixed all your feedbacks, thanks !

agones-bot · 2019-08-23T16:02:06Z

Build Failed 😱

Build Id: 28b5e835-1d6d-4acf-90e5-c627a53090bd

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2019-08-23T18:56:34Z

Oops, make gen-install 🤣

cyriltovena · 2019-08-24T08:38:59Z

I'll check everything one more time.

cyriltovena · 2019-08-24T08:42:41Z

Oops, make gen-install 🤣

it got rollback by a rebase....

cyriltovena · 2019-08-24T10:04:39Z

errr, something went wrong with the rebase.... I lost some new dashboard....

agones-bot · 2019-08-24T13:24:33Z

Build Failed 😱

Build Id: 4147cd83-c2af-4452-98d6-5884072f035f

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2019-08-24T15:15:33Z

Build Succeeded 👏

Build Id: 36203b24-b4f4-44c6-8558-2002bbd8b1df

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-e6dc4cf
image: gcr.io/agones-images/agones-sdk:1.0.0-e6dc4cf
image: gcr.io/agones-images/agones-ping:1.0.0-e6dc4cf
Linux C++ SDK (build): agonessdk-1.0.0-e6dc4cf-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-e6dc4cf.zip

A preview of the website (the last 30 builds are retained):

https://e6dc4cf-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/963/head:pr_963 && git checkout pr_963
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-e6dc4cf

agones-bot · 2019-08-24T20:30:42Z

Build Succeeded 👏

Build Id: a4ffeae0-f819-479e-a2b1-4cc24626c0fc

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.0.0-f76570a
image: gcr.io/agones-images/agones-sdk:1.0.0-f76570a
image: gcr.io/agones-images/agones-ping:1.0.0-f76570a
Linux C++ SDK (build): agonessdk-1.0.0-f76570a-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.0.0-f76570a.zip

A preview of the website (the last 30 builds are retained):

https://f76570a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/963/head:pr_963 && git checkout pr_963
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.0.0-f76570a

This comment has been minimized.

Sign in to view

cyriltovena force-pushed the gsa-metrics branch from 79c87ad to 6c42a03 Compare July 30, 2019 17:32

googleforgames deleted a comment from agones-bot Jul 30, 2019

This comment has been minimized.

Sign in to view

pooneh-m reviewed Jul 31, 2019

View reviewed changes

markmandel added area/operations Installation, updating, metrics etc feature-freeze-do-not-merge Only eligible to be merged once we are out of feature freeze (next full release) kind/feature New features for Agones labels Aug 1, 2019

markmandel removed the feature-freeze-do-not-merge Only eligible to be merged once we are out of feature freeze (next full release) label Aug 13, 2019

cyriltovena added 6 commits August 23, 2019 17:00

instrument http server with metrics and count gsa

2879f49

fixes kind-push

610ef1b

adds missing image for kind

2e31946

wip

0178337

cleanup recording code

63d5493

wip

6e3b5e8

cyriltovena force-pushed the gsa-metrics branch from 2034569 to 6e3b5e8 Compare August 23, 2019 15:02

fixes review feedbacks

75aa30f

pooneh-m approved these changes Aug 23, 2019

View reviewed changes

cyriltovena added 2 commits August 24, 2019 15:06

fix allocations dashboard lost by rebase

bd85ae9

fix controller logic lost in rebase

af0b5f6

fixes helm affected by rebased

e6dc4cf

fixes kind-push

f76570a

cyriltovena merged commit 8331c01 into googleforgames:master Aug 24, 2019

cyriltovena mentioned this pull request Aug 24, 2019

Instrument Kubernetes API and Allocator with Tracing #1012

Open

markmandel added this to the 1.0.0 milestone Sep 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add allocations metrics #963

Add allocations metrics #963

cyriltovena commented Jul 30, 2019 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

agones-bot commented Jul 30, 2019

aLekSer commented Jul 30, 2019

agones-bot commented Jul 30, 2019

markmandel commented Jul 31, 2019

agones-bot commented Jul 31, 2019

roberthbailey commented Jul 31, 2019

agones-bot commented Jul 31, 2019

cyriltovena commented Jul 31, 2019

pooneh-m left a comment

pooneh-m Jul 31, 2019

cyriltovena Aug 23, 2019

markmandel commented Jul 31, 2019

cyriltovena commented Aug 5, 2019

markmandel commented Aug 14, 2019

agones-bot commented Aug 23, 2019

cyriltovena commented Aug 23, 2019

agones-bot commented Aug 23, 2019

markmandel commented Aug 23, 2019

cyriltovena commented Aug 24, 2019

cyriltovena commented Aug 24, 2019

cyriltovena commented Aug 24, 2019

agones-bot commented Aug 24, 2019

agones-bot commented Aug 24, 2019

agones-bot commented Aug 24, 2019

Add allocations metrics #963

Add allocations metrics #963

Conversation

cyriltovena commented Jul 30, 2019 • edited Loading

This comment has been minimized.

This comment has been minimized.

agones-bot commented Jul 30, 2019

aLekSer commented Jul 30, 2019

agones-bot commented Jul 30, 2019

markmandel commented Jul 31, 2019

agones-bot commented Jul 31, 2019

roberthbailey commented Jul 31, 2019

agones-bot commented Jul 31, 2019

cyriltovena commented Jul 31, 2019

pooneh-m left a comment

Choose a reason for hiding this comment

pooneh-m Jul 31, 2019

Choose a reason for hiding this comment

cyriltovena Aug 23, 2019

Choose a reason for hiding this comment

markmandel commented Jul 31, 2019

cyriltovena commented Aug 5, 2019

markmandel commented Aug 14, 2019

agones-bot commented Aug 23, 2019

cyriltovena commented Aug 23, 2019

agones-bot commented Aug 23, 2019

markmandel commented Aug 23, 2019

cyriltovena commented Aug 24, 2019

cyriltovena commented Aug 24, 2019

cyriltovena commented Aug 24, 2019

agones-bot commented Aug 24, 2019

agones-bot commented Aug 24, 2019

agones-bot commented Aug 24, 2019

cyriltovena commented Jul 30, 2019 •

edited

Loading