Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for persisting logs in 'emptyDir' volume attached to agones controller. #620

Merged
merged 2 commits into from Feb 26, 2019

Conversation

@jkowalski
Copy link
Collaborator

commented Feb 22, 2019

(This creates a secondary place to look at logs for debugging when controller is logging too fast and fluentd can't sync to stackdriver fast enough)

The logs are saved in 100MB chunks under /home/agones/logs up to a specified limit (10000 MB by default).

Example:

$  kubectl exec -n agones-system -it agones-controller-7d85b5c996-8c22b -- ls -lh /home/agones/logs
total 757764
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:22 agones-controller-20190222_212001-2019-02-22T21-22-37.499.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:26 agones-controller-20190222_212001-2019-02-22T21-26-31.904.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-01.546.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-42.934.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:28 agones-controller-20190222_212001-2019-02-22T21-28-46.574.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:32 agones-controller-20190222_212001-2019-02-22T21-32-31.224.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:33 agones-controller-20190222_212001-2019-02-22T21-33-30.350.log
-rw-r--r--    1 agones   agones     40.0M Feb 22 21:41 agones-controller-20190222_212001.log
@agones-bot

This comment has been minimized.

Copy link
Collaborator

commented Feb 22, 2019

Build Failed 😱

Build Id: 7cabd1cf-51be-4a31-af82-6e3f2564cff7

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

Added support for persisting logs in 'emptyDir' volume attached to ag…
…ones controller.

The logs are saved in 100MB chunks under `/home/agones/logs` up to a specified limit (10000 MB by default).

Example:

```shell
$ ls -lh /home/agones/logs
total 757764
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:22 agones-controller-20190222_212001-2019-02-22T21-22-37.499.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:26 agones-controller-20190222_212001-2019-02-22T21-26-31.904.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-01.546.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:27 agones-controller-20190222_212001-2019-02-22T21-27-42.934.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:28 agones-controller-20190222_212001-2019-02-22T21-28-46.574.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:32 agones-controller-20190222_212001-2019-02-22T21-32-31.224.log
-rw-r--r--    1 agones   agones    100.0M Feb 22 21:33 agones-controller-20190222_212001-2019-02-22T21-33-30.350.log
-rw-r--r--    1 agones   agones     40.0M Feb 22 21:41 agones-controller-20190222_212001.log

```

@jkowalski jkowalski force-pushed the jkowalski:persistent-logs branch from 8d753bc to 8285096 Feb 22, 2019

@agones-bot

This comment has been minimized.

Copy link
Collaborator

commented Feb 22, 2019

Build Succeeded 👏

Build Id: 999cd8c7-3476-4ece-9881-eb20aba98a66

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/620/head:pr_620 && git checkout pr_620
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.9.0-8285096

@jkowalski jkowalski marked this pull request as ready for review Feb 22, 2019

@cyriltovena
Copy link
Collaborator

left a comment

LGTM

@markmandel
Copy link
Collaborator

left a comment

'couple o' thoughts 😄

@@ -49,6 +49,8 @@ agones:
operator: Exists
generateTLS: true
safeToEvict: false
persistentLogs: true

This comment has been minimized.

Copy link
@markmandel

markmandel Feb 24, 2019

Collaborator

Thought: Should this be on by default? I'm thinking most people won't be doing load tests, and stackdriver/fluentd are likely fast enough. WDYT?

This comment has been minimized.

Copy link
@jkowalski

jkowalski Feb 25, 2019

Author Collaborator

it's not just load testing, we currently emit enough logs to overwhelm fluentd. For any fleets of significant size there are dropped messages.

This comment has been minimized.

Copy link
@markmandel

markmandel Feb 25, 2019

Collaborator

Ah. In that case, yes - let's keep this on by default. I didn't realise we were getting dropped messages, that's unfortunate.

Sounds like something we should push upstream? (or should we be logging less?)

But otherwise, this makes sense, and looks good to me.

Only other thing I can think of - should we add a section in (to be released in 0.9.0) https://agones.dev/site/docs/guides/troubleshooting/ to let people know that this is an option?

This comment has been minimized.

Copy link
@jkowalski

jkowalski Feb 26, 2019

Author Collaborator

sure, I'll submit this one now and make next PR with doc change and ability to use PVC for durable logs

@agones-bot

This comment has been minimized.

Copy link
Collaborator

commented Feb 26, 2019

Build Succeeded 👏

Build Id: 7b48d1a6-aa76-402d-a719-8a8db012f765

The following development artifacts have been built, and will exist for the next 30 days:

A preview of the website (the last 30 builds are retained):

To install this version:

  • git fetch https://github.com/GoogleCloudPlatform/agones.git pull/620/head:pr_620 && git checkout pr_620
  • helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.9.0-7b7ec63

@jkowalski jkowalski merged commit d46947c into googleforgames:master Feb 26, 2019

2 checks passed

agones-bot Build 7b48d1a6-aa76-402d-a719-8a8db012f765 succeeded
Details
cla/google All necessary CLAs are signed

@markmandel markmandel added this to the 0.9.0 milestone Mar 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.