feat(kumactl) add install loki for log aggregation #820

xbauquet · 2020-06-10T12:52:30Z

Summary

Add the Loki stack to kumactl install allowing to see logs
directly in Grafana. The yaml for loki was inspired by:
https://grafana.github.io/loki/charts

Full changelog

Add a new install command to kubectl:
kubectl install logging
Upgrate Grafana version to 7.0.3 that accepts loki as a
data source.

Issues resolved

#819

Documentation

Documentation PR: kumahq/kuma-website#222

CLAassistant · 2020-06-10T12:52:35Z

All committers have signed the CLA.

xbauquet · 2020-06-11T15:43:08Z

Follow up on the meeting of the 2020/06/10:

Question

Can we use Loki and Grafana to aggregate and visualise TrafficLogs ?

Response

Here are the steps of my test:

Step 1: Access logs in default format

My first try was to make the data-planes send the Access Logs to stdout

logging:
    defaultBackend: file
    backends:
      - name: file
        type: file
        conf:
          path: /dev/stdout

That worked well I get the Access Logs with the default format.

Step 2: Access logs in custom format

My second test was to use a custom format for my Access Logs

logging:
    defaultBackend: file
    backends:
      - name: file
        type: file
        format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%"}'
        conf:
          path: /dev/stdout

When I use the format I loose the Access Logs

Step 3: Back to the default format

This is where this test becomes weird.
I removed the custom format to be back with the configuration in Step 1.
As a result my Access Logs are back in default format BUT my Access Logs with the custom format from the Step 2 are displayed concatenated in one single line.

Conclusion

It looks like using a custom format with the type: file makes the data-planes concatenate all the Access Logs until a new Log is generated.
I don't know where to look for that specific problem, but this could be a little bug.
If someone has any idea ?!!

jakubdyszkiewicz · 2020-06-12T11:14:51Z

Try to put the new line in the format. Something like this:

logging:
    defaultBackend: file
    backends:
      - name: file
        type: file
        format: '{"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%"}
'
        conf:
          path: /dev/stdout

or

logging:
    defaultBackend: file
    backends:
      - name: file
        type: file
        format: | {"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%"}

        conf:
          path: /dev/stdout

xbauquet · 2020-06-12T11:36:58Z

I'll try that thank you.

xbauquet · 2020-06-12T11:45:27Z

  logging:
    defaultBackend: file
    backends:
      - name: file
        format: |
          {"start_time": "%START_TIME%", "source": "%KUMA_SOURCE_SERVICE%", "destination": "%KUMA_DESTINATION_SERVICE%"}

        type: file
        conf:
          path: /dev/stdout

This works (not the other option).
This is a good work around but I still think we should fix it when possible.

jakubdyszkiewicz · 2020-06-15T09:46:49Z

To be honest, I'll be willing to add the new line automatically in http_access_log_configurer and in network_access_log_configurer.

I see we do it in default format. I'd suggest a change so instead of doing it only for default format, we add it for all formats, even ones provided by the user.

xbauquet · 2020-06-15T10:57:08Z

I think adding automatically a new line for all formats is a good idea.
Can you tell me where is the code taking care of that? I can do the modification if you want.

jakubdyszkiewicz · 2020-06-15T11:39:54Z

I think convertLoggingBackend is a place. Add a new line before
format, err := accesslog.ParseFormat(formatString)

Also I think that in result of this change, the line
kuma/app/kuma-dp/pkg/dataplane/accesslogs/sender.go:33 is not needed anymore.

With the TrafficLog policy, using a custom format with the `type: file` makes the data-planes concatenate all the Access Logs until a new Log is generated. This bug is not observed when omiting the format. This is fixed by addind a new line after each Access Log. For more informations see kumahq#820

tharun208 · 2020-06-16T10:03:13Z

@xbauquet , Imports are not ordered. try to run make imports.

This is the documentation for the kuma#820 PR adding the yamls to install Loki and aggregate logs. * PR: https://github.com/Kong/kuma/pull/820/

xbauquet · 2020-06-18T11:30:53Z

Documentation PR: kumahq/kuma-website#222

nickolaev

Overall I am OK with this one, especially now that we have docs alongside.
One small question - how can that be replicated for universal? we don't have the nice installation with kumactl there, yet people may want to also have access to this feature? Or maybe this is a matter of updating docs?

nickolaev · 2020-06-18T14:40:01Z

app/kuma-dp/pkg/dataplane/accesslogs/sender.go

@@ -30,7 +30,7 @@ func (s *sender) Connect() error {
 }

 func (s *sender) Send(record string) error {
-	_, err := s.conn.Write(append([]byte(record), byte('\n')))
+	_, err := s.conn.Write([]byte(record))


@jakubdyszkiewicz what do you think?

xbauquet · 2020-06-18T14:50:43Z

I just looked at the Loki documentation and it is possible to install it without Kubernetes so I guess I could do a bit of doc on how to install it for universal or at least add some links to help people installing it.

nickolaev · 2020-06-18T14:58:52Z

Can you also:

merge master
run make check, make test, make integration and make test/e2e (all separate as some of these are pretty heavy). The latter two assume you have Docker running and accissebal from your account. Please make sure you fix tests related to logging. If something else fails, reach out over slack to check it with us.

Thanks a lot for your contributions!

With the TrafficLog policy, using a custom format with the `type: file` makes the data-planes concatenate all the Access Logs until a new Log is generated. This bug is not observed when omiting the format. This is fixed by addind a new line after each Access Log. For more informations see kumahq#820

xbauquet · 2020-06-19T08:03:44Z

I opened a issue #841 because I have a problem when trying to fix tests.

Add the Loki stack to `kumactl install` allowing to see logs directly in Grafana. The yaml for loki was inspired by: `https://grafana.github.io/loki/charts` * Add a new install command to kubectl: `kubectl install logging` * Upgrate Grafana version to 7.0.3 that accepts loki as a data source.

With the TrafficLog policy, using a custom format with the `type: file` makes the data-planes concatenate all the Access Logs until a new Log is generated. This bug is not observed when omiting the format. This is fixed by addind a new line after each Access Log. For more informations see kumahq#820

xbauquet · 2020-06-22T14:09:47Z

@nickolaev I corrected all the tests I could. I still see a few problems in Circle CI but I don't understand the problem.

nickolaev · 2020-06-22T15:22:38Z

@xbauquet CI is sometimes flaky. I re-run the failing tests and all is fine now

nickolaev

I am willing to move this one, once the docs get moved to 0.6.0. Thanks for the contribution!

nickolaev · 2020-07-06T11:10:03Z

app/kumactl/data/install/k8s/metrics/grafana/grafana.yaml

@@ -263,7 +263,7 @@ spec:
        runAsUser: 472
      containers:
        - name: grafana
-          image: "grafana/grafana:6.6.0"
+          image: "grafana/grafana:7.0.3"


Please bump the version to 7.0.5.

nickolaev

Please merge master and do bump grafana to 7.0.5.
We are merging this one!

…logging

xbauquet requested a review from a team June 10, 2020 12:52

xbauquet force-pushed the feat/kumactl-install-logging branch from e7752c4 to 6a406c4 Compare June 15, 2020 10:46

xbauquet force-pushed the feat/kumactl-install-logging branch from 3eb51b7 to a8567bf Compare June 16, 2020 07:33

xbauquet referenced this pull request in xbauquet/kuma-website Jun 18, 2020

docs(log policy) visualization of logs in Grafana

3a03508

This is the documentation for the kuma#820 PR adding the yamls to install Loki and aggregate logs. * PR: https://github.com/Kong/kuma/pull/820/

xbauquet mentioned this pull request Jun 18, 2020

docs(log policy) visualization of logs in Grafana kumahq/kuma-website#222

Merged

nickolaev approved these changes Jun 18, 2020

View reviewed changes

xbauquet force-pushed the feat/kumactl-install-logging branch from e2aa4ed to 316ed0c Compare June 18, 2020 15:08

xbauquet mentioned this pull request Jun 19, 2020

Problem fixing logging policy tests #841

Closed

xbauquet added 5 commits June 20, 2020 17:06

tests(kumactl) tests for kumactl install logging

4422335

refactor(kumactl) order imports

56f522a

tests(data-planes) fix access logs tests

19cb442

xbauquet force-pushed the feat/kumactl-install-logging branch from 316ed0c to 19cb442 Compare June 20, 2020 15:06

nickolaev approved these changes Jul 6, 2020

View reviewed changes

nickolaev reviewed Jul 6, 2020

View reviewed changes

nickolaev approved these changes Jul 6, 2020

View reviewed changes

xbauquet added 2 commits July 6, 2020 14:01

feat(kumactl): bump grafana version to 7.0.5

abd5171

Merge remote-tracking branch 'kuma/master' into feat/kumactl-install-…

001c29b

…logging

nickolaev merged commit 70bfda1 into kumahq:master Jul 6, 2020

nickolaev mentioned this pull request Jul 13, 2020

Add log aggregation to kumactl install #819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kumactl) add install loki for log aggregation #820

feat(kumactl) add install loki for log aggregation #820

xbauquet commented Jun 10, 2020 •

edited by nickolaev

CLAassistant commented Jun 10, 2020 •

edited

xbauquet commented Jun 11, 2020

jakubdyszkiewicz commented Jun 12, 2020

xbauquet commented Jun 12, 2020

xbauquet commented Jun 12, 2020 •

edited

jakubdyszkiewicz commented Jun 15, 2020

xbauquet commented Jun 15, 2020

jakubdyszkiewicz commented Jun 15, 2020

tharun208 commented Jun 16, 2020

xbauquet commented Jun 18, 2020

nickolaev left a comment

nickolaev Jun 18, 2020

xbauquet commented Jun 18, 2020

nickolaev commented Jun 18, 2020

xbauquet commented Jun 19, 2020

xbauquet commented Jun 22, 2020 •

edited

nickolaev commented Jun 22, 2020

nickolaev left a comment

nickolaev Jul 6, 2020

nickolaev left a comment

feat(kumactl) add install loki for log aggregation #820

feat(kumactl) add install loki for log aggregation #820

Conversation

xbauquet commented Jun 10, 2020 • edited by nickolaev

Summary

Full changelog

Issues resolved

Documentation

CLAassistant commented Jun 10, 2020 • edited

xbauquet commented Jun 11, 2020

Question

Response

Step 1: Access logs in default format

Step 2: Access logs in custom format

Step 3: Back to the default format

Conclusion

jakubdyszkiewicz commented Jun 12, 2020

xbauquet commented Jun 12, 2020

xbauquet commented Jun 12, 2020 • edited

jakubdyszkiewicz commented Jun 15, 2020

xbauquet commented Jun 15, 2020

jakubdyszkiewicz commented Jun 15, 2020

tharun208 commented Jun 16, 2020

xbauquet commented Jun 18, 2020

nickolaev left a comment

Choose a reason for hiding this comment

nickolaev Jun 18, 2020

Choose a reason for hiding this comment

xbauquet commented Jun 18, 2020

nickolaev commented Jun 18, 2020

xbauquet commented Jun 19, 2020

xbauquet commented Jun 22, 2020 • edited

nickolaev commented Jun 22, 2020

nickolaev left a comment

Choose a reason for hiding this comment

nickolaev Jul 6, 2020

Choose a reason for hiding this comment

nickolaev left a comment

Choose a reason for hiding this comment

xbauquet commented Jun 10, 2020 •

edited by nickolaev

CLAassistant commented Jun 10, 2020 •

edited

xbauquet commented Jun 12, 2020 •

edited

xbauquet commented Jun 22, 2020 •

edited