Skip to content

control-service: Format logs in JSON - #308

Merged
gabrielgeorgiev1 merged 2 commits into
mainfrom
person/gageorgiev/cs-json
Sep 28, 2021
Merged

control-service: Format logs in JSON#308
gabrielgeorgiev1 merged 2 commits into
mainfrom
person/gageorgiev/cs-json

Conversation

@gabrielgeorgiev1

@gabrielgeorgiev1 gabrielgeorgiev1 commented Sep 28, 2021

Copy link
Copy Markdown
Contributor

Control Service logs must be able to be formatted in JSON using
an environment variable due to the importance of structured logging
capability. This change adds an encoder which formats logs in JSON
for an instance of the Control Service with the LOGGING_FORMAT
environment variable set to JSON.
This change allows us to retain the previous log config while
still allowing us the freedom to configure a logger to format logs
to JSON. Logging configuration should be much more in depth,
however it would require changes outside of the scope of the
current story.

Testing done: ran CS with the encoder set for local logs and the
LOGGING_FORMAT env variable set to JSON and observed the logs,
also with other values and saw that the logs were formatted in
the default format.

Signed-off-by: gageorgiev gageorgiev@vmware.com

@gabrielgeorgiev1 gabrielgeorgiev1 changed the title control-service: Format logs in JSON [DRAFT] control-service: Format logs in JSON Sep 28, 2021
@gabrielgeorgiev1
gabrielgeorgiev1 force-pushed the person/gageorgiev/cs-json branch from dd9f6b2 to ad13de4 Compare September 28, 2021 13:07
@gabrielgeorgiev1 gabrielgeorgiev1 changed the title [DRAFT] control-service: Format logs in JSON control-service: Format logs in JSON Sep 28, 2021
@gabrielgeorgiev1
gabrielgeorgiev1 force-pushed the person/gageorgiev/cs-json branch from ad13de4 to f1e7a07 Compare September 28, 2021 13:39
@antoniivanov

antoniivanov commented Sep 28, 2021

Copy link
Copy Markdown
Contributor

In the commit message, the problem statement can be generalized. We are not adding some specific support for Kibana. What we are doing would make it better for Kibana users.
But not just only that we want logs to be visible in Kibana. We want to enable structured logging in Control Service (https://stackify.com/what-is-structured-logging-and-why-developers-need-it/) which enables a lot of benefits for developers (explained in the article I linked - I picked the first I googled - there a lots.). It makes it easier to filter and browse in modern logging tools (like Kibana but also other like log intelligence or Loggly and so on).

Please fix it since some reading it may think this is a very specific feature and it is not.

@antoniivanov

Copy link
Copy Markdown
Contributor

I almost forgot :

From https://github.com/vmware/versatile-data-kit/blob/main/CONTRIBUTING.md#how-to-submit-and-merge

projects/*/CHANGELOG.md Next version section should be updated accordingly.

/offtopic:
Maybe We need to think how to automate changelog updates. I forget often to do it.
Maybe can automatically create it from the commit message. But then we'd need more structured commit messages.
/end offtopic.

@gabrielgeorgiev1
gabrielgeorgiev1 force-pushed the person/gageorgiev/cs-json branch from f1e7a07 to 8739f42 Compare September 28, 2021 13:58
Control Service logs must be formatted in JSON to be able to be
parsed correctly in Kibana. This change adds an encoder which
formats logs in JSON for an instance of the Control Service with
the LOGGING_SYS_LOG_HOST env variable defined.

Testing done: ran CS with the encoder set for local logs

Signed-off-by: gageorgiev <gageorgiev@vmware.com>
@gabrielgeorgiev1
gabrielgeorgiev1 force-pushed the person/gageorgiev/cs-json branch from 8739f42 to 68e5754 Compare September 28, 2021 14:01
@gabrielgeorgiev1
gabrielgeorgiev1 enabled auto-merge (squash) September 28, 2021 18:16
@gabrielgeorgiev1
gabrielgeorgiev1 merged commit 3474630 into main Sep 28, 2021
@gabrielgeorgiev1
gabrielgeorgiev1 deleted the person/gageorgiev/cs-json branch September 28, 2021 18:30
tpalashki pushed a commit that referenced this pull request Sep 29, 2021
Control Service logs must be able to be formatted in JSON using
an environment variable due to the importance of structured logging
capability. This change adds an encoder which formats logs in JSON
for an instance of the Control Service with the LOGGING_FORMAT
environment variable set to JSON.
This change lets us to retain the previous log config while still
allowing us the freedom to configure a logger to format logs
to JSON. Logging configuration should be much more in depth,
however it would require changes outside of the scope of the
current story.

Testing done: ran CS with the encoder set for local logs and the
LOGGING_FORMAT env variable set to JSON and observed the logs,
also with other values and saw that the logs were formatted in
the default format.

Signed-off-by: gageorgiev <gageorgiev@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants