vdk-logging-json: Escape newlines in log messages

[gageorgiev]

The message in logs formatted by the vdk-logging-json now
have any newline characters replaced by 4 spaces. This change
is made due to how logs are currently parsed by separating
at newline characters.

Testing done: local, pipelines
Before installing plugin:

2021-08-23 10:29:26,908=1629703767[VDK] example-process-in-star-schema [DEBUG] taurus.vdk.builtin_plugins.con    managed_cursor.py:29   execute         [OpId:1629703766-571a7d-67d79b]- Executing query:
-- job_name: example-process-in-star-schema
-- op_id: 1629703766
CREATE VIEW IF NOT EXISTS super_collider.vw_gageorgiev_dim_users AS
SELECT id, name, username, email FROM history_staging.sc_code_samples__users

After installing plugin:

{"@timestamp":"2021-08-23T10:26:39.355Z","created":"1629703599.355760","jobname":"example-process-in-star-schema","level":"DEBUG","modulename":"taurus.vdk.builtin_plugins.connection.impl.wrapped_connection","filename":"managed_cursor.py","lineno":"29","funcname":"execute","attemptid":"1629703599-8dcccb-def749","message":"Executing query:    -- job_name: example-process-in-star-schema    -- op_id: 1629703599    CREATE VIEW IF NOT EXISTS super_collider.vw_gageorgiev_dim_users AS    SELECT id, name, username, email FROM history_staging.sc_code_samples__users    "}

Signed-off-by: gageorgiev gageorgiev@vmware.com

[antoniivanov]

Is this the recommended way to handle new lines with JSON parser? That would make multi-line logs harder to consume. Imagine 50 line stack trace on a single line. There should be a way to escape new line but for Kibana/log insight to visualize it.

[antoniivanov]

vdk-core: Replace newlines ...

The component name is the plugin name :

vdk-logging-json: Replace newlines ..

[gageorgiev]

Is this the recommended way to handle new lines with JSON parser? That would make multi-line logs harder to consume. Imagine 50 line stack trace on a single line. There should be a way to escape new line but for Kibana/log insight to visualize it.

Dug around a bit and it seems like the recommended way is to escape the backwards slash, i.e. replace \n with \\n. Not sure exactly how that'll get parsed and then visualized in the end, but I'm willing to experiment.

[antoniivanov]

Dug around a bit and it seems like the recommended way is to escape the backwards slash, i.e. replace \n with \n. Not sure exactly how that'll get parsed and then visualized in the end, but I'm willing to experiment.

Can you please add link as a comment in the code to this recommendation.

We should consider installing and configuring either locally (minikube) or in our CICD environment to use fluentd. I did install it now (it's easy with helm install) but have no idea how to configure it in any meaningful way to test something .