Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(external docs): Fix parsing-csv-logs-with-lua.md example for postgres > 13 #20513

Merged
merged 2 commits into from
May 29, 2024
Merged

fix(external docs): Fix parsing-csv-logs-with-lua.md example for postgres > 13 #20513

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b727b86
Fix parsing-csv-logs-with-lua.md example for postgres > 13
t0k4rt May 17, 2024
8d40486
remove whitespace
jszwedko May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
17 changes: 12 additions & 5 deletions website/content/en/guides/advanced/parsing-csv-logs-with-lua.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,15 +155,19 @@ the whole transform:
"query_pos",
"location",
"application_name",
-- available only in postgres > 13, to remove for postgres <= 13
"backend_type",
"leader_pid",
"query_id"
}
"""
hooks.process = """
function (event, emit)
fields = csv.openstring(event.log.message):lines()() -- parse the `message` field
event.log.message = nil -- drop the `message` field

for column, value in ipairs(fields) do -- iterate over CSV columns
column_name = column_names[column] -- get column name
for column, column_name in ipairs(column_names) do -- iterate over column names
value = fields[column] -- get field value
event.log[column_name] = value -- set the corresponding field in the event
end

Expand All @@ -176,16 +180,17 @@ the whole transform:
Trying to run `vector --config vector.toml` with the same input file results in structured events being output:

```json
{"application_name":"","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"Future log output will go to log destination \"csvlog\".","host":"localhost","internal_query":"","internal_query_pos":"","location":"","log_time":"2020-04-09 12:48:49.661 UTC","message":"ending log output to stderr","process_id":"1","query":"","query_pos":"","session_id":"localhost.1","session_line_num":"1","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
{"application_name":"","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"","host":"localhost","internal_query":"","internal_query_pos":"","location":"","log_time":"2020-04-09 12:48:49.669 UTC","message":"database system was shut down at 2020-04-09 12:48:25 UTC","process_id":"27","query":"","query_pos":"","session_id":"localhost.1b","session_line_num":"1","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
{"application_name":"","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"","host":"localhost","internal_query":"","internal_query_pos":"","location":"","log_time":"2020-04-09 12:48:49.683 UTC","message":"database system is ready to accept connections","process_id":"1","query":"","query_pos":"","session_id":"localhost.1","session_line_num":"2","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
{"application_name":"","backend_type":"not initialized","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"Future log output will go to log destination \"csvlog\".","host":"localhost","internal_query":"","internal_query_pos":"","leader_pid":"","location":"","log_time":"2020-04-09 12:48:49.661 UTC","message":"ending log output to stderr","process_id":"1","query":"","query_id":"0","query_pos":"","session_id":"localhost.1","session_line_num":"1","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
{"application_name":"","backend_type":"not initialized","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"","host":"localhost","internal_query":"","internal_query_pos":"","leader_pid":"","location":"","log_time":"2020-04-09 12:48:49.669 UTC","message":"database system was shut down at 2020-04-09 12:48:25 UTC","process_id":"27","query":"","query_id":"0","query_pos":"","session_id":"localhost.1b","session_line_num":"1","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
{"application_name":"","backend_type":"not initialized","command_tag":"","connection_from":"","context":"","database_name":"","detail":"","error_severity":"LOG","file":"log.csv","hint":"","host":"localhost","internal_query":"","internal_query_pos":"","leader_pid":"","location":"","log_time":"2020-04-09 12:48:49.683 UTC","message":"database system is ready to accept connections","process_id":"1","query":"","query_id":"0","query_pos":"","session_id":"localhost.1","session_line_num":"2","session_start_time":"2020-04-09 12:48:49 UTC","sql_state_code":"00000","timestamp":"2020-04-09T19:49:07Z","transaction_id":"0","user_name":"","virtual_transaction_id":""}
```

Or, applying pretty formatting to one of the output events:

```json
{
"application_name": "",
"backend_type":"not initialized",
"command_tag": "",
"connection_from": "",
"context": "",
Expand All @@ -197,11 +202,13 @@ Or, applying pretty formatting to one of the output events:
"host": "localhost",
"internal_query": "",
"internal_query_pos": "",
"leader_pid":"",
"location": "",
"log_time": "2020-04-09 12:48:49.661 UTC",
"message": "ending log output to stderr",
"process_id": "1",
"query": "",
"query_id":"0",
"query_pos": "",
"session_id": "localhost.1",
"session_line_num": "1",
Expand Down