CSV Agent serialize doesn't output proper CSV file, please provide example

[radry]

When using the CSV Agent the output is never a correctly formated CSV file.
Maybe I'm misunderstanding the CSV Agent but what's the point of it if I can't generate a properly formated CSV File with it?

No matter what I feed the CSV agent, be it JSON or Object, the output is unusable and incorrect CSV. I tried sending events from either Event Formatter Agent and Json Parse Agent to it.

Sample Data Input:

From Json Parse Agent:

{
  "message": {
    "1707838936": {
      "title": "foo",
      "price": "111"
    },
    "1707849865": {
      "title": "foo",
      "price": "111"
    }
  }
}

OR
From Event Formatting Agent:

{
  "message": "{\"1707838936\":{\"title\":\"foo\",\"price\":\"111.0\"},\"1707849865\":{\"title\":\"foo\",\"price\":\"111\"}}"
}

Sample Output of CSV Agent:

{
  "data": "\"message\"\n\"{\"\"1707838936\"\"=>{\"\"title\"\"=>\"\"foo\"\", \"\"price\"\"=>\"\"111.0\"\"}, \"\"1707849865\"\"=>{\"\"title\"\"=>\"\"foo\"\", \"\"price\"\"=>\"\"111\"\"}}\"\n"
}

OR

{
  "data": "\"{\"\"1707838936\"\":{\"\"title\"\":\"\"foo\"\",\"\"price\"\":\"\"111.0\"\"},\"\"1707849865\"\":{\"\"title\"\":\"\"foo\"\",\"\"price\"\":\"\"111\"\"}}\"\n"
}

Expected Result:

A proper CSV like this (generate with https://www.convertcsv.com/json-to-csv.htm):

_key,title,price
1707838936,"foo",111.0
1707849865,"foo",111

[Unending]

I don't think the CSV Agent works the way you want it to.

When using the serialise function it takes in multiple events and combines them into a single CSV format. It doesn't remap objects from a single JSON to a single CSV and it doesn't actually create files.

Example:
Send the following two separate events:
{ "key": "1707838936", "title": "foo", "price": "111" }
{ "key": "1707849865", "title": "bar", "price": "222" }

With default CSV Agent options + "mode": "serialize" and "data_path": "$" This will result in the payload:

{
  "data": "\"key\",\"title\",\"price\"\n\"1707838936\",\"foo\",\"111\"\n\"1707849865\",\"bar\",\"222\"\n"
}

I you then send that event to the Local File Agent it will write it to disk as:

"key","title","price"
"1707838936","foo","111"
"1707849865","bar","222"

[radry]

I wanted to send it to an FTP Agent, not Local File Agent. However the FTP Agent writes a nonsense CSV File.

Does that mean the CSV Agent absolutely needs the local file Agent to translate the serialized string into an actual CSV compliant File? FTP Agent can't do it aparently.

Also, I can't send separate events to the CSV Agent because I want to read data from a Key Value Store Agent and then send it as one event containing all the data.

Essentially what I want is: Several Agents send Events to the Key Value Store Agent. At certain intervals the collected Data is written into a CSV File and published to an FTP.

As a workaround I currently just publish a JSON File, which works with the current agent setup.

[Unending]

The CSV agent just creates an event with a json name value pair, which contains just a string of characters in it. It doesn't matter whether you send it to the Local File Agent or the Ftpsite Agent (or any other agent), and just writes the data to file. There is no further "translation" happening.

Previous example (escaped JSON event):
{ "data": "\"key\",\"title\",\"price\"\n\"1707838936\",\"foo\",\"111\"\n\"1707849865\",\"bar\",\"222\"\n" }

When writing the content of "data" that would be:
"key","title","price"\n"1707838936","foo","111"\n"1707849865","bar","222"

Which is the same as

"key","title","price"
"1707838936","foo","111"
"1707849865","bar","222"

Which is a valid CSV file.