add Date timestamp_format for clickhouse sink ( DateTime timestamp_format option already exist )

[dmitrysdm]

Grafana clickhouse plugin needs two timestamp fields:
Column:Date (EventDate)
Column:DateTime (DateTime) or Column:TimeStamp (UInt32).
https://github.com/Vertamedia/clickhouse-grafana
Column:DateTime can be created with special option:
timestamp_format = "rfc3339"
but what about Column:Date
https://clickhouse.tech/docs/en/data_types/date/
i think needed one more timestamp_format, because in clickhouse exist Date type as well as DateTime.

[dmitrysdm]

I am stiil try to understand how vector are working,
and now i think coercer transform sufficient for me.

[Hoverbear]

@dmitrysdm Did it work out? :)

[dmitrysdm]

I prepared a config, but did not check with real database:

[sources.stdin]
    type = "stdin"

[transforms.add_time_transform]
    type = "add_fields"
    inputs = ["stdin"]
    fields.EventTime ="{{timestamp}}"

[transforms.add_date_transform]
    type = "concat"
    inputs = ["add_time_transform"]
    items = ["EventTime[..10]"]
    target = "EventDate"

[transforms.json_parse_transform]
  inputs       = ["add_date_transform"]
  type         = "json_parser"
  field        = "message"

[sinks.stdout]
    inputs = ["json_parse_transform"]
    type = "console"
    encoding = "json"

@dmitrysdm That config should work. But with Vector's strftime specifiers in templates it could be made even more concise:

 [sources.stdin]
    type = "stdin"
< 
< [transforms.add_time_transform]
<     type = "add_fields"
<     inputs = ["stdin"]
<     fields.EventTime ="{{timestamp}}"

[transforms.add_date_transform]
<     type = "concat"
<     inputs = ["add_time_transform"]
<     items = ["EventTime[..10]"]
<     target = "EventDate"
>     type = "add_fields"
>     inputs = ["stdin"]
>     fields.EventTime = "%Y-%m-%d %H:%M:%S"
>     fields.EventDate = "%Y-%m-%d"

[transforms.json_parse_transform]
  inputs       = ["add_date_transform"]
  type         = "json_parser"
  field        = "message"

[sinks.stdout]
    inputs = ["json_parse_transform"]
    type = "console"
    encoding = "json"

I tested it with clickhouse sink using the following definition of the sink:

[sinks.clickhouse]
  type = "clickhouse"
  inputs = ["add_date_transform"]

  host = "http://localhost:8123"
  table = "service_log"
  encoding.only_fields = ["EventTime", "EventDate", "host"]

and the table created using the following query:

CREATE TABLE
service_log (EventTime DateTime, EventDate Date, host String)
ENGINE = MergeTree()
PARTITION BY EventDate
ORDER BY (host, EventTime);

and it worked:

854eb3edb59f :) select * from service_log limit 5

SELECT *
FROM service_log
LIMIT 5

┌───────────EventTime─┬──EventDate─┬─host───────┐
│ 2020-03-20 19:31:25 │ 2020-03-20 │ 172.20.0.3 │
│ 2020-03-20 19:31:26 │ 2020-03-20 │ 172.20.0.3 │
│ 2020-03-20 19:31:27 │ 2020-03-20 │ 172.20.0.3 │
│ 2020-03-20 19:31:28 │ 2020-03-20 │ 172.20.0.3 │
│ 2020-03-20 19:31:29 │ 2020-03-20 │ 172.20.0.3 │
└─────────────────────┴────────────┴────────────┘

5 rows in set. Elapsed: 0.002 sec. 

[dmitrysdm]

seems like solution, thank you, and thank you for testing it.