Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to populate timestamp column #116

Open
bschooly opened this issue Nov 5, 2021 · 2 comments
Open

How to populate timestamp column #116

bschooly opened this issue Nov 5, 2021 · 2 comments

Comments

@bschooly
Copy link

bschooly commented Nov 5, 2021

What is the proper/recommended way to populate the timestamp?

If have:

<source>
  @type tail
  path /tmp/example.log
  pos_file /var/log/td-agent/example.log.pos
  tag example.*
 <parse>
        @type json
        keep_time_key true
  </parse>
</source>
...

<match example.**>
  @type sql
  host localhost
  port 3306
  database example
  adapter mysql2
  username example
  password XXXXXX
  socket /var/lib/mysql/mysql.sock
  remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql

  <table>
    table logs
    column_mapping 'time,host,size,method'
    # This is the default table because it has no "pattern" argument in <table>
    # The logic is such that if all non-default <table> blocks
    # do not match, the default one is chosen.
    # The default table is required.
  </table>
</match>

The above works but I couldn't get the time value reformatted. How do I do this? Also how do I use the implicit timestamp generated by fluentd? Ie. the timestamp that is not part of the json payload.

Here is a input record:

{"time":"1362020444","host":"localhost","size":777,"method":"PUT"}
This works. I get exacly the data shown above in the db but I want to know if I can reformat the time and how to populate the timestamp if it is not part of the json data.

Thanks in advance for a reply...

Bill

@husamrahman
Copy link

Any luck on the issue? Facing the same thing at the moment

@untsakas
Copy link

untsakas commented Mar 3, 2022

Maybe you can use filter

<filter app.*>
  @type record_transformer
  enable_ruby true
  <record>
    @timestamp ${time.strftime('%Y-%m-%dT%H:%M:%S.%L')}
  </record>
</filter>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants