Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DateIndexNameProcessor does not support unix epoch format #26890

Closed
iksnalybok opened this issue Oct 5, 2017 · 1 comment

Comments

Projects
None yet
3 participants
@iksnalybok
Copy link
Contributor

commented Oct 5, 2017

Elasticsearch+Filebeat 6.0.0-rc1.

DateIndexNameProcessor expects the date to be a String. At line 64: String date = ingestDocument.getFieldValue(field, String.class);.

The log4j2 json logger outputs timestamp as unix_ms epoch (e.g. {"timeMillis":1507099254201,…}).
I want the log to be indexed in a daily index, on the day it was generated (vs. ingested), so I tell filebeat to use an elasticsearch date_index_name pipeline. It fails with the following exception:

Caused by: java.lang.IllegalArgumentException: field [json.timeMillis] of type [java.lang.Long] cannot be cast to [java.lang.String]
at o.e.ingest.IngestDocument.cast(IngestDocument.java:542)
at o.e.ingest.IngestDocument.getFieldValue(IngestDocument.java:107)
at o.e.ingest.common.DateIndexNameProcessor.execute(DateIndexNameProcessor.java:64)
at o.e.ingest.CompoundProcessor.execute(CompoundProcessor.java:100)

Reproducer:

  1. Start elasticsearch (unzip and start) and create the pipeline:
PUT /_ingest/pipeline/bugTimestampPipeline
  {
    "description": "bugTimestampPipeline",
    "processors" : [
      {
        "date_index_name" : {
          "field"             : "json.timeMillis",
          "date_formats"      : [ "UNIX_MS" ],
          "index_name_prefix" : "myDailyIndex-",
          "date_rounding"     : "d",
          "index_name_format" : "yyyy.MM.dd"
        }
      }
    ]
  }
  1. Create the filebeat configuration, and run filebeat --path.config confBugTimestamp -c filebeat-bugTimestamp.yml:

confBugTimestamp/fields.yml: a copy of <filebeatDir>/fields.yml

confBugTimestamp/filebeat-bugTimestamp.yml

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: bugTimestampPipeline

filebeat.prospectors:
- type: log
  enabled: true
  paths:
  - logsBugTimestamp/*.json.log
  json.keys_under_root: false
  json.add_error_key: true
  json.message_key: message
  close_inactive: 24h
  close_renamed: true  # because Windows (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-renamed)
  close_removed: true  # because Windows (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-removed)
  1. copy bugTimestamp.json.log into logsBugTimestamp/

bugTimestamp.json.log

{"timeMillis":1507099254201,"level":"INFO","message":"foobar"}
@martijnvg

This comment has been minimized.

Copy link
Member

commented Oct 6, 2017

@iksnalybok Thanks for sharing this bug! I'll fix this soon.

@martijnvg martijnvg closed this in bba7020 Oct 10, 2017

martijnvg added a commit that referenced this issue Oct 10, 2017

ingest: Fix bug that prevent date_index_name processor from accepting…
… timestamps specified as a json number

Closes #26890

martijnvg added a commit that referenced this issue Oct 10, 2017

ingest: Fix bug that prevent date_index_name processor from accepting…
… timestamps specified as a json number

Closes #26890

martijnvg added a commit that referenced this issue Oct 10, 2017

ingest: Fix bug that prevent date_index_name processor from accepting…
… timestamps specified as a json number

Closes #26890
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.