Incorrect weekly index name YYYY-52 for events on 1st Jan when using logstash-%{+YYYY.ww} pattern?

[joshuar]

• Version: 5.1.1

• Operating System: Fedora Linux 25

• Description:
  It looks like when using the index pattern name logstash-%{+YYYY.ww}, i.e., a weekly time-based pattern, events from the 1st Jan of the year are incorrectly indexed into a YYYY-52 index rather than a YYYY-01 index. Events from the 2nd Jan are correctly indexed into the latter.

• Steps to Reproduce:
  Consider the following configuration:

input {
    stdin { }
}

filter {
    grok {
        match => {
            "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:message}"
        }
    }
    date {
        match => ["timestamp","ISO8601"]
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        user => "elastic"
        password => "changeme"
        index => "logstash-%{+YYYY.ww}"
    }
}

Note the index pattern: "logstash-%{+YYYY.ww}"

And the following log file:

2017-01-01T00:00:03.374Z On the 1st 1
2017-01-01T23:59:45.193Z On the 1st 2
2017-01-02T00:00:04.224Z On the 2nd 1
2017-01-02T00:00:05.227Z On the 2nd 2

Note we have two log lines on both the 1st and 2nd of Jan

Running Logstash on this file with cat test.log | /path/to/logstash -f ./logstash.conf results in the following documents indexed in Elasticsearch:

curl -XGET -u elastic:changeme "http://localhost:9200/logstash-*/_search?pretty"
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logstash-2017.01",
        "_type" : "logs",
        "_id" : "AVlxvjx_Pu2sMQyIrcqW",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-01-02T00:00:05.227Z",
          "@version" : "1",
          "host" : "pinback",
          "message" : [
            "2017-01-02T00:00:05.227Z On the 2nd 2",
            "On the 2nd 2"
          ],
          "tags" : [ ],
          "timestamp" : "2017-01-02T00:00:05.227Z"
        }
      },
      {
        "_index" : "logstash-2017.01",
        "_type" : "logs",
        "_id" : "AVlxvjx_Pu2sMQyIrcqV",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-01-02T00:00:04.224Z",
          "@version" : "1",
          "host" : "pinback",
          "message" : [
            "2017-01-02T00:00:04.224Z On the 2nd 1",
            "On the 2nd 1"
          ],
          "tags" : [ ],
          "timestamp" : "2017-01-02T00:00:04.224Z"
        }
      },
      {
        "_index" : "logstash-2017.52",
        "_type" : "logs",
        "_id" : "AVlxvjx_Pu2sMQyIrcqT",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-01-01T00:00:03.374Z",
          "@version" : "1",
          "host" : "pinback",
          "message" : [
            "2017-01-01T00:00:03.374Z On the 1st 1",
            "On the 1st 1"
          ],
          "tags" : [ ],
          "timestamp" : "2017-01-01T00:00:03.374Z"
        }
      },
      {
        "_index" : "logstash-2017.52",
        "_type" : "logs",
        "_id" : "AVlxvjx_Pu2sMQyIrcqU",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2017-01-01T23:59:45.193Z",
          "@version" : "1",
          "host" : "pinback",
          "message" : [
            "2017-01-01T23:59:45.193Z On the 1st 2",
            "On the 1st 2"
          ],
          "tags" : [ ],
          "timestamp" : "2017-01-01T23:59:45.193Z"
        }
      }
    ]
  }
}

The two log lines from the 1st Jan are indexed into a logstash-2017.52 index?

Attached is the config and test log:

test-log-and-config.zip

[sandeepkanabar]

Thanks @joshuar . A small note: Events from Jan 2, 2017 are indexed correctly because Jan 2 happens to be Monday which is first day of week. Until the first day of week is encountered, I believe it will continue to log into YYYY.52 index i.e 2017.52

[untergeek]

This is not a bug. You can even have a 53 week year in the ISO standard: https://en.wikipedia.org/wiki/ISO_week_date

[sandeepkanabar]

Hi @untergeek - I do get your point that there can be 53 weeks in a year. But in that case, it should log in 2016.53 and not 2017.53. That's what I'm trying to say. I've updated my previous comment to reflect that. Hope that clarifies.

[jordansissel]

@sandeepkanabar Your confusion is understandable, but the format fields in Joda behave indepedently.

The format YYYY asks for the year. The format of ww asks for "week of weekyear".

The differences between weekyear and year are important here and honestly have confused me in the past :)

January 1 2017 for format ww is 52 week of the weekyear (2016). YYYY does not account for weeks, so it simply returns the calendar year, 2017. If you want "weekyear" you want to use xxxx

From the joda docs:

 x       weekyear                     year          1996
 w       week of weekyear             number        27

Here's an example in irb showing this in action:

>> org.joda.time.DateTime.new(2017,1,1,0,0,0).toString("YYYY ww")
=> "2017 52"
>> org.joda.time.DateTime.new(2017,1,1,0,0,0).toString("xxxx ww")
=> "2016 52"

This is something easy for users to stumble over, I agree.

[untergeek]

This is a rare one, for sure. Seems like it's not a bug, so we'll close this for now.

[sandeepkanabar]

Thank you @jordansissel for the clarification. Can you please let me know how I can run and verify myself? I launched irb but when i run the above command I get undefined local variable or method 'org' for main:Object. I reckon I need to import the library. Can you help me out. Am a total newbie to Ruby. Never run it before.

[jordansissel]

@sandeepkanabar I provided an example in irb because, for me, that was the shortest path. The exact same library is used for Logstash to format dates. you can verify this with Logstash instead using @joshuar's example at the top of this issue.

The change you should is the following to @joshuar's example Logstash configuration:

-        index => "logstash-%{+YYYY.ww}"
+        index => "logstash-%{+xxxx.ww}"

If you wish to reproduce this with irb you will need JRuby, and from your report, it appears you are running MRI (the default implementation of Ruby), which doesn't support Java.

[joshuar]

Thanks for the explanations @untergeek and @jordansissel!

[sandeepkanabar]

@jordansissel thanks for the pointer on jruby. Able to make it work with jirb. This is pretty cool. Learnt quite a few things today 👍

[Rohlik]

Today I hit on this problem too.