Add new `default` option for timestamp field #7036

dadoonet · 2014-07-25T13:21:51Z

Index process fails when having _timestamp enabled and path option is set.
It fails with a TimestampParsingException[failed to parse timestamp [null]] message.

Reproduction:

DELETE test
PUT  test
{
    "mappings": {
        "test": {
            "_timestamp" : {
                "enabled" : "yes",
                "path" : "post_date"
            }
        }
    }
}
PUT test/test/1
{
  "foo": "bar"
}

You can now define a default value for when timestamp is not provided
within the index request or in the _source document.

By default, the default value is now which means the date the document was processed by the indexing chain.

You can disable that default value by setting default to null. It means that timestamp is mandatory:

{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "default" : null
        }
    }
}

If you don't provide any timestamp value, indexation will fail.

You can also set the default value to any date respecting timestamp format:

{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "format" : "YYYY-MM-dd",
            "default" : "1970-01-01"
        }
    }
}

If you don't provide any timestamp value, indexation will fail.

Closes #4718.

jpountz · 2014-07-28T06:45:20Z

Should the default be null rather than now when a path is specified? I don't like the fact that a malformed document would get the current time as a timestamp instead of raising an error?

jpountz · 2014-07-28T08:18:46Z

src/main/java/org/elasticsearch/cluster/metadata/MappingMetaData.java

@@ -528,6 +541,7 @@ public static void writeTo(MappingMetaData mappingMd, StreamOutput out) throws I
            out.writeBoolean(false);
        }
        out.writeString(mappingMd.timestamp().format());
+        out.writeString(mappingMd.timestamp().defaultTimestamp());


This breaks the stream backward compatibility. The default timestamp should only be written if the output version is >= 1.4.0

dadoonet · 2014-07-28T14:04:46Z

About default value it was more about not to break bwc. Before this change, when you activate _timestamp in mapping, if you don't set any timestamp in the request, "now" is applied by default.

I think we should keep it as it was previously.

Index process fails when having `_timestamp` enabled and `path` option is set. It fails with a `TimestampParsingException[failed to parse timestamp [null]]` message. Reproduction: ``` DELETE test PUT test { "mappings": { "test": { "_timestamp" : { "enabled" : "yes", "path" : "post_date" } } } } PUT test/test/1 { "foo": "bar" } ``` You can define a default value for when timestamp is not provided within the index request or in the `_source` document. By default, the default value is `now` which means the date the document was processed by the indexing chain. You can disable that default value by setting `default` to `null`. It means that `timestamp` is mandatory: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "default" : null } } } ``` If you don't provide any timestamp value, indexation will fail. You can also set the default value to any date respecting timestamp format: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "format" : "YYYY-MM-dd", "default" : "1970-01-01" } } } ``` If you don't provide any timestamp value, indexation will fail. Closes elastic#4718.

jpountz · 2014-07-29T16:21:02Z

src/test/java/org/elasticsearch/index/mapper/timestamp/TimestampMappingTests.java

+        assertThat(request.timestamp(), notNullValue());
+
+        // TODO compute that generated date is closed to current date


Should you address these TODOs?

Yeah! I just saw that I forgot to code that part :(

jpountz · 2014-07-29T16:24:09Z

@dadoonet Left one comment about the null case

dadoonet · 2014-07-29T17:39:25Z

@jpountz PR updated. Let me know

jpountz · 2014-07-30T14:59:41Z

LGTM

Index process fails when having `_timestamp` enabled and `path` option is set. It fails with a `TimestampParsingException[failed to parse timestamp [null]]` message. Reproduction: ``` DELETE test PUT test { "mappings": { "test": { "_timestamp" : { "enabled" : "yes", "path" : "post_date" } } } } PUT test/test/1 { "foo": "bar" } ``` You can define a default value for when timestamp is not provided within the index request or in the `_source` document. By default, the default value is `now` which means the date the document was processed by the indexing chain. You can disable that default value by setting `default` to `null`. It means that `timestamp` is mandatory: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "default" : null } } } ``` If you don't provide any timestamp value, indexation will fail. You can also set the default value to any date respecting timestamp format: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "format" : "YYYY-MM-dd", "default" : "1970-01-01" } } } ``` If you don't provide any timestamp value, indexation will fail. Closes #4718. Closes #7036. (cherry picked from commit 85eb0ea)

Index process fails when having `_timestamp` enabled and `path` option is set. It fails with a `TimestampParsingException[failed to parse timestamp [null]]` message. Reproduction: ``` DELETE test PUT test { "mappings": { "test": { "_timestamp" : { "enabled" : "yes", "path" : "post_date" } } } } PUT test/test/1 { "foo": "bar" } ``` You can define a default value for when timestamp is not provided within the index request or in the `_source` document. By default, the default value is `now` which means the date the document was processed by the indexing chain. You can disable that default value by setting `default` to `null`. It means that `timestamp` is mandatory: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "default" : null } } } ``` If you don't provide any timestamp value, indexation will fail. You can also set the default value to any date respecting timestamp format: ``` { "tweet" : { "_timestamp" : { "enabled" : true, "format" : "YYYY-MM-dd", "default" : "1970-01-01" } } } ``` If you don't provide any timestamp value, indexation will fail. Closes #4718. Closes #7036.

… paths PR elastic#7036 changed the behavior for timestamp when provided as a parameter or within the document using `path` attribute. This PR now considers that: * when using timestamp as a parameter, we use a default value of `now`. Which means that if no timestamp is provided, the current time is used when the index operation is performed. * when getting the timestamp from `path`, we use a default value of `null`. Which means that if no value is provided within the document, indexation will fail. If users want to set the default value for `timestamp`, they can explicitly set one or set `"default": "now"`. Closes elastic#8882.

When creating an index with: ``` PUT new_index { "mappings": { "power": { "_timestamp" : { "enabled" : true, "default": null } } } } ``` When restarting the cluster, `now` is applied instead of `null`. So index become: ``` { "mappings": { "power": { "_timestamp" : { "enabled" : true, "default": "now" } } } } ``` This PR fix that and applies `null` when it was explicitly set. Note that this won't happen anymore in 1.5 as `null` is not allowed anymore as a `default` value. See elastic#9104. See also: * elastic#7036 * elastic#9049 * elastic#9426#issuecomment-71534871

dadoonet added review labels Jul 25, 2014

dadoonet self-assigned this Jul 25, 2014

dadoonet mentioned this pull request Jul 25, 2014

generate timestamp when path is null #4718

Closed

jpountz reviewed Jul 28, 2014
View reviewed changes

jpountz removed the review label Jul 28, 2014

dadoonet added 2 commits July 28, 2014 16:52

Update after @jpountz review

10b4473

jpountz reviewed Jul 29, 2014
View reviewed changes

Update after @jpountz 2nd review

bb0d5f2

jpountz removed the review label Jul 30, 2014

dadoonet added v1.4.0 labels Jul 31, 2014

dadoonet closed this in 85eb0ea Jul 31, 2014

dadoonet changed the title ~~Add new default option for timestamp field~~ Mapping: add new default option for timestamp field Aug 4, 2014

dadoonet deleted the issue/4718-timestamp-master branch August 12, 2014 13:47

clintongormley changed the title ~~Mapping: add new default option for timestamp field~~ Mapping: Add new default option for timestamp field Sep 8, 2014

This was referenced Dec 8, 2014

Mapping: Add new default option for timestamp field elastic/elasticsearch-net#1105

Closed

New timestamp default option changed the default behavior for missing paths #8882

Closed

dadoonet mentioned this pull request Dec 23, 2014

New timestamp default option changed the default behavior for missing paths #9049

Closed

dadoonet mentioned this pull request Jan 26, 2015

Mapping: Explicit _timestamp default null is set to now #9428

Merged

clintongormley added the :Search/Mapping Index mappings, including merging and defining field types label Jun 6, 2015

clintongormley changed the title ~~Mapping: Add new default option for timestamp field~~ Add new default option for timestamp field Jun 6, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new `default` option for timestamp field #7036

Add new `default` option for timestamp field #7036

dadoonet commented Jul 25, 2014

jpountz commented Jul 28, 2014

jpountz Jul 28, 2014

dadoonet Jul 28, 2014

dadoonet commented Jul 28, 2014

jpountz Jul 29, 2014

dadoonet Jul 29, 2014

jpountz commented Jul 29, 2014

dadoonet commented Jul 29, 2014

jpountz commented Jul 30, 2014

		assertThat(request.timestamp(), notNullValue());

		// TODO compute that generated date is closed to current date

Add new default option for timestamp field #7036

Add new default option for timestamp field #7036

Conversation

dadoonet commented Jul 25, 2014

jpountz commented Jul 28, 2014

jpountz Jul 28, 2014

Choose a reason for hiding this comment

dadoonet Jul 28, 2014

Choose a reason for hiding this comment

dadoonet commented Jul 28, 2014

jpountz Jul 29, 2014

Choose a reason for hiding this comment

dadoonet Jul 29, 2014

Choose a reason for hiding this comment

jpountz commented Jul 29, 2014

dadoonet commented Jul 29, 2014

jpountz commented Jul 30, 2014

Add new `default` option for timestamp field #7036

Add new `default` option for timestamp field #7036