Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New timestamp default option changed the default behavior for missing paths #8882

Closed
Mpdreamz opened this issue Dec 10, 2014 · 1 comment · Fixed by #9104
Closed

New timestamp default option changed the default behavior for missing paths #8882

Mpdreamz opened this issue Dec 10, 2014 · 1 comment · Fixed by #9104
Assignees
Labels
>regression :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch v1.5.0 v2.0.0-beta1

Comments

@Mpdreamz
Copy link
Member

PR #7036 introduced a new default option to the _timestamp mapping.

The main goal was to combat the case when you have a _timestamp mapping with path set but index a document without the given path present.

e.g you have this mapping:

"_timestamp" : {
    "enabled" : "yes",
    "path" : "post_date"
}

and present this document:

{
  "foo": "bar"
}

pre 1.3 the indexation would fail, using 1.4 we default to using now() for _timestamp when path is missing.
To reintroduce the 1.3 behavior you now have to pass default: null giving null the special meaning of mandatory. I can not think of anywhere else where we give null a special meaning.

I personally feel that when you map _timestamp with _path the default result when you index a document without that path should be an exception (as it was before 1.4).

If you know your data may not contain the value you should proactively configure for this situation setting a bool ignore_missing that falls back to the behavior had path not been specified in the mapping in the first place.

1.3 behaviour with a new ignore_missing bool configuration:

_timestamp config Result.
enabled now
()
using path value of path or 💥
ignore_missing value of path or now()

1.4 handling of _timestamp

_timestamp config Result.
enabled now()
using path value or now
()
using default now(), <fixed date> or 💥 if set to null

Out of these two the 1.3 way of handling _timestamp mappings makes the most sense to me.

@clintongormley
Copy link
Contributor

I agree - the default if a path is specified should be null

dadoonet added a commit to dadoonet/elasticsearch that referenced this issue Dec 23, 2014
… paths

PR elastic#7036 changed the behavior for timestamp when provided as a parameter or within the document using `path` attribute.

This PR now considers that:

* when using timestamp as a parameter, we use a default value of `now`. Which means that if no timestamp is provided, the current time is used when the index operation is performed.
* when getting the timestamp from `path`, we use a default value of `null`. Which means that if no value is provided within the document, indexation will fail.

If users want to set the default value for `timestamp`, they can explicitly set one or set `"default":  "now"`.

Closes elastic#8882.
@dadoonet dadoonet added v1.5.0 v2.0.0-beta1 :Search Foundations/Mapping Index mappings, including merging and defining field types and removed v1.3.8 v1.4.3 labels Dec 31, 2014
dadoonet added a commit to dadoonet/elasticsearch that referenced this issue Jan 20, 2015
Related to elastic#9049.

By default, the default value for `timestamp` is `now` which means the date the document was processed by the indexing chain.

You can now reject documents which not provide a `timestamp` value by setting `ignore_missing` to false (default to `true`):

```js
{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "ignore_missing" : false
        }
    }
}
```

When you update the cluster to 1.5 or master, this index created with 1.4 we automatically migrate an index created with 1.4 to the 1.5 syntax.

Let say you have defined this in elasticsearch 1.4.x:

```js
DELETE test
PUT test
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}
PUT test/type/_mapping
{
  "type" : {
      "_timestamp" : {
          "enabled" : true,
          "default" : null
      }
  }
}
```

After migration, the mapping become:

```js
{
   "test": {
      "mappings": {
         "type": {
            "_timestamp": {
               "enabled": true,
               "store": false,
               "ignore_missing": false
            },
            "properties": {}
         }
      }
   }
}
```

Closes elastic#8882.
dadoonet added a commit that referenced this issue Jan 20, 2015
Related to #9049.

By default, the default value for `timestamp` is `now` which means the date the document was processed by the indexing chain.

You can now reject documents which not provide a `timestamp` value by setting `ignore_missing` to false (default to `true`):

```js
{
    "tweet" : {
        "_timestamp" : {
            "enabled" : true,
            "ignore_missing" : false
        }
    }
}
```

When you update the cluster to 1.5 or master, this index created with 1.4 we automatically migrate an index created with 1.4 to the 1.5 syntax.

Let say you have defined this in elasticsearch 1.4.x:

```js
DELETE test
PUT test
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}
PUT test/type/_mapping
{
  "type" : {
      "_timestamp" : {
          "enabled" : true,
          "default" : null
      }
  }
}
```

After migration, the mapping become:

```js
{
   "test": {
      "mappings": {
         "type": {
            "_timestamp": {
               "enabled": true,
               "store": false,
               "ignore_missing": false
            },
            "properties": {}
         }
      }
   }
}
```

Closes #8882.
(cherry picked from commit fb10346)
@javanna javanna added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>regression :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch v1.5.0 v2.0.0-beta1
Projects
None yet
4 participants