Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Date index name processor doesn't apply to updates with doc #81764

Open
nik9000 opened this issue Dec 15, 2021 · 2 comments
Open

Date index name processor doesn't apply to updates with doc #81764

nik9000 opened this issue Dec 15, 2021 · 2 comments
Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team

Comments

@nik9000
Copy link
Member

nik9000 commented Dec 15, 2021

It looks like the date_index_name processor only works for index commands. Most ingest processors are happy to consume the update commands with a doc. Example:

curl -XPUT -HContent-Type:application/json -uelastic:password 'localhost:9200/_ingest/pipeline/monthlyindex?pretty' -d'{
  "description": "monthly date-time index naming",
  "processors" : [
    {
      "date_index_name" : {
        "field" : "date1",
        "index_name_prefix" : "my-index-",
        "date_rounding" : "M"
      }
    }
  ]
}'

curl -XPUT -HContent-Type:application/json -uelastic:password 'localhost:9200/_template/test_template?pretty' -d'{
  "index_patterns": ["my-index*"],
  "settings": {
    "index.default_pipeline": "monthlyindex"
  }
}'


curl -XDELETE -HContent-Type:application/json -uelastic:password localhost:9200/my-test-index
curl -XDELETE -HContent-Type:application/json -uelastic:password localhost:9200/my-index-2099-03-01
curl -XPOST -HContent-Type:application/json -uelastic:password 'localhost:9200/my-index-test/_bulk?pretty' -d'
{ "create": {}}
{ "date1": "2099-03-07T11:04:06.000Z", "my-keyword-field": "foo" }
'
# Expected:
# "create" : { "_index" : "my-index-2099-03-01", ...

curl -XPOST -HContent-Type:application/json -uelastic:password 'localhost:9200/my-index-test/_bulk?pretty' -d'
{"update": {"_id": "1"}}
{"doc": { "date1": "2099-03-07T11:04:06.000Z", "my-keyword-field": "foo" }, "doc_as_upsert": "true"}
'
# Unexpected:
# "update" : { "_index" : "my-index-test",
# Expected it to have been
# "update" : { "_index" : "my-index-2099-03-01",

I'm not sure the processor is meant to work on update's doc here, but if it isn't, I think we should document that.

@nik9000 nik9000 added >bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP team-discuss labels Dec 15, 2021
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Dec 15, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@joegallo joegallo added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP and removed :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Mar 22, 2023
@joegallo
Copy link
Contributor

I'm removing the team-discuss label from some older Team:Data Management issues -- we've had plenty of time to discuss them, but we haven't, so the label isn't serving its purpose. Feel free to delete this comment and/or re-add the team-discuss label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

3 participants