Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest: Align syntax to use _ingest._value across ingest processors #51162

Open
fkelbert opened this issue Jan 17, 2020 · 3 comments
Open

Ingest: Align syntax to use _ingest._value across ingest processors #51162

fkelbert opened this issue Jan 17, 2020 · 3 comments
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team

Comments

@fkelbert
Copy link
Contributor

Different ingest processors require different syntax to access the value of _ingest._value ("_ingest._value" vs "{{_ingest._value}}").

Instead, syntax across different processors should be the same for better user experience.

E.g., append vs uppercase:

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "foreach": {
          "field": "recipients",
          "processor": {
            "append": {
              "field": "test",
              "value": [ "{{_ingest._value}}" ]
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "recipients": [
          "Bar@foo.com"
        ]
      }
    }
  ]
}

vs

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "foreach": {
          "field": "recipients",
          "processor": {
            "uppercase": {
              "field": "_ingest._value"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "recipients": [
          "Bar@foo.com"
        ]
      }
    }
  ]
}
@fkelbert fkelbert added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP labels Jan 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@danhermann
Copy link
Contributor

@fkelbert, the uppercase processor takes only a field name and not literal values (it would make no sense to uppercase a literal value) and therefore does not require the template snippet syntax. The append processor, on the other hand, accepts a value rather than a field name. The template snippet syntax is available should you wish the append processor to use the value of a field. These are intentionally different behaviors (and their parameters are intentionally named field and value).

@fkelbert
Copy link
Contributor Author

@danhermann Thanks for the clarification. The difference does make sense.

And yet, I am wondering if this can be simplified for users. At the very least, we should provide documentation explaining the difference and in which context which syntax needs to be used.

@rjernst rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

4 participants