Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch 8.x support #3571

Closed
Tornhoof opened this issue Mar 9, 2022 · 50 comments · Fixed by #4829
Closed

Elasticsearch 8.x support #3571

Tornhoof opened this issue Mar 9, 2022 · 50 comments · Fixed by #4829
Labels
help wanted Features that maintainers are willing to accept but do not have cycles to implement storage/elasticsearch

Comments

@Tornhoof
Copy link

Tornhoof commented Mar 9, 2022

Requirement - what kind of business use case are you trying to solve?

Elastic recently released Elasticsearch 8.0, by now they've released 8.1.

Problem - what in Jaeger blocks you from solving the requirement?

The ES index can't be created anymore, I think the legacy template api part is now completely deprecated:

jaeger-collector    | {"level":"info","ts":1646824639.4679213,"caller":"config/config.go:205","msg":"Elasticsearch detected","version":8}
jaeger-collector    | {"level":"info","ts":1646824639.4697242,"caller":"zapgrpc/zapgrpc.go:129","msg":"Deprecation warning: 299 Elasticsearch-8.1.0-3700f7679f7d95e36da0b43762189bab189bc53a \"Legacy index templates are deprecated in favor of composable templates.\""}
jaeger-collector    | {"level":"fatal","ts":1646824639.4699457,"caller":"./main.go:84","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template  [type=parse_exception]","stacktrace":"main.main.func1\n\t./main.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.3.0/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.3.0/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

The https://www.elastic.co/guide/en/elasticsearch/reference/8.1/migrating-8.0.html lists now (somewhere in the middle)

The [create or update index template API](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/indices-templates-v1.html)’s template parameter has been removed.

Details
In 6.0, we deprecated the template parameter in create or update index template requests in favor of using index_patterns. Support for the template parameter is now removed in 8.0.

Impact
Use the create or update index template API's index_patterns parameter. Requests that include the template parameter will return an error.

I think the code falls back to the unversioned mappings if it does not find a specific mapping

So, in my opinion the code should not fall back to the jaeger-span.json mapping but instead use the
jaeger-span-7.json for newer versions, e.g. use the -7 jsons for 8 etc.

e.g. change line 42 from == 7 to >=7

// GetMapping returns the rendered mapping based on elasticsearch version
func (mb *MappingBuilder) GetMapping(mapping string) (string, error) {
if mb.EsVersion == 7 {
return mb.fixMapping(mapping + "-7.json")
}
return mb.fixMapping(mapping + ".json")
}

Workaround

Force the ES Version to 7 via --es.version=7

@yurishkuro
Copy link
Member

Would you like to submit a PR? Besides the logic fix it would need to add v8 to the CI steps.

@yurishkuro yurishkuro added help wanted Features that maintainers are willing to accept but do not have cycles to implement good first issue Good for beginners labels Mar 9, 2022
@Tornhoof
Copy link
Author

Tornhoof commented Mar 9, 2022

Sure, I'll give it a try.

@Ashmita152
Copy link
Contributor

Ashmita152 commented Mar 12, 2022

I spent time investigating the effort in achieving it. I think this is going to be difficult now that we use same elasticsearch client olivere/elastic to support both of our elasticsearch and opensearch implementations.

Major challenge:
Right now we are using v6 of olivere/elastic which doesn't have support for new index template. To support it we need to update to v7 of olivere/elastic specifically for olivere/elastic#1458 and olivere/elastic#1415 which inturn breaks the opensearch support.

Not sure how we should proceed.

cc @yurishkuro

@s7an-it
Copy link

s7an-it commented Mar 20, 2022

Workaround

Force the ES Version to 7 via --es.version=7

I have tried latest helm operator installation and setting this parameter didn't result in anything better.

@bitfactory-sem-denbroeder

Would it be possible for Jaeger to add a separate docker image/branch which maintains opensearch support and the main branch getting support for Elasticsearch 8.x ?

@smithyj
Copy link

smithyj commented Apr 27, 2022

hope to support elasticsearch 8.1.x

@pavolloffay
Copy link
Member

pavolloffay commented Apr 27, 2022

We are working on bootstrapping Jaeger v2 (#3500) which will be based on OTEL collector. This will allow us to have ES exporter per ES version.

@DanielBruzualRW
Copy link

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

@kruczjak
Copy link

kruczjak commented May 2, 2022

I would be thankful for any workaround too.

We need to upgrade our elasticsearch cluster but we are stuck because of this problem

@seilorjunior
Copy link

we also need this asap!!!!!!1

@foxracle
Copy link

we also need this asap! +1

@sergiomcalzada
Copy link

sergiomcalzada commented May 12, 2022

Hi,

Workaround

Force the ES Version to 7 via --es.version=7 and es.create-index-templates: false

Add add this two indexes. If some of them fails related to "index priority" just change it

PUT _index_template/jaeger-service
{
  "priority": 1,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "operationName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-service-*"
  ],
  "composed_of": []
}
PUT _index_template/jaeger-span
{
  "priority": 2,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "traceID": {
          "type": "keyword"
        },
        "process": {
          "type": "object",
          "properties": {
            "tag": {
              "type": "object"
            },
            "serviceName": {
              "type": "keyword"
            },
            "tags": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "references": {
          "type": "nested",
          "properties": {
            "spanID": {
              "type": "keyword"
            },
            "traceID": {
              "type": "keyword"
            },
            "refType": {
              "type": "keyword"
            }
          }
        },
        "startTimeMillis": {
          "type": "date"
        },
        "flags": {
          "type": "integer"
        },
        "operationName": {
          "type": "keyword"
        },
        "parentSpanID": {
          "type": "keyword"
        },
        "tags": {
          "type": "nested",
          "properties": {
            "tagType": {
              "type": "keyword"
            },
            "value": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        },
        "duration": {
          "type": "long"
        },
        "spanID": {
          "type": "keyword"
        },
        "startTime": {
          "type": "long"
        },
        "tag": {
          "type": "object",
          "properties": {}
        },
        "logs": {
          "type": "nested",
          "properties": {
            "fields": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            },
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-span-*"
  ],
  "composed_of": []
}

@aletor92
Copy link

aletor92 commented May 13, 2022

Workaround

It Works ! thanks 👍

@Evesy
Copy link

Evesy commented May 13, 2022

As part of adding support for the newer index template API, it might be nice to have the option for Jaeger to create component templates instead of index templates.

With the current API we have a couple of other index templates targetting jaeger indices that all get merged together to perform the following modifications:

  • Set custom default settings that apply to all our indices (Tier routing, refresh interval, translog settings etc.)
  • Modify some field mappings (e.g. we set eager_global_ordinals on traceID for better search performance)

With the new index template API this won't be possible as only one index template wll be chosen (highest priority) and even if Jaeger supporting passing composed_of values to the template, which would allow us to pull in our default index settings, anything still explicitly set by Jaeger template (e.g. the traceID field mapping) would take precedence over our modifications.

I'm not sure if this is a common use case or not, otherwise we may choose to disable Jaeger index template creation and just managed them entirely ourselves, however pulling in any new field mappings etc. automatically when upgrading Jaeger is handy

@jasperjonker
Copy link

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

@sylvainOL
Copy link

Fyi, https://github.com/jaegertracing/spark-dependencies is not working with es8.x also.

That means it cannot really be used in production IMHO

@billalley
Copy link

We are experiencing problems connecting Jaeger to our existing ES setup due to this issue.
I've gone ahead and followed the work around instructions above, currently using the Helm chart for setup.

Ex:

   cmdlineParams:
      es.version: 7
      es.create-index-templates: false 

But no matter I still see this error relating to the template index patterns:

"caller":"./main.go:69","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates

I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things.

Anyone have further workaround info/updates on ES8 support?

@billalley
Copy link

Looks like this was some yaml mangling - No matter what was specified, checking the config state on the ingester showed

spec:
  containers:
  - args:
    - --es.create-index-templates
    - --es.version=7

Solution: quote the false

   cmdlineParams:
      es.version: 7
      es.create-index-templates: "false"

Which becomes:

spec:
  containers:
  - args:
    - --es.create-index-templates=false
    - --es.version=7

@yurishkuro
Copy link
Member

@Ashmita152 I came across olivere/elastic#1533, where the author of the library unequivocally states:

Elastic v8 won't happen. I will polish v7 if I have the time and energy, but v8 won't happen. I'm sorry.

So there seems to be little point in upgrading to v7 of the library as it won't support v8 of ES.

Unlike our Cassandra implementation that uses an internal abstraction over gocql driver, the ES implementation has a large exposure to many different types from olivere/elastic driver. This complicates things a lot if we want to support both ESv8 and OpenSearch using different drivers.

ESv7 will be EOL-ed by 2023-08-01.

@ajax-osadchuk-r
Copy link

ajax-osadchuk-r commented Sep 6, 2022

@yurishkuro Could you please clarify whether Jaeger is going to support ElasticSearch v8.+? At least, if there are any plans to do so in near times? This issue blocks our team and forces either to use outdated infrastructure or switch to another distributed tracing system.

@ajax-osadchuk-r
Copy link

ajax-osadchuk-r commented Sep 29, 2022

@yurishkuro @pavolloffay Any thoughts about my previous comment?

@Renatas-M
Copy link

Facing issues after upgrading to ES8. Jaeger Collector startup errors can be fixed with workaround by adding --es.version=7 and --es.create-index-templates=false command parameters. But then Jaeger Query is erroring with Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]. Thanks to sergiomcalzada, the suggested manual index template creation is helping to restore Jaeger Query functionality. I am applying suggested templates using Kibana->Dev Tools->Console.

@sunxuan11
Copy link

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

@sunxuan11
Copy link

Have you solved it? How did you solve it?

@sunxuan11
Copy link

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

I have the same problem with you

@pie-r
Copy link

pie-r commented Mar 6, 2023

I think the index-template workaround, could be implemented in the initContainers of the collector deployment. I've created the MR jaegertracing/helm-charts#452. If you are using the jaeger operator helm chart, you may get adavantage from this.

@mikednight
Copy link

for the most part our system is working, however the init phase of rollover still returns:
Error: failed to create template: jaeger-span, request failed, status code: 400, body: {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [template] in the template "}],"type":"parse_exception","reason":"unknown key [template] in the template "},"status":400}

the actual rollover returns:
Error: failed to create rollover target: jaeger-span-write, request failed, status code: 404, body: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [jaeger-span-write]","resource.type":"index_or_alias","resource.id":"jaeger-span-write","index_uuid":"_na_","index":"jaeger-span-write"}],"type":"index_not_found_exception","reason":"no such index [jaeger-span-write]","resource.type":"index_or_alias","resource.id":"jaeger-span-write","index_uuid":"_na_","index":"jaeger-span-write"},"status":404}

is there a known workaround for this?

Also, is there a timeline on when actual support will be ready for elasticsearch 8?

thank you.

@MephistoLynn
Copy link

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

Thanks. It works.感谢方法有效。

@s7an-it
Copy link

s7an-it commented Apr 6, 2023

@sergiomcalzada, on 8.7 I had to delete from both templates, should I replace it with something else?
"_size": {
"enabled": false
},
Thanks it started working like that, I had to restart all (jaeger and eck operator components) and then I had to do delete all old indexes from Jaeger and wait a bit.

@jpuskar
Copy link

jpuskar commented May 3, 2023

Opensearch is a good alternative. Not sure there is interest in building support for 8.x+. There are also other backends which are efficient as options.

Unfortunately, none of the others have operators as robust as ECK.
E.g. Sure OpenSearch has a helm chart, but it's missing a ton of functionality.

@fmassot
Copy link

fmassot commented Jun 8, 2023

@jpuskar what are your requirements in terms of features/dataset size?

@paulc0te
Copy link

paulc0te commented Jul 3, 2023

Hi. This topic is still relevant. Is there any plan to support Elasticsearch 8? The problem is exacerbated by the fact that there are currently no supported helm charts Elasticsearch 7.

@henrygas2018
Copy link

henrygas2018 commented Jul 18, 2023

The _template refers to the legacy template, while the _index_template is a new version of the template known as composable templates. In ES 7.x, both _template and _index_template are supported. However, users should manage their index_patterns to be split. As such, you may need to modify the _template created by the jaeger-collector and remove the prefix * in index_patterns.

First, execute the following command:

GET /_template/jaeger-span

Next, copy the response content. Change *jaeger-span-* to jaeger-span-*, and keep the rest of the content unchanged.

Finally, execute the following command in ES:

PUT /_template/jaeger-span
{
    ...
    "index_patterns": [
        "jaeger-span-*"
    ],
    ...
}

This will update the template accordingly, notice the response warning.

@odinsy
Copy link

odinsy commented Jul 27, 2023

@henrygas2018 or you can just don't set prefix and everything will be work (if you created ILM policy before jaeger start)

storage:
  type: elasticsearch
  elasticsearch:
    # indexPrefix: jaeger
    ...
    cmdlineParams:
      es.version: 7
      es.create-index-templates: "false"
      # es.index-prefix: jaeger
collector:
  enabled: true
  cmdlineParams:
    es.use-aliases: true
    es.use-ilm: true
query:
  enabled: true
  cmdlineParams:
    es.use-aliases: true
    es.use-ilm: true

image

@fatihdogmus
Copy link

Elastic v7 has ben EOL as of 1st of August. If someone wants to use jaeger properly, they would have to install an EOL version.

Is there at least a timeline on when can we expect support for Elasticsearch 8?

@haanhvu
Copy link
Contributor

haanhvu commented Aug 3, 2023

@fatihdogmus This has been proposed as a project for LFX mentorship: #4600 So someone's going to start working on it in Sep.

@stmlange
Copy link

stmlange commented Sep 5, 2023

Just in case someone bumps into this:
With ES_VERSION=7 (or --es.version=7 and ES_CREATE_INDEX_TEMPLATES="false" (or es.create-index-templates="false") as suggested above one still might get the following cryptic error message:

"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available"

Any suggestions?

@sergeykad
Copy link

Is there a way to create all the configurations in a fresh installation on
Elasticsearch 8 or the only option is to install with ES7 and then upgrade to 8 using the workarounds mentioned above?

@sergiomcalzada
Copy link

You can apply all the workarounds in a fresh v8 (tested). You will need to create the index by your self, you can find the code for the indexes in a previous comment from me

@sergeykad
Copy link

You can apply all the workarounds in a fresh v8 (tested). You will need to create the index by your self, you can find the code for the indexes in a previous comment from me

I did this, but there are missing aliases and ILM related configurations.

yurishkuro added a commit that referenced this issue Nov 23, 2023
## Which problem is this PR solving?
- Resolves #3571 

## Description of the changes
- Added index templates for esv8
- Added a esv8 client to handle the put index template request that is
not currently possible by olivere/elastic
- After these changes are merged , users no more need to use
create.index-templates= false unless they want to add their custom index
templates and force the es.version = 7

---------

Signed-off-by: bugslayer-332 <ayashwanth9503@gmail.com>
Signed-off-by: Yashwanth Reddy  <89122324+pmuls99@users.noreply.github.com>
Signed-off-by: Yuri Shkuro <github@ysh.us>
Co-authored-by: bugslayer-332 <ayashwanth9503@gmail.com>
Co-authored-by: Yuri Shkuro <github@ysh.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Features that maintainers are willing to accept but do not have cycles to implement storage/elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.