Jaeger and elasticsearch ILM

[kdevries]

Hi,

After fiddling around for a long time and getting bits of useful information on various spots I finally got ILM working for Jaeger.

Usefull info for me was:
#1242 (comment)
https://dimmaski.com/ilm-elasticsearch-jaeger/

Steps to follow.

• first create an ILM policy on elastic for jaeger:
  PUT on https://{{elastic_server}}/_ilm/policy/jaeger-ilm-policy with JSON (cahnge what is needed):

{ "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "3d" }, "set_priority": { "priority": 100 } } }, "delete": { "min_age": "1d", "actions": { "delete": {} } } } } }

(I use the jaeger-service from here on but also do the steps below for jaeger-span:

• create jaeger-service template:
  PUT on https://{{elastic_server}}/_template/jaeger-service with JSON:

{
        "order": 0,
        "index_patterns": [
            "*jaeger-service-*"
        ],
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "jaeger-ilm-policy",
                    "rollover_alias": "jaeger-service-write"
                },
                "mapping": {
                    "nested_fields": {
                        "limit": "50"
                    }
                },
                "requests": {
                    "cache": {
                        "enable": "true"
                    }
                },
                "number_of_shards": "5",
                "number_of_replicas": "1"
            }
        },
        "mappings": {
            "dynamic_templates": [
                {
                    "span_tags_map": {
                        "path_match": "tag.*",
                        "mapping": {
                            "ignore_above": 256,
                            "type": "keyword"
                        }
                    }
                },
                {
                    "process_tags_map": {
                        "path_match": "process.tag.*",
                        "mapping": {
                            "ignore_above": 256,
                            "type": "keyword"
                        }
                    }
                }
            ],
            "properties": {
                "operationName": {
                    "ignore_above": 256,
                    "type": "keyword"
                },
                "serviceName": {
                    "ignore_above": 256,
                    "type": "keyword"
                }
            }
        },
        "aliases": {
            "jaeger-service-read": {}
        }
    }
}

• create jaeger-service alias:
  PUT on https://{{elastic_server}}/jaeger-service-write-000001 with JSON:

{
  "aliases": {
    "jaeger-service-write": {"is_write_index": true} // note that is_write_index works only in ES6.4
  }
}

• then start jaeger-collector and jaeger-query with flags:
  --es.use-aliases=true --es.create-index-templates=false --es.use-ilm=true

If there is any other way of doing this (use everything default and then change ILM settings ?) that is handier please let me know.

Friendly greetings,

Kevin

[albertteoh]

@kdevries did you know about Jaeger's ILM documentation?
https://www.jaegertracing.io/docs/1.22/deployment/#elasticsearch-ilm-support

If possible, please let us know how it could be improved to make it easier for users like yourself to follow.

[kdevries]

Hi,

Thanks for the tip, problem was when I followed the standard setup of ILM the rollover failed due to missing aliases (I might have done something wrong)
Also I run the jaeger components as systemd services, documentation to set this up and how to configure the env variable (to talk to ES) and the ES parameters might be very usefull to syadmins that can not use docker :) (I have it all documented if wanted)

greetings,

Kevin.

[albertteoh]

I see, in the documentation linked above, there is a step for creating the alias. If the ilm policy isn't yet created, a useful error message should appear like so:

$ docker run -it --rm --net=host -e ES_USE_ILM=true jaegertracing/jaeger-es-rollover:latest init http://localhost:9200

Detected ElasticSearch Version 7
ILM policy 'jaeger-ilm-policy' doesn't exist in Elasticsearch. Please create it and rerun init

Then after creating the ilm policy it should successfully initialize the aliases:

$ docker run -it --rm --net=host -e ES_USE_ILM=true jaegertracing/jaeger-es-rollover:latest init http://localhost:9200

Detected ElasticSearch Version 7
Creating index template jaeger-span
{"acknowledged":true}
Creating index jaeger-span-000001
...

These shouldn't talk to jaeger components, just ES over HTTP, so you should be able to run these with your setup right?

[kdevries]

Hi,

I don't use docker :). I just wanted to share the above to show what worked for me.
When I did the normal way of setting it up, with a correct policy in place I encountered alias and rollover problems and also the query part did not work (this really want jaeger-<service/span>-<write/read> naming in EL) to work in alias mode.
Doing the templating,configuration and aliasses of ILM manually did the trick for me, it all works very nicely now.
When I have more time I will give the docker preset way of doing this a go :)

greetings,

Kevin

[albertteoh]

Ah, I misunderstood, I'd thought you just weren't using docker for Jaeger.

Thanks for sharing your approach in any case, @kdevries :)

[jkutes]

How would I make this ILM initializer work with ES version 8.x?

docker run -it --rm --net=host -e ES_USE_ILM=true  jaegertracing/jaeger-es-rollover:latest init $ENDPOINT --es.username=$ESUSER --es.password=$ESPASS

Error: ILM is supported only for ES version 7+

[albertteoh]

@jkutes forgive my ignorance with ES, but does Elasticsearch offer guarantees on the backwards compatibility of the ILM feature for version 8.x (and onwards)? If so, it should involve a fairly trivial change. Currently, jaeger strictly checks for version 7, so the error message is misleading/incorrect.

[jkutes]

Perhaps, it's only change for the check to include v7 & 8, but I don't know enough to say for sure.
As a workaround, I ran a bash script to create ILM policy and update existing jager index templates and indices with Elastic API.