Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Elasticsearch ILM Polices #2454

Closed
wants to merge 56 commits into from
Closed

Add support for Elasticsearch ILM Polices #2454

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
bfb2e87
#2048 - Adds support for Elasticsearch ILM Polices
bhiravabhatla Sep 3, 2020
6acf5da
#2048 - Fixes index templates to work well with Index prefixes
bhiravabhatla Sep 4, 2020
17a8e1c
#2048 - Modify existing index templates to use ILM - avoid creating n…
bhiravabhatla Sep 26, 2020
06896ec
Merge branch 'master' into master
bhiravabhatla Sep 26, 2020
5f0dac7
#2048 - Fix perform_action method call by passing missing argument
bhiravabhatla Sep 26, 2020
f24f9fa
Merge branch 'master' into master
bhiravabhatla Oct 22, 2020
09ba770
Merge branch 'master' into master
bhiravabhatla Nov 27, 2020
fe5938f
#2048[WIP] - Add support for enabling ILM using command line param
bhiravabhatla Dec 4, 2020
f02553e
#2048[WIP] - Rebasing with upstream master
bhiravabhatla Dec 4, 2020
1af12df
#2048[WIP] - Fix lint issues
bhiravabhatla Dec 5, 2020
0f5bb18
#2048[WIP] - Fix integration tests and remobe ES_USE_ILM parameter fr…
bhiravabhatla Dec 5, 2020
8d3c483
#2048[WIP] - Fix failing es-otel-exporter-integration-test
bhiravabhatla Dec 5, 2020
e5fda11
#2048[WIP] - Implement suggested feedback and add integration test fo…
bhiravabhatla Dec 12, 2020
016f626
Merge branch 'master' of https://github.com/jaegertracing/jaeger
bhiravabhatla Dec 12, 2020
f524da3
#2048[WIP] - Fix lint issues
bhiravabhatla Dec 12, 2020
0076eaf
#2048[WIP] - Add write_index setting only if use_ilm is enabled & fix…
bhiravabhatla Dec 13, 2020
653b861
#2048[WIP] - Fix failing exporter integration tests
bhiravabhatla Dec 13, 2020
526f214
Update plugin/storage/es/options.go
bhiravabhatla Dec 14, 2020
8a51584
2048[WIP] - Implement @albertteoh feedback on error variable names an…
bhiravabhatla Dec 19, 2020
fcbf676
Fix fmt issue
bhiravabhatla Dec 19, 2020
a3e8f0e
Fix misleading use-alias help doc (#2692)
albertteoh Dec 15, 2020
8e0b750
#2048[WIP] - Fix failing build and implement feedback on escaping quotes
bhiravabhatla Dec 19, 2020
e2a0fee
#2048[WIP] - Add integration test to check index template names with …
bhiravabhatla Dec 19, 2020
3f31372
#2048[WIP] - Add check to fail if delete ILM policy fails
bhiravabhatla Dec 19, 2020
aacf4c9
#2048[WIP] - Move Index rollover tests to another file
bhiravabhatla Dec 19, 2020
77a8a17
#2048[WIP] - Skip TestIndexRollover_FailIfILMNotPresent test for es v…
bhiravabhatla Dec 19, 2020
4755013
#2048[WIP] - Clean ES before running rollover test for ES version < 7
bhiravabhatla Dec 19, 2020
667a511
Update plugin/storage/es/options.go
bhiravabhatla Dec 14, 2020
3696256
#2048[WIP] - Replace external dependency for rendering templates with…
bhiravabhatla Jan 16, 2021
4b044bc
Fix potential cross-site scripting issues (#2697)
yurishkuro Dec 20, 2020
877175c
Remove 'Report Security Issue' link since GitHub already adds it prom…
yurishkuro Dec 21, 2020
f57c970
Add gh action for hotrod ci jobs (#2702)
Ashmita152 Dec 24, 2020
c514c91
Clean-up "tcollector" references (#2704)
sniperking1234 Dec 25, 2020
183af3c
Fix flaky TestHotReloadUIConfigTempFile (#2713)
albertteoh Jan 7, 2021
d1e9dc4
[pkg/queue] Add `StartConsumersWithFactory` function (#2714)
mx-psi Jan 7, 2021
4c8ad45
[mem store] Return clones of spans to prevent in-place changes by adj…
bobrik Jan 9, 2021
602b943
Show plaintext as supported Kafka auth option in CLI help (#2721)
pdepaepe Jan 11, 2021
00b7daa
[agent] Add metrics to show connections status between agent and coll…
WalkerWang731 Jan 13, 2021
b980e87
[refactor] Minor clean-up of #2657 (#2728)
yurishkuro Jan 13, 2021
5b80343
[tests] Increase timeout for TestGRPCResolverRoundRobin (#2729)
yurishkuro Jan 14, 2021
dc1b855
Add ability to use JS file for UI configuration (jaeger-ui/#123) (#2707)
th3M1ke Jan 14, 2021
76256d6
Add support for Kafka SASL/PLAIN authentication via SCRAM-SHA-256 or …
WalkerWang731 Jan 14, 2021
1a888aa
#2048[WIP] - Fix fmt issues
bhiravabhatla Jan 16, 2021
9bec1a4
#2048[WIP] - Implement feedback on CLI - use cobra & unit tests
bhiravabhatla Jan 22, 2021
01654f9
Increase HotROD CI test timeout to 30sec
yurishkuro Jan 18, 2021
ea4e71c
Update jaeger-ui to use memoization for getConfig (#2732)
th3M1ke Jan 18, 2021
23f53dd
Fix broken CNCF logo link
yurishkuro Jan 18, 2021
ef16c6a
TLS support for HTTP API of Query server (#2337)
rjs211 Jan 19, 2021
0f0cc90
Add ability to remove inactive maintainers; introduce Emeritus status…
yurishkuro Jan 19, 2021
d4895d7
Transition black-adder and tiffon to Emeritus Maintainers (#2735)
yurishkuro Jan 20, 2021
6864653
Enable stale bot for PRs and questions (#2731)
yurishkuro Jan 20, 2021
6f8bfb1
Expose cache ttl for es span writer index+service (#2737)
necrolyte2 Jan 21, 2021
d85d954
#2048[WIP] - Fix failing integration tests and lint
bhiravabhatla Jan 22, 2021
c0f3e93
#2048[WIP] - Fix fmt issues
bhiravabhatla Jan 22, 2021
09e4e9d
#2048[WIP] - add tests to increase coverage
bhiravabhatla Jan 22, 2021
a3b465b
Fix merge conflicts
bhiravabhatla Jan 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions plugin/storage/es/esRollover.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import requests
import ssl
import sys
import re
from pathlib2 import Path
from requests.auth import HTTPBasicAuth

Expand Down Expand Up @@ -36,6 +37,7 @@ def main():
print('ES_TLS_CERT ... Path to TLS certificate file.')
print('ES_TLS_KEY ... Path to TLS key file.')
print('ES_TLS_SKIP_HOST_VERIFY ... (insecure) Skip server\'s certificate chain and host name verification.')
print('USE_ILM ... Use ILM to manage jaeger span and service indices')
print('ES_VERSION ... The major Elasticsearch version. If not specified, the value will be auto-detected from Elasticsearch.')
print('init configuration:')
print('\tSHARDS ... the number of shards per index in Elasticsearch (default {}).'.format(SHARDS))
Expand All @@ -54,17 +56,24 @@ def main():

action = sys.argv[1]

if str2bool(os.getenv('USE_ILM', 'false')):
span_template = 'jaeger-span-with-ilm'
service_template = 'jaeger-service-with-ilm'
else:
span_template = 'jaeger-span'
service_template = 'jaeger-service'

if str2bool(os.getenv('ARCHIVE', 'false')):
write_alias = prefix + ARCHIVE_INDEX + '-write'
read_alias = prefix + ARCHIVE_INDEX + '-read'
perform_action(action, client, write_alias, read_alias, prefix+'jaeger-span-archive', 'jaeger-span')
else:
write_alias = prefix + 'jaeger-span-write'
read_alias = prefix + 'jaeger-span-read'
perform_action(action, client, write_alias, read_alias, prefix+'jaeger-span', 'jaeger-span')
perform_action(action, client, write_alias, read_alias, prefix+'jaeger-span', span_template)
write_alias = prefix + 'jaeger-service-write'
read_alias = prefix + 'jaeger-service-read'
perform_action(action, client, write_alias, read_alias, prefix+'jaeger-service', 'jaeger-service')
perform_action(action, client, write_alias, read_alias, prefix+'jaeger-service', service_template)


def perform_action(action, client, write_alias, read_alias, index_to_rollover, template_name):
Expand Down Expand Up @@ -118,9 +127,12 @@ def create_aliases(client, alias_name, archive_index_name):
"""
ilo = curator.IndexList(client)
ilo.filter_by_regex(kind='regex', value='^'+archive_index_name+'$')
alias = curator.Alias(client=client, name=alias_name)
for index in ilo.working_list():
print("Adding index {} to alias {}".format(index, alias_name))
if re.search(r'write', alias_name):
bhiravabhatla marked this conversation as resolved.
Show resolved Hide resolved
alias = curator.Alias(client=client, name=alias_name, extra_settings={'is_write_index': True})
else:
alias = curator.Alias(client=client, name=alias_name)
alias.add(ilo)
alias.do_action()

Expand Down
49 changes: 49 additions & 0 deletions plugin/storage/es/mappings/jaeger-service-with-ilm-7.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"index_patterns": "*jaeger-service-*",
"order": 1,
"aliases": {
"jaeger-service-read": {}
},
"settings": {
"index.number_of_shards": ${__NUMBER_OF_SHARDS__},
"index.number_of_replicas": ${__NUMBER_OF_REPLICAS__},
"index.mapping.nested_fields.limit": 50,
"index.requests.cache.enable": true,
"lifecycle": {
"name": "jaeger",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to add a new set of index templates.

Could we instead add the lifecycle configuration to our existing index templates?

Copy link
Contributor Author

@bhiravabhatla bhiravabhatla Sep 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavolloffay - I tried that initially, When jaeger comes up, it is also trying to create the index templates ( even with --es.use-alias enabled) and if the template name is same - it is overriding it. Hence had to use new set of templates with higher priority (order = 1).

Ideally, if --es-use-alias is enabled - I believe template creation should also be left to external component (esRollover) along with alias creation. There might be a valid reason why it is not - I am not sure.

Right now, both jaeger and esRollover try to create the templates. As Jaeger starts after esRollover is run, its overriding the templates created by esRollover.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if the templates are overridden it should not be an issue since the definition does not change it just references the name of ILM.

--es.create-index-templates can be used to disable template creation

Copy link
Contributor Author

@bhiravabhatla bhiravabhatla Sep 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lifecycle or ILM is inside settings of index template -

"settings": {
    "index.number_of_shards": ${__NUMBER_OF_SHARDS__},
    "index.number_of_replicas": ${__NUMBER_OF_REPLICAS__},
    "index.mapping.nested_fields.limit": 50,
    "index.requests.cache.enable": true,
    "lifecycle": {
      "name": "jaeger",
      "rollover_alias": "${__INDEX_WRITE_ALIAS__}"
    }

Default template has settings without lifecycle. Hence when we start jaeger - it overrides the settings of the template ( unless --es.create-index-templates is used to disable it). That means we are saying user has to use --es.use-alias with --es.create-index-templates always - if ILM is needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get it now, I want to avoid having multiple variations of the index templates.

Can be the lifecycle be configured but a PUT (change) request?

Copy link
Contributor Author

@bhiravabhatla bhiravabhatla Sep 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get it now, I want to avoid having multiple variations of the index templates.

Yes - completely agree. Ideal thing would be to disable creation of templates from jaeger end whenever we have --es.use-aliases.

Can be the lifecycle be configured but a PUT (change) request?

Unfortunately we have to run init before starting jaeger. And index-templates must be created before 1st index is created. So, right now, I see no way but to have another set of templates for ILM with different name and higher order - so that when index templates are applied - ILM template takes precedence.

Either this or we somehow mandate to always use --es.use-alias with --es.create-index-templates set to false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideal thing would be to disable creation of templates from jaeger end whenever we have --es.use-aliases

@pavolloffay, I think I know why this was not done. As of now, If we enable use-aliases and start jaeger with out running esRollover init first --> Jaeger is creating the default index templates and two initial indices with names jaeger-span-write & jaeger-span-read, so that it would continue to work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavolloffay - As suggested - changes made to code to avoid using multiple variations of index templates. Please go through the latest commit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavolloffay Any feedback on this?

Copy link
Contributor Author

@bhiravabhatla bhiravabhatla Oct 22, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavolloffay / @jaegertracing/elasticsearch - Let me know, if you have any thoughts on the approach

"rollover_alias": "jaeger-service-write"
}
},
"mappings": {
"dynamic_templates": [
{
"span_tags_map": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "tag.*"
}
},
{
"process_tags_map": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "process.tag.*"
}
}
],
"properties": {
"serviceName": {
"type": "keyword",
"ignore_above": 256
},
"operationName": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
165 changes: 165 additions & 0 deletions plugin/storage/es/mappings/jaeger-span-with-ilm-7.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
{
"index_patterns": "*jaeger-span-*",
"order": 1,
"aliases": {
"jaeger-span-read": {}
},
"settings": {
"index.number_of_shards": ${__NUMBER_OF_SHARDS__},
"index.number_of_replicas": ${__NUMBER_OF_REPLICAS__},
"index.mapping.nested_fields.limit": 50,
"index.requests.cache.enable": true,
"lifecycle": {
"name": "jaeger",
"rollover_alias": "jaeger-span-write"
}
},
"mappings": {
"dynamic_templates": [
{
"span_tags_map": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "tag.*"
}
},
{
"process_tags_map": {
"mapping": {
"type": "keyword",
"ignore_above": 256
},
"path_match": "process.tag.*"
}
}
],
"properties": {
"traceID": {
"type": "keyword",
"ignore_above": 256
},
"parentSpanID": {
"type": "keyword",
"ignore_above": 256
},
"spanID": {
"type": "keyword",
"ignore_above": 256
},
"operationName": {
"type": "keyword",
"ignore_above": 256
},
"startTime": {
"type": "long"
},
"startTimeMillis": {
"type": "date",
"format": "epoch_millis"
},
"duration": {
"type": "long"
},
"flags": {
"type": "integer"
},
"logs": {
"type": "nested",
"dynamic": false,
"properties": {
"timestamp": {
"type": "long"
},
"fields": {
"type": "nested",
"dynamic": false,
"properties": {
"key": {
"type": "keyword",
"ignore_above": 256
},
"value": {
"type": "keyword",
"ignore_above": 256
},
"tagType": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"process": {
"properties": {
"serviceName": {
"type": "keyword",
"ignore_above": 256
},
"tag": {
"type": "object"
},
"tags": {
"type": "nested",
"dynamic": false,
"properties": {
"key": {
"type": "keyword",
"ignore_above": 256
},
"value": {
"type": "keyword",
"ignore_above": 256
},
"tagType": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"references": {
"type": "nested",
"dynamic": false,
"properties": {
"refType": {
"type": "keyword",
"ignore_above": 256
},
"traceID": {
"type": "keyword",
"ignore_above": 256
},
"spanID": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tag": {
"type": "object"
},
"tags": {
"type": "nested",
"dynamic": false,
"properties": {
"key": {
"type": "keyword",
"ignore_above": 256
},
"value": {
"type": "keyword",
"ignore_above": 256
},
"tagType": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}