new curator version breaks the TLS verify functionality

[rgaduput]

When upgraded the curator from 8.0.8 to 8.0.10 we have observed that it fails to connect with Elastic with TLS exception even though cert verify is set to false, this functionality works just fine if we go back to older version 8.0.8.
please note i have tried the latest version 8.0.12 as well.

Expected Behavior

Certs verification ignored while connecting to Elasticseach.

Actual Behavior

Fails to connect with below

2024-03-26 12:04:52,549 INFO      Preparing Action ID: 1, "delete_indices"
2024-03-26 12:04:52,550 INFO      Creating client object and testing connection
2024-03-26 12:04:53,051 CRITICAL  Unable to establish client connection to Elasticsearch!
2024-03-26 12:04:53,051 CRITICAL  Exception encountered: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)))

Steps to Reproduce the Problem

1. Create actions.yml with below content

actions:
  1:
    action: delete_indices
    description: "Clean up ES by deleting old indices"
    options:
      continue_if_exception: False
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: kibana
      exclude: True
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

2. Create config.yml and set the verify_certs as false

elasticsearch:
  client:
    hosts: https://infra-elasticsearch.infra:9200
    verify_certs: false
  other_settings:
    username: ${ES_USER}
    password: ${ES_PASSWD}

3. Run the curator with config and action files

  args:
    - '--config'
    - /etc/es-curator/config.yml
    - /etc/es-curator/action_file.yml

Specifications

• Version: 8.0.10
• Platform: Kubernetes
• Subsystem: n/a

Context (Environment)

Trying to perform Elastic index clean up but ignore the certs while connecting to it since certs are self signed.

Detailed Description

[untergeek]

Can you provide debug output of this, please?

[rgaduput]

@untergeek Hi,
I have tried below config to produce the DEBUG ouput, but the ouput did not changed. May be i have to try something else ?

elasticsearch:
  client:
    hosts: https://infra-elasticsearch.infra:9200
    verify_certs: false
  other_settings:
    username: ${ES_USER}
    password: ${ES_PASSWD}
logging:
  loglevel: DEBUG

[rgaduput]

@untergeek had to retry with --loglevel instead of config file, and here is the debug

For some reason Elastic search config ignored the verify_certs: false and log contains it as true

"Elasticsearch Configuration" config: {'client': {'hosts': 'https://infra-elasticsearch.infra:9200', 'http_compress': False, 'verify_certs': True}, 'other_settings': {'master_only': False, 'skip_version_test': False, 'username': 'elastic', 'password': 'REDACTED', 'api_key': {}}}

2024-03-26 16:01:59,517 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: <function validfilters.<locals>.func at 0x7f79e4121120>
2024-03-26 16:01:59,517 DEBUG     es_client.helpers.schemacheck               __init__:58   "filters" config: [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7}]
2024-03-26 16:01:59,517 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41204a0>, msg=None), msg=None)}
2024-03-26 16:01:59,517 DEBUG     es_client.helpers.schemacheck               __init__:56   "filter" config: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,517 DEBUG     curator.validators.filter_functions                   func:78   Filter #0: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,517 DEBUG     curator.defaults.filtertypes.age                    age:81   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4123420>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'field': Any(<class 'str'>, msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
2024-03-26 16:01:59,518 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4123420>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
2024-03-26 16:01:59,518 DEBUG     es_client.helpers.schemacheck               __init__:56   "filter" config: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7}
2024-03-26 16:01:59,518 DEBUG     curator.validators.filter_functions                   func:78   Filter #1: {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'stats_result': 'min_value', 'timestring': None, 'exclude': False, 'epoch': None}
2024-03-26 16:01:59,518 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
2024-03-26 16:01:59,518 DEBUG     es_client.helpers.schemacheck               __init__:56   "action type" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]}
2024-03-26 16:01:59,518 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'cold2frozen', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2024-03-26 16:01:59,521 DEBUG     es_client.helpers.schemacheck               __init__:56   "structure" config: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]}
2024-03-26 16:01:59,522 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41374c0>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135800>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135620>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4135b20>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
2024-03-26 16:01:59,522 DEBUG     es_client.helpers.schemacheck               __init__:56   "options" config: {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True}
2024-03-26 16:01:59,522 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: <function validfilters.<locals>.func at 0x7f79e4123600>
2024-03-26 16:01:59,522 DEBUG     es_client.helpers.schemacheck               __init__:58   "filters" config: [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}]
2024-03-26 16:01:59,523 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e41204a0>, msg=None), msg=None)}
2024-03-26 16:01:59,523 DEBUG     es_client.helpers.schemacheck               __init__:56   "filter" config: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,523 DEBUG     curator.validators.filter_functions                   func:78   Filter #0: {'filtertype': 'kibana', 'exclude': True}
2024-03-26 16:01:59,523 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'size', 'space', 'state']"), 'disk_space': Any(Coerce(float, msg=None), msg=None), 'reverse': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122b60>, msg=None), msg=None), 'use_age': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122840>, msg=None), msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f79e4122ac0>, msg=None), msg=None), 'threshold_behavior': Any('greater_than', 'less_than', msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
2024-03-26 16:01:59,523 DEBUG     es_client.helpers.schemacheck               __init__:56   "filter" config: {'filtertype': 'space', 'disk_space': 6, 'use_age': True, 'source': 'creation_date'}
2024-03-26 16:01:59,523 DEBUG     curator.validators.filter_functions                   func:78   Filter #1: {'filtertype': 'space', 'disk_space': 6.0, 'use_age': True, 'source': 'creation_date', 'reverse': True, 'threshold_behavior': 'greater_than', 'stats_result': 'min_value', 'timestring': None, 'exclude': False}
2024-03-26 16:01:59,524 DEBUG           curator.classdef               __init__:49   Action Configuration: {'actions': {1: {'action': 'delete_indices', 'description': 'Clean up ES by deleting old indices', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'age', 'source': 'creation_date', 'direction': 'older', 'unit': 'days', 'unit_count': 7, 'stats_result': 'min_value', 'timestring': None, 'exclude': False, 'epoch': None}]}, 2: {'action': 'delete_indices', 'description': 'Clean up ES by deleting indices using excess disk space', 'options': {'continue_if_exception': False, 'disable_action': False, 'ignore_empty_list': True, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'kibana', 'exclude': True}, {'filtertype': 'space', 'disk_space': 6.0, 'use_age': True, 'source': 'creation_date', 'reverse': True, 'threshold_behavior': 'greater_than', 'stats_result': 'min_value', 'timestring': None, 'exclude': False}]}}}
2024-03-26 16:01:59,524 DEBUG      curator.cli.ActionDef        log_the_options:225  For action delete_indices: disable_action=Falsecontinue_if_exception=False, timeout_override=Noneignore_empty_list=True, allow_ilm_indices=False
2024-03-26 16:01:59,524 DEBUG      curator.cli.ActionDef        log_the_options:225  For action delete_indices: disable_action=Falsecontinue_if_exception=False, timeout_override=Noneignore_empty_list=True, allow_ilm_indices=False
2024-03-26 16:01:59,524 INFO                 curator.cli                    run:153  Preparing Action ID: 1, "delete_indices"
2024-03-26 16:01:59,524 INFO                 curator.cli                    run:160  Creating client object and testing connection
2024-03-26 16:01:59,524 DEBUG     es_client.helpers.config             get_client:291  Creating client object and testing connection
2024-03-26 16:01:59,525 DEBUG     es_client.helpers.schemacheck               __init__:54   Schema: {'other_settings': {'master_only': <function Boolean at 0x7f79e4137f60>, 'skip_version_test': <function Boolean at 0x7f79e41342c0>, 'username': Any(None, <class 'str'>, msg=None), 'password': Any(None, <class 'str'>, msg=None), 'api_key': {'id': Any(None, <class 'str'>, msg=None), 'api_key': Any(None, <class 'str'>, msg=None), 'token': Any(None, <class 'str'>, msg=None)}}, 'client': {'hosts': Any(None, <class 'list'>, <class 'str'>, msg=None), 'cloud_id': Any(None, <class 'str'>, msg=None), 'api_key': Any(None, <class 'tuple'>, msg=None), 'basic_auth': Any(None, <class 'tuple'>, msg=None), 'bearer_auth': Any(None, <class 'str'>, msg=None), 'opaque_id': Any(None, <class 'str'>, msg=None), 'headers': Any(None, <class 'dict'>, msg=None), 'connections_per_node': Any(None, All(Coerce(int, msg=None), Range(min=1, max=100, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'http_compress': <function Boolean at 0x7f79e4134e00>, 'verify_certs': <function Boolean at 0x7f79e4135300>, 'ca_certs': Any(None, <class 'str'>, msg=None), 'client_cert': Any(None, <class 'str'>, msg=None), 'client_key': Any(None, <class 'str'>, msg=None), 'ssl_assert_hostname': Any(None, <class 'str'>, msg=None), 'ssl_assert_fingerprint': Any(None, <class 'str'>, msg=None), 'ssl_version': Any(None, <class 'str'>, msg=None), 'ssl_context': Any(None, <class 'str'>, msg=None), 'ssl_show_warn': <function Boolean at 0x7f79e41367a0>, 'transport_class': Any(None, <class 'str'>, msg=None), 'request_timeout': Any(None, All(Coerce(float, msg=None), Range(min=0.1, max=86400.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'node_class': Any(None, <class 'str'>, msg=None), 'node_pool_class': Any(None, <class 'str'>, msg=None), 'randomize_nodes_in_pool': <function Boolean at 0x7f79e4136ac0>, 'node_selector_class': Any(None, <class 'str'>, msg=None), 'dead_node_backoff_factor': Any(None, <class 'float'>, msg=None), 'max_dead_node_backoff': Any(None, <class 'float'>, msg=None), 'serializer': Any(None, <class 'str'>, msg=None), 'serializers': Any(None, <class 'dict'>, msg=None), 'default_mimetype': Any(None, <class 'str'>, msg=None), 'max_retries': Any(None, All(Coerce(int, msg=None), Range(min=1, max=100, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'retry_on_status': Any(None, <class 'tuple'>, msg=None), 'retry_on_timeout': <function Boolean at 0x7f79e4135bc0>, 'sniff_on_start': <function Boolean at 0x7f79e4136200>, 'sniff_before_requests': <function Boolean at 0x7f79e4148040>, 'sniff_on_node_failure': <function Boolean at 0x7f79e4148180>, 'sniff_timeout': Any(None, All(Coerce(float, msg=None), Range(min=0.1, max=100.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'min_delay_between_sniffing': Any(None, All(Coerce(float, msg=None), Range(min=1, max=100.0, min_included=True, max_included=True, msg=None), msg=None), msg=None), 'sniffed_node_callback': Any(None, <class 'str'>, msg=None), 'meta_header': <function Boolean at 0x7f79e41484a0>, 'host_info_callback': Any(None, <class 'str'>, msg=None), '_transport': Any(None, <class 'str'>, msg=None)}}
2024-03-26 16:01:59,525 DEBUG     es_client.helpers.schemacheck               __init__:56   "Elasticsearch Configuration" config: {'client': {'hosts': 'https://infra-elasticsearch.infra:9200', 'http_compress': False, 'verify_certs': True}, 'other_settings': {'master_only': False, 'skip_version_test': False, 'username': 'elastic', 'password': 'REDACTED', 'api_key': {}}}
2024-03-26 16:02:00,114 CRITICAL  es_client.helpers.config             get_client:301  Unable to establish client connection to Elasticsearch!
2024-03-26 16:02:00,114 CRITICAL  es_client.helpers.config             get_client:302  Exception encountered: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)))

[untergeek]

No worries. Found it. Will have an update out shortly.

[untergeek]

@rgaduput, I apologize for the inconvenience. This is totally fixed now in Curator 8.0.13, which has been published to PyPI and Docker as well (as untergeek/curator:8.0.13).

Since you're using Kubernetes, you might be interested to know that you can avoid the config YAML file altogether now using exclusively environment variables. You can see these at https://es-client.readthedocs.io/en/latest/envvars.html

[rgaduput]

@untergeek thanks for the quick check and the fix.
sure will check the env vars config.