Merge branch 'master' into 8.x

curator/_version.py

@@ -1,2 +1,2 @@
 """Curator Version"""
-__version__ = '8.0.0rc1'
+__version__ = '8.0.0'

docs/Changelog.rst

@@ -3,6 +3,58 @@
 Changelog
 =========
 
+8.0.0 (31 January 2023)
+-----------------------
+
+**Announcement**
+
+This release is a *major* refactoring of the Curator code to work with both Elasticsearch
+8.x and the Elasticsearch-py Python module of the same major and minor versions.
+
+I apologize for the crazy merge messes trying to get this all to work. In the end, I had to delete
+my fork on github and start over clean.
+
+**Breaking Changes**
+
+  * Curator is now version locked. Curator v8.x will only work with Elasticsearch v8.x
+  * Your old Curator ``config.yml`` file will no longer work as written. There have been more than
+    a few changes necessitated by the updates in the ``elasticsearch8`` Python client library.
+    The client connection code has also been extracted to its own module, ``es_client``. This is
+    actually a good thing, however, as new options for configuring the client connection become
+    possible.
+  * Going forward, Curator will only be released as a tarball via GitHub, as an ``sdist`` or
+    ``wheel`` via ``pip`` on PyPI, and to Docker Hub. There will no longer be RPM, DEB, or Windows
+    ZIP releases. I am sorry if this is inconvenient, but one of the reasons the development and
+    release cycle was delayed so long is because of how painfully difficult it was to do releases.
+  * Curator will only work with Python 3.8+, and will more tightly follow the Python version releases.
+
+**Changes**
+
+  * Last minute doc fixes. Mostly updated links to Elasticsearch documentation.
+  * Python 3.11.1 is fully supported, and all versions of Python 3.8+ should be fully supported.
+  * Use ``hatch`` and ``hatchling`` for package building & publishing
+  * Because of ``hatch`` and ``pyproject.toml``, the release version still only needs to be tracked
+    in ``curator/_version.py``.
+  * Maintain the barest ``setup.py`` for building a binary version of Curator for Docker using
+    ``cx_Freeze``.
+  * Remove ``setup.cfg``, ``requirements.txt``, ``MANIFEST.in``, and other files as functionality
+    is now handled by ``pyproject.toml`` and doing ``pip install .`` to grab dependencies and
+    install them. YAY! Only one place to track dependencies now!!!
+  * Preliminarily updated the docs.
+  * Migrate towards ``pytest`` and away from ``nose`` tests.
+  * Revamped almost every integration test
+  * Scripts provided now that aid in producing and destroying Docker containers for testing. See
+    ``docker_test/scripts/create.sh``. To spin up a numbered version release of Elasticsearch, run
+    ``docker_test/scripts/create.sh 8.6.1``. It will download any necessary images, launch them,
+    and tell you when it's ready, as well as provide ``REMOTE_ES_SERVER`` environment variables for
+    testing the ``reindex`` action, e.g.
+    ``REMOTE_ES_SERVER="http://172.16.0.1:9201" pytest --cov=curator``. These tests are skipped
+    if this value is not provided. To clean up afterwards, run ``docker_test/scripts/destroy.sh``
+  * The action classes were broken into their own path, ``curator/actions/filename.py``.
+  * ``curator_cli`` has been updated with more client connection settings, like ``cloud_id``.
+  * As Curator 8 is version locked and will not use AWS credentials to connect to any ES 8.x
+    instance, all AWS ES connection settings and references have been removed.
+
 8.0.0rc1 (30 January 2023)
 --------------------------
 
@@ -99,21 +151,95 @@ This release-candidate is a *major* refactoring of the Curator code to work with
   * As Curator 8 is version locked and will not use AWS credentials to connect to any ES 8.x
     instance, all AWS ES connection settings and references have been removed.
 
-7.0.0 (x y z)
--------------
+7.0.0 (31 January 2023)
+-----------------------
 
 **Announcement**
 
   * This release is a simplified release for only ``pip`` and Docker. It only works
-    with Elasticsearch 7.x and is functionally identical to 5.8.4 and 5.8.5
+    with Elasticsearch 7.x and is functionally identical to 5.8.4
 
-6.0.0 (x y z)
--------------
+**Breaking Changes**
+
+  * Curator is now version locked. Curator v7.x will only work with Elasticsearch v7.x
+  * Going forward, Curator will only be released as a tarball via GitHub, as an ``sdist`` or
+    ``wheel`` via ``pip`` on PyPI, and to Docker Hub. There will no longer be RPM, DEB, or Windows
+    ZIP releases. I am sorry if this is inconvenient, but one of the reasons the development and
+    release cycle was delayed so long is because of how painfully difficult it was to do releases.
+  * Curator will only work with Python 3.8+, and will more tightly follow the Python version releases.
+
+**New**
+
+  * Python 3.11.1 is fully supported, and all versions of Python 3.8+ should be fully supported.
+  * Use ``hatch`` and ``hatchling`` for package building & publishing
+  * Because of ``hatch`` and ``pyproject.toml``, the release version still only needs to be tracked
+    in ``curator/_version.py``.
+  * Maintain the barest ``setup.py`` for building a binary version of Curator for Docker using
+    ``cx_Freeze``.
+  * Remove ``setup.cfg``, ``requirements.txt``, ``MANIFEST.in``, and other files as functionality
+    is now handled by ``pyproject.toml`` and doing ``pip install .`` to grab dependencies and
+    install them. YAY! Only one place to track dependencies now!!!
+  * Preliminarily updated the docs.
+  * Migrate towards ``pytest`` and away from ``nose`` tests.
+  * Scripts provided now that aid in producing and destroying Docker containers for testing. See
+    ``docker_test/scripts/create.sh``. To spin up a numbered version release of Elasticsearch, run
+    ``docker_test/scripts/create.sh 7.17.8``. It will download any necessary images, launch them,
+    and tell you when it's ready, as well as provide ``REMOTE_ES_SERVER`` environment variables for
+    testing the ``reindex`` action, e.g.
+    ``REMOTE_ES_SERVER="172.16.0.1:9201" pytest --cov=curator``. These tests are skipped
+    if this value is not provided. To clean up afterwards, run ``docker_test/scripts/destroy.sh``
+  * Add filter by size feature. #1612 (IndraGunawan)
+  * Update Elasticsearch client to 7.17.8
+
+**Security Fixes**
+
+  * Use `urllib3` 1.26.5 or higher #1610 (tsaarni) — This dependency is now fully handled by the
+    ``elasticsearch7`` module and not a separate ``urllib3`` import.
+
+6.0.0 (31 January 2023)
+-----------------------
 
 **Announcement**
 
   * This release is a simplified release for only ``pip`` and Docker. It only works
-    with Elasticsearch 6.x and is functionally identical to 5.8.4 and 5.8.5
+    with Elasticsearch 6.x and is functionally identical to 5.8.4
+
+**Breaking Changes**
+
+  * Curator is now version locked. Curator v6.x will only work with Elasticsearch v6.x
+  * Going forward, Curator will only be released as a tarball via GitHub, as an ``sdist`` or
+    ``wheel`` via ``pip`` on PyPI, and to Docker Hub. There will no longer be RPM, DEB, or Windows
+    ZIP releases. I am sorry if this is inconvenient, but one of the reasons the development and
+    release cycle was delayed so long is because of how painfully difficult it was to do releases.
+  * Curator will only work with Python 3.8+, and will more tightly follow the Python version releases.
+
+**New**
+
+  * Python 3.11.1 is fully supported, and all versions of Python 3.8+ should be fully supported.
+  * Use ``hatch`` and ``hatchling`` for package building & publishing
+  * Because of ``hatch`` and ``pyproject.toml``, the release version still only needs to be tracked
+    in ``curator/_version.py``.
+  * Maintain the barest ``setup.py`` for building a binary version of Curator for Docker using
+    ``cx_Freeze``.
+  * Remove ``setup.cfg``, ``requirements.txt``, ``MANIFEST.in``, and other files as functionality
+    is now handled by ``pyproject.toml`` and doing ``pip install .`` to grab dependencies and
+    install them. YAY! Only one place to track dependencies now!!!
+  * Preliminarily updated the docs.
+  * Migrate towards ``pytest`` and away from ``nose`` tests.
+  * Scripts provided now that aid in producing and destroying Docker containers for testing. See
+    ``docker_test/scripts/create.sh``. To spin up a numbered version release of Elasticsearch, run
+    ``docker_test/scripts/create.sh 6.8.23``. It will download any necessary images, launch them,
+    and tell you when it's ready, as well as provide ``REMOTE_ES_SERVER`` environment variables for
+    testing the ``reindex`` action, e.g.
+    ``REMOTE_ES_SERVER="172.16.0.1:9201" pytest --cov=curator``. These tests are skipped
+    if this value is not provided. To clean up afterwards, run ``docker_test/scripts/destroy.sh``
+  * Add filter by size feature. #1612 (IndraGunawan)
+  * Update Elasticsearch client to 6.8.2
+
+**Security Fixes**
+
+  * Use `urllib3` 1.26.5 or higher #1610 (tsaarni) — This dependency is now fully handled by the
+    ``elasticsearch7`` module and not a separate ``urllib3`` import.
 
 5.8.5 (* * *)
 -------------

docs/asciidoc/actions.asciidoc

@@ -328,7 +328,7 @@ options:
 
 For the `create_index` action, the <<option_name,name>> option can be in
 Elasticsearch
-{ref}/date-math-index-names.html[date math] format.  This allows index names
+{ref}/api-conventions.html#api-date-math-index-names[date math] format.  This allows index names
 containing dates to use deterministic math to set a date name in the past or the
 future.
 
@@ -337,7 +337,7 @@ create an index named `logstash-2017.03.27`. If you wanted to create
 _tomorrow's_ index, you would use the name `<logstash-{now/d+1d}>`, which adds 1
 day.  This pattern creates an index named `logstash-2017.03.28`.  For many more
 configuration options, read the Elasticsearch
-{ref}/date-math-index-names.html[date math] documentation.
+{ref}/api-conventions.html#api-date-math-index-names[date math] documentation.
 
 === Extra Settings
 
@@ -670,7 +670,6 @@ this action.  All other options are as follows.
 * <<option_remote_client_cert,remote_client_cert>>
 * <<option_remote_client_key,remote_client_key>>
 * <<option_remote_filters,remote_filters>>
-* <<option_remote_ssl_no_validate,remote_ssl_no_validate>>
 * <<option_remote_url_prefix,remote_url_prefix>>
 * <<option_request_body,request_body>>
 * <<option_requests_per_second,requests_per_second>>
@@ -972,9 +971,8 @@ options:
   index before triggering a rollover.  This _must_ be nested under
   `conditions:`. There is no default value.  If this condition is specified, it
   must have a value, or Curator will generate an error.
-* <<option_max_size,max_size>> **Only supported in Elasticsearch 6.1.0 and up.**
-  The maximum size the index can be before a rollover is triggered.  This _must_
-  be nested under `conditions:`. There is no default value.  If this condition
+* <<option_max_size,max_size>> The maximum size the index can be before a rollover is triggered.
+  This _must_ be nested under `conditions:`. There is no default value.  If this condition
   is specified, it must have a value, or Curator will generate an error.
 
 

docs/asciidoc/filters.asciidoc

@@ -456,7 +456,7 @@ indices which do not contain any documents. Indices that are closed are automati
 removed from consideration. They will remain in, or be removed from the actionable list
 based on the value of <<fe_exclude,exclude>>.
 
-By default the indices matched by the empty filter will be excluded since 
+By default the indices matched by the empty filter will be excluded since
 the exclude setting defaults to True. To include matching indices rather than
 exclude, set the exclude setting to False.
 
@@ -503,7 +503,7 @@ remain in, or be removed from the actionable list based on the value of
   exclude: True
 -------------
 
-This <<filtertype,filtertype>> will remove any index matching the regular 
+This <<filtertype,filtertype>> will remove any index matching the regular
 expression `^\.kibana.*$` from the list of indices, if present.
 
 This <<filtertype,filtertype>> will iterate over the actionable list and match

docs/asciidoc/ilm.asciidoc

@@ -49,7 +49,7 @@ NOTE: All Beats share a similar ILM configuration. Filebeats is used as a
 Starting with version 7.0, Filebeat uses index lifecycle management by default when it connects to a cluster that supports lifecycle management. Filebeat loads the default policy automatically and applies it to any indices created by Filebeat.
 
 You can view and edit the policy in the Index lifecycle policies UI in Kibana. For more information about working with the UI, see
-{kbref}/index-lifecycle-policies.html[Index lifecyle policies].
+{esref}/index-lifecycle-management.html[Index lifecyle policies].
 
 Read more about Filebeat and ILM {fbref}/ilm.html[here].
 

docs/asciidoc/inc_datemath.asciidoc

@@ -1,4 +1,4 @@
-This setting may be a valid {ref}/date-math-index-names.html[Elasticsearch date math string].
+This setting may be a valid {ref}/api-conventions.html#api-date-math-index-names[Elasticsearch date math string].
 
 A date math name takes the following form:
 

docs/asciidoc/inc_filter_by_aliases.asciidoc

@@ -1,11 +1,10 @@
 [IMPORTANT]
-.API Change in Elasticsearch 5.5.0
+.Matching Indices and Aliases
 ============================
-https://www.elastic.co/guide/en/elasticsearch/reference/5.5/breaking-changes-5.5.html#breaking_55_rest_changes[An update to Elasticsearch 5.5.0 changes the behavior of this filter, differing from previous 5.x versions].
+https://www.elastic.co/guide/en/elasticsearch/reference/5.5/breaking-changes-5.5.html#breaking_55_rest_changes[Indices must be in all aliases to match].
 
-If a list of <<fe_aliases,aliases>> is provided (instead of only one), indices
-must appear in _all_ listed <<fe_aliases,aliases>> or a 404 error will result,
-leading to no indices being matched.  In older versions, if the index was
-associated with even one of the aliases in <<fe_aliases,aliases>>, it would
-result in a match.
+If a list of <<fe_aliases,aliases>> is provided (instead of only one), indices must appear in _all_
+listed <<fe_aliases,aliases>> or a 404 error will result, leading to no indices being matched.
+In older versions, if the index was associated with even one of the aliases in
+<<fe_aliases,aliases>>, it would result in a match.
 ============================

docs/asciidoc/index.asciidoc

@@ -1,4 +1,4 @@
-:curator_version: 8.0.0rc1
+:curator_version: 8.0.0
 :curator_major: 8
 :curator_doc_tree: 8.0
 :es_py_version: 8.6.1

docs/asciidoc/options.asciidoc

@@ -39,7 +39,6 @@ Options are settings used by <<actions,actions>>.
 * <<option_remote_client_cert,remote_client_cert>>
 * <<option_remote_client_key,remote_client_key>>
 * <<option_remote_filters,remote_filters>>
-* <<option_remote_ssl_no_validate,remote_ssl_no_validate>>
 * <<option_remote_url_prefix,remote_url_prefix>>
 * <<option_rename_pattern,rename_pattern>>
 * <<option_rename_replacement,rename_replacement>>
@@ -532,12 +531,12 @@ filters:
 
 When the `ignore_unavailable` option is `False` and an index is missing,
 or if the request is to apply a
-https://www.elastic.co/guide/en/elasticsearch/reference/5.4/index-modules.html#_static_index_settings[static]
+{esref}/index-modules.html#_static_index_settings[static]
 setting and the index is opened, the index setting request will fail.
 The `ignore_unavailable` option allows these indices to be skipped, when set to
 `True`.
 
-NOTE: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/index-modules.html#dynamic-index-settings[Dynamic]
+NOTE: {esref}/index-modules.html#dynamic-index-settings[Dynamic]
 index settings can be applied to either open or closed indices.
 
 [[option_include_aliases]]
@@ -791,7 +790,7 @@ NOTE: At least one of <<option_max_age,max_age>>, <<option_max_docs,max_docs>>,
 
 The maximum approximate size an index is allowed to be before triggering a
 rollover.  Sizes must use Elasticsearch approved
-{ref}/common-options.html#byte-units[byte units]. It _must_ be nested under
+{ref}/common-options.html#byte-units[human-readable byte units]. It _must_ be nested under
 `conditions:` There is no default value.  If this condition is specified, it
 must have a value, or Curator will generate an error.
 
@@ -1089,7 +1088,7 @@ options:
   # ...
 -------------
 
-or use Elasticsearch {ref}/date-math-index-names.html[date math]
+or use Elasticsearch {ref}/api-conventions.html#api-date-math-index-names[date math]
 
 [source,yaml]
 -------------
@@ -1530,48 +1529,6 @@ provided, and reindex them to the _local_ cluster as the name provided in the
 `dest` `index`.  In this example, that is `index2`.
 
 
-
-[[option_remote_ssl_no_validate]]
-== remote_ssl_no_validate
-
-This should be `True`, `False` or left empty.
-
-[source,yaml]
--------------
-actions:
-  1:
-    description: "Reindex index1 into index2"
-    action: reindex
-    options:
-      wait_interval: 9
-      max_wait: -1
-      remote_ssl_no_validate: True
-      request_body:
-        source:
-          remote:
-            host: https://otherhost:9200
-          index: index1
-        dest:
-          index: index2
-    filters:
-    - filtertype: none
--------------
-
-
-If access to your Elasticsearch instance is protected by SSL encryption, you may
-set `ssl_no_validate` to `True` to disable SSL certificate verification.
-
-Valid use cases for doing so include the use of self-signed certificates that
-cannot be otherwise verified and would generate error messages.
-
-WARNING: Setting `ssl_no_validate` to `True` will likely result in a warning
-    message that your SSL certificates are not trusted. This is expected
-    behavior.
-
-The default value is `False`.
-
-
-
 [[option_remote_url_prefix]]
 == remote_url_prefix
 
@@ -2310,7 +2267,7 @@ were `-shrink`, the resulting target index name would be `foo-index-shrink`.
 NOTE: This setting is only used by the <<reindex,reindex>> action.
 
 This setting can speed up reindexing operations by using
-{ref}/search-request-body.html#sliced-scroll[Sliced Scroll] to slice on the
+{ref}/paginate-search-results.html#slice-scroll[Sliced Scroll] to slice on the
 \_uid.
 
 [source,yaml]