New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix the hang of the shring method with ES 7.x #1528
Fix the hang of the shring method with ES 7.x #1528
Conversation
Tests are failing against ES 6.0.1:
It may be that the |
All right, I will fix it, thx. |
curator/actions.py
Outdated
if extra_settings: | ||
self._merge_extra_settings(extra_settings) | ||
|
||
if utils.get_version(self.client) > (6, 0, 1): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please make this >= (6, 1, 0)
as all of the 6.0.x versions will have the API difference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Thank you, the fix works, but please change the version check as indicated. |
You're welcome, thank you for this tool. |
* Fix the hang of the shring method with ES 7.x * simple quotes instead of double quotes used for the json body object * Remove settings from source index only when ES version supports it (> 6.0.1) * Remove settings from source index only when ES version supports it (>= 6.1.0)
* Code: 401. Error: {"Message":"settings.role_arn is needed for snapshot registration."}. * Docs: Fix bad delimiter Asciidoctor needs the start and end delimiters for blocks to line up perfectly. So this fixes a block that doesn't match. * If no documents processed in reindex, do not fail * Fix wrong number of params * Should have not removed the task assignment * Using logger instead of self.loggit * Grammar correction of ilm.asciidoc * [DOCS] Fixes broken link * [DOCS] Fix broken link * Exclude closed indices from consideration when using the empty filter * Update the docs regarding the change to the empty filtertype * Catch all .kibana* indices This fixes the issue many users upgrading to 7.x encountered. Fixes #1363 * Filter closed indices before performing an empty list check * Document the bug fix in the changelog * Changelog + Reindex Exception test coverage * Add PR number to the Changelog * Fix shrink action to count only primary shard sizes It was considering replicas until this fix. Fixes #1431 * Disallow ILM by default with the rollover action Now `allow_ilm_indices` affects rollover aliases as would be expected. Do not test ILM in versions predating ES 6.6 Fixes #1418 * Update get_client and cli.py client logic This is an attempt to make more clear what's going on behind the scenes in issues like #1434. This may or may not fix said issues, but it should improve visibility enough to take additional steps, if needed. Marking the issue as closed, as any further issues will be different. Fixes #1434 * add skip_flush option for close action * add changelog entry * Allow connection through proxies Initially raised in #510, the issue was addressed in part by @raynigon in #1259. Unfortunately, the one line change was not able to be merged due to many other changes since August 2018. Adaptation and fixes were necessary. Fixes #510 * Catch empty/None values for snapshots without ages This prevents the error described in #1366 where this value did not exist. Fixes #1366 * Report 40x errors properly When connecting in the `get_client()` function, ensure that all TransportError type exceptions are properly caught and the messages reported properly, too. Fixes #1413 * Fix crashes in restore of "large" number of indices * Split to query "chunks" of 3KB of indices at a time * The logic checks restoration of each of the chunks * Fixes #1360 * Add ignore_existing option to create_index Allow existing indices to be skipped gracefully, if `ignore_existing` is set to `true` in the options. Other stuff is Travis CI version backlogs and version bumpage for pending 5.8 release. Fixes #1352 * Remove AWS fail refs * [DOCS] Fix broken link * bugfix: should check if list is empty. * The change was already added. Update the Changelog This is tied to #1448 * PR for adding freeze/unfreeze indices support to curator (#1454) * Commit for all the code changes * Integration tests added * Examples added * Contributor list updated * Corrected typos * Removing test file * Test fixed for supporting older versions * Test fixed for supporting older version * Changelog.rst updated * Changelog.rst updated * Indentation fix in changelog.rst * Allow the close action to ignore sync flush fails (#1460) If closing a very large batch of indices, potentially with some being live at runtime, a failure to perform a synced flush is a real possibility. This new option for the ``close`` action, ``ignore_sync_failures``, will allow index closures to continue. Fixes #1248 * Release prep for 5.8.0 (#1461) * Add Python 3.7, client creation fix, delay release (#1462) The release is bumped back as I figure out why the frozen build is not able to see the `requests` module, when it's clearly present. Update dependencies for Python 3.7 Which hopefully will correct build issues for pyyaml in Travis-CI * changes_done_#1335 (#1466) * Release prep for 5.8.0 (again) (#1468) This time it should be for real. * Version bump for new release build process. (#1469) The actual release scripts will be updated in the future and committed to the repository. For now, suffice to say that they've been tested, and the LD_LIBRARY_PATH is no longer set system-wide, but rather in the `/usr/bin` scripts themselves, which call the binaries in `/opt/elasticsearch-curator` * Next release dev branch version bump (#1470) * Add username and password to client configuration (#1473) Deprecate http_auth in favor of username and password. Lots of code clean-up. `get_client` was broken out into separate functions to clean it up, as it was enormous. Ignore module import monkey-patch for Python 2 Python2 is no longer going to be supported, either by Python or by Curator after 2019-12-31 * More code cleanup. (#1474) Aiming for PEP-8. * Updated for PEP-8 while testing sdh/8392 (#1475) Tried (unsuccessfully) to replicate the failure in sdh/8392. Committing the code cleanup anyway. * Improve code compliance with PEP-8 (cont'd) (#1477) More of the same code cleanup. This time for the singletons. * More code cleanup. (#1481) Cleaned up the defaults this time. * Update Dockerfile (#1485) Build tested, verified, and pushed * There was a “the” hanging in no mans land (#1501) * Pinning 7.1.0 elasticsearch-py module (#1527) Can't release until this is done. Many other PRs will need this before they can run. Fix ES 7.6 ilm-tracking indices causing test fails * [DOC] Freeze and unfreeze documentation (#1497) * Documentation for freeze/unfreeze Series of changes for Freeze/Unfreeze Doc * Freeze/Unfreeze doc changes * Add freeze/unfreeze action doc * Curator 5.6 is not compatible with ES 7.2 (#1522) * Curator 5.6 is not compatible with ES 7.2 * Pinning 7.1.0 elasticsearch-py module (#1527) Can't release until this is done. Many other PRs will need this before they can run. Fix ES 7.6 ilm-tracking indices causing test fails * [DOC] Freeze and unfreeze documentation (#1497) * Documentation for freeze/unfreeze Series of changes for Freeze/Unfreeze Doc * Freeze/Unfreeze doc changes * Add freeze/unfreeze action doc * Curator 5.6 is not compatible with ES 7.2 Co-authored-by: Aaron Mildenstein <aaron@mildensteins.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * Fix the hang of the shring method with ES 7.x (#1528) * Fix the hang of the shring method with ES 7.x * simple quotes instead of double quotes used for the json body object * Remove settings from source index only when ES version supports it (> 6.0.1) * Remove settings from source index only when ES version supports it (>= 6.1.0) * Add ECS log format (#1529) * Ensure proper API call is used (#1530) In older releases (< 7.5) of ES Python module, `client.tasks.get()` will return a list of all tasks. However, starting in 7.5, this results in a failed API call. See elastic/elasticsearch-py#1110 Correct behavior is to use `client.tasks.list()` when no task id is provided. * Remove a few errant commas (#1523) * Code: 401. Error: {"Message":"settings.role_arn is needed for snapshot registration."}. Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com> Co-authored-by: Slavik <Slavikca@gmail.com> Co-authored-by: lcawl <lcawley@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Mike DuRussel <mdurussel@squarespace.com> Co-authored-by: Aaron Mildenstein <aaron@mildensteins.com> Co-authored-by: Mike DuRussel <heyitsmdr@gmail.com> Co-authored-by: Loet Avramson <loet@forter.com> Co-authored-by: Anand Singh Kunwar <anandkunwar95@gmail.com> Co-authored-by: Xinxin Huo <xinxinh4@gmail.com> Co-authored-by: junmuz <mjunaidmuzammil@gmail.com> Co-authored-by: Harsh Bajaj <35989538+harshbajaj16@users.noreply.github.com> Co-authored-by: Greg Trahair <greg.trahair@gmail.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Jibu Chacko <jibsonline@gmail.com> Co-authored-by: Jean-Charles Legras <jcaragog@gmail.com> Co-authored-by: Michael Sverdlik <m1keil@users.noreply.github.com> Co-authored-by: acrewdson <acrewdson@users.noreply.github.com>
* Fix the hang of the shring method with ES 7.x * simple quotes instead of double quotes used for the json body object * Remove settings from source index only when ES version supports it (> 6.0.1) * Remove settings from source index only when ES version supports it (>= 6.1.0)
* Code: 401. Error: {"Message":"settings.role_arn is needed for snapshot registration."}. * Docs: Fix bad delimiter Asciidoctor needs the start and end delimiters for blocks to line up perfectly. So this fixes a block that doesn't match. * If no documents processed in reindex, do not fail * Fix wrong number of params * Should have not removed the task assignment * Using logger instead of self.loggit * Grammar correction of ilm.asciidoc * [DOCS] Fixes broken link * [DOCS] Fix broken link * Exclude closed indices from consideration when using the empty filter * Update the docs regarding the change to the empty filtertype * Catch all .kibana* indices This fixes the issue many users upgrading to 7.x encountered. Fixes elastic#1363 * Filter closed indices before performing an empty list check * Document the bug fix in the changelog * Changelog + Reindex Exception test coverage * Add PR number to the Changelog * Fix shrink action to count only primary shard sizes It was considering replicas until this fix. Fixes elastic#1431 * Disallow ILM by default with the rollover action Now `allow_ilm_indices` affects rollover aliases as would be expected. Do not test ILM in versions predating ES 6.6 Fixes elastic#1418 * Update get_client and cli.py client logic This is an attempt to make more clear what's going on behind the scenes in issues like elastic#1434. This may or may not fix said issues, but it should improve visibility enough to take additional steps, if needed. Marking the issue as closed, as any further issues will be different. Fixes elastic#1434 * add skip_flush option for close action * add changelog entry * Allow connection through proxies Initially raised in elastic#510, the issue was addressed in part by @raynigon in elastic#1259. Unfortunately, the one line change was not able to be merged due to many other changes since August 2018. Adaptation and fixes were necessary. Fixes elastic#510 * Catch empty/None values for snapshots without ages This prevents the error described in elastic#1366 where this value did not exist. Fixes elastic#1366 * Report 40x errors properly When connecting in the `get_client()` function, ensure that all TransportError type exceptions are properly caught and the messages reported properly, too. Fixes elastic#1413 * Fix crashes in restore of "large" number of indices * Split to query "chunks" of 3KB of indices at a time * The logic checks restoration of each of the chunks * Fixes elastic#1360 * Add ignore_existing option to create_index Allow existing indices to be skipped gracefully, if `ignore_existing` is set to `true` in the options. Other stuff is Travis CI version backlogs and version bumpage for pending 5.8 release. Fixes elastic#1352 * Remove AWS fail refs * [DOCS] Fix broken link * bugfix: should check if list is empty. * The change was already added. Update the Changelog This is tied to elastic#1448 * PR for adding freeze/unfreeze indices support to curator (elastic#1454) * Commit for all the code changes * Integration tests added * Examples added * Contributor list updated * Corrected typos * Removing test file * Test fixed for supporting older versions * Test fixed for supporting older version * Changelog.rst updated * Changelog.rst updated * Indentation fix in changelog.rst * Allow the close action to ignore sync flush fails (elastic#1460) If closing a very large batch of indices, potentially with some being live at runtime, a failure to perform a synced flush is a real possibility. This new option for the ``close`` action, ``ignore_sync_failures``, will allow index closures to continue. Fixes elastic#1248 * Release prep for 5.8.0 (elastic#1461) * Add Python 3.7, client creation fix, delay release (elastic#1462) The release is bumped back as I figure out why the frozen build is not able to see the `requests` module, when it's clearly present. Update dependencies for Python 3.7 Which hopefully will correct build issues for pyyaml in Travis-CI * changes_done_#1335 (elastic#1466) * Release prep for 5.8.0 (again) (elastic#1468) This time it should be for real. * Version bump for new release build process. (elastic#1469) The actual release scripts will be updated in the future and committed to the repository. For now, suffice to say that they've been tested, and the LD_LIBRARY_PATH is no longer set system-wide, but rather in the `/usr/bin` scripts themselves, which call the binaries in `/opt/elasticsearch-curator` * Next release dev branch version bump (elastic#1470) * Add username and password to client configuration (elastic#1473) Deprecate http_auth in favor of username and password. Lots of code clean-up. `get_client` was broken out into separate functions to clean it up, as it was enormous. Ignore module import monkey-patch for Python 2 Python2 is no longer going to be supported, either by Python or by Curator after 2019-12-31 * More code cleanup. (elastic#1474) Aiming for PEP-8. * Updated for PEP-8 while testing sdh/8392 (elastic#1475) Tried (unsuccessfully) to replicate the failure in sdh/8392. Committing the code cleanup anyway. * Improve code compliance with PEP-8 (cont'd) (elastic#1477) More of the same code cleanup. This time for the singletons. * More code cleanup. (elastic#1481) Cleaned up the defaults this time. * Update Dockerfile (elastic#1485) Build tested, verified, and pushed * There was a “the” hanging in no mans land (elastic#1501) * Pinning 7.1.0 elasticsearch-py module (elastic#1527) Can't release until this is done. Many other PRs will need this before they can run. Fix ES 7.6 ilm-tracking indices causing test fails * [DOC] Freeze and unfreeze documentation (elastic#1497) * Documentation for freeze/unfreeze Series of changes for Freeze/Unfreeze Doc * Freeze/Unfreeze doc changes * Add freeze/unfreeze action doc * Curator 5.6 is not compatible with ES 7.2 (elastic#1522) * Curator 5.6 is not compatible with ES 7.2 * Pinning 7.1.0 elasticsearch-py module (elastic#1527) Can't release until this is done. Many other PRs will need this before they can run. Fix ES 7.6 ilm-tracking indices causing test fails * [DOC] Freeze and unfreeze documentation (elastic#1497) * Documentation for freeze/unfreeze Series of changes for Freeze/Unfreeze Doc * Freeze/Unfreeze doc changes * Add freeze/unfreeze action doc * Curator 5.6 is not compatible with ES 7.2 Co-authored-by: Aaron Mildenstein <aaron@mildensteins.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> * Fix the hang of the shring method with ES 7.x (elastic#1528) * Fix the hang of the shring method with ES 7.x * simple quotes instead of double quotes used for the json body object * Remove settings from source index only when ES version supports it (> 6.0.1) * Remove settings from source index only when ES version supports it (>= 6.1.0) * Add ECS log format (elastic#1529) * Ensure proper API call is used (elastic#1530) In older releases (< 7.5) of ES Python module, `client.tasks.get()` will return a list of all tasks. However, starting in 7.5, this results in a failed API call. See elastic/elasticsearch-py#1110 Correct behavior is to use `client.tasks.list()` when no task id is provided. * Remove a few errant commas (elastic#1523) * Code: 401. Error: {"Message":"settings.role_arn is needed for snapshot registration."}. Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com> Co-authored-by: Slavik <Slavikca@gmail.com> Co-authored-by: lcawl <lcawley@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: Mike DuRussel <mdurussel@squarespace.com> Co-authored-by: Aaron Mildenstein <aaron@mildensteins.com> Co-authored-by: Mike DuRussel <heyitsmdr@gmail.com> Co-authored-by: Loet Avramson <loet@forter.com> Co-authored-by: Anand Singh Kunwar <anandkunwar95@gmail.com> Co-authored-by: Xinxin Huo <xinxinh4@gmail.com> Co-authored-by: junmuz <mjunaidmuzammil@gmail.com> Co-authored-by: Harsh Bajaj <35989538+harshbajaj16@users.noreply.github.com> Co-authored-by: Greg Trahair <greg.trahair@gmail.com> Co-authored-by: Luca Belluccini <luca.belluccini@elastic.co> Co-authored-by: Jibu Chacko <jibsonline@gmail.com> Co-authored-by: Jean-Charles Legras <jcaragog@gmail.com> Co-authored-by: Michael Sverdlik <m1keil@users.noreply.github.com> Co-authored-by: acrewdson <acrewdson@users.noreply.github.com>
Fixes #
Proposed Changes
Related to issue #1494.
With this fix, when we're shrinking we're doing as shown by the ES documentation :
Hence, the shrink index operation doesn't hang anymore because of the routing requirement.
It seems that all tests be played against a single ES instance.
This use case requires a cluster in order to reproduce this error with integration tests (need help).