Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delete with prefix, doesn't work (on its own) #37

Closed
dharrigan opened this issue Feb 10, 2014 · 5 comments
Closed

Delete with prefix, doesn't work (on its own) #37

dharrigan opened this issue Feb 10, 2014 · 5 comments

Comments

@dharrigan
Copy link
Contributor

Hi,

Thanks for the wonderful tool. I'm trying to delete indices matching a pattern with this command:

curator -p .marvel- -d 0

Unfortunately, this doesn't work. It seems to suggest that it would work in the documentation without a prefix, i.e.,

curator --host blah -d 0

Could curator support prefixes?

Thank you.

-=david=-
@untergeek
Copy link
Member

It does work. The problem is the 0. You cannot use -d 0. I suppose that should be better documented.

This was an intentional design choice so that you could not accidentally delete the current index. If you want to delete the current index you will have to use a curl statement: curl -XDELETE localhost:9200/.marvel-2014.02.10?pretty

Perhaps we could correct the script to allow for this in the future, but it was intentional.

Here's what it looks like for me:

curator --host blackbox -c 6 -b 1 -d 7 -p .marvel-
2014-02-10T06:55:27.821 INFO                        main:325  Job starting...
2014-02-10T06:55:27.821 INFO                        main:345  Deleting indices older than 7 days...
2014-02-10T06:55:27.822 INFO                   _new_conn:257  Starting new HTTP connection (1): blackbox
2014-02-10T06:55:27.829 INFO         log_request_success:49   GET http://blackbox:9200/_settings [status:200 request:0.007s]
2014-02-10T06:55:27.833 INFO                  index_loop:290  Attempting to delete index .marvel-2014.01.31 because it is 3 days, 0:00:00 older than cutoff.
2014-02-10T06:55:28.040 INFO         log_request_success:49   DELETE http://blackbox:9200/.marvel-2014.01.31 [status:200 request:0.206s]
2014-02-10T06:55:28.040 INFO                  index_loop:300  .marvel-2014.01.31: Successfully deleted.
2014-02-10T06:55:28.040 INFO                  index_loop:290  Attempting to delete index .marvel-2014.02.01 because it is 2 days, 0:00:00 older than cutoff.
2014-02-10T06:55:28.402 INFO         log_request_success:49   DELETE http://blackbox:9200/.marvel-2014.02.01 [status:200 request:0.362s]
2014-02-10T06:55:28.402 INFO                  index_loop:300  .marvel-2014.02.01: Successfully deleted.
2014-02-10T06:55:28.402 INFO                  index_loop:290  Attempting to delete index .marvel-2014.02.02 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:29.118 INFO         log_request_success:49   DELETE http://blackbox:9200/.marvel-2014.02.02 [status:200 request:0.716s]
2014-02-10T06:55:29.118 INFO                  index_loop:300  .marvel-2014.02.02: Successfully deleted.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.03 is 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.04 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.05 is 2 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.06 is 3 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.07 is 4 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.08 is 5 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.09 is 6 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO        find_expired_indices:188  .marvel-2014.02.10 is 7 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO                  index_loop:301  DELETE index operations completed.
2014-02-10T06:55:29.120 INFO                        main:350  Closing indices older than 6 days...
2014-02-10T06:55:29.122 INFO         log_request_success:49   GET http://blackbox:9200/_settings [status:200 request:0.002s]
2014-02-10T06:55:29.122 INFO                  index_loop:290  Attempting to close index .marvel-2014.02.03 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:29.130 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.03&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.287 INFO         log_request_success:49   POST http://blackbox:9200/.marvel-2014.02.03/_close [status:200 request:0.153s]
2014-02-10T06:55:29.287 INFO                  index_loop:300  .marvel-2014.02.03: Successfully closed.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.04 is 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.05 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.06 is 2 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.07 is 3 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.08 is 4 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.09 is 5 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO        find_expired_indices:188  .marvel-2014.02.10 is 6 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO                  index_loop:301  CLOSE index operations completed.
2014-02-10T06:55:29.289 INFO                        main:355  Disabling bloom filter on indices older than 1 days...
2014-02-10T06:55:29.291 INFO         log_request_success:49   GET http://blackbox:9200/_settings [status:200 request:0.002s]
2014-02-10T06:55:29.291 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.03 because it is 6 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.299 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.03&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.303 INFO                _bloom_index:267  Skipping index .marvel-2014.02.03: Already closed.
2014-02-10T06:55:29.303 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.04 because it is 5 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.312 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.04&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.376 INFO         log_request_success:49   PUT http://blackbox:9200/.marvel-2014.02.04/_settings [status:200 request:0.059s]
2014-02-10T06:55:29.376 INFO                  index_loop:300  .marvel-2014.02.04: Successfully bloom filter disabled.
2014-02-10T06:55:29.376 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.05 because it is 4 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.385 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.05&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.446 INFO         log_request_success:49   PUT http://blackbox:9200/.marvel-2014.02.05/_settings [status:200 request:0.057s]
2014-02-10T06:55:29.447 INFO                  index_loop:300  .marvel-2014.02.05: Successfully bloom filter disabled.
2014-02-10T06:55:29.447 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.06 because it is 3 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.455 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.06&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:31.644 INFO         log_request_success:49   PUT http://blackbox:9200/.marvel-2014.02.06/_settings [status:200 request:2.185s]
2014-02-10T06:55:31.645 INFO                  index_loop:300  .marvel-2014.02.06: Successfully bloom filter disabled.
2014-02-10T06:55:31.645 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.07 because it is 2 days, 0:00:00 older than cutoff.
2014-02-10T06:55:31.653 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.07&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:35.807 INFO         log_request_success:49   PUT http://blackbox:9200/.marvel-2014.02.07/_settings [status:200 request:4.150s]
2014-02-10T06:55:35.807 INFO                  index_loop:300  .marvel-2014.02.07: Successfully bloom filter disabled.
2014-02-10T06:55:35.807 INFO                  index_loop:290  Attempting to disable bloom filter for index .marvel-2014.02.08 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:35.816 INFO         log_request_success:49   GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.08&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:40.046 INFO         log_request_success:49   PUT http://blackbox:9200/.marvel-2014.02.08/_settings [status:200 request:4.226s]
2014-02-10T06:55:40.047 INFO                  index_loop:300  .marvel-2014.02.08: Successfully bloom filter disabled.
2014-02-10T06:55:40.047 INFO        find_expired_indices:188  .marvel-2014.02.09 is 0:00:00 above the cutoff.
2014-02-10T06:55:40.047 INFO        find_expired_indices:188  .marvel-2014.02.10 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:40.047 INFO                  index_loop:301  DISABLE BLOOM FILTER FOR index operations completed.
2014-02-10T06:55:40.047 INFO                        main:365  Done in 0:00:12.228052.

@dharrigan
Copy link
Contributor Author

Hi There!

Thank you, that makes sense. Yes, I reverted to using the CURL command. I
suppose it might be useful to have a "-f" for force if value is 0. It's
useful at times to completely remove the index, but meh, it's neither
here or there since I can use CURL if required.

Thank you again for a great tool! :-)

-=david=-

On 10 February 2014 12:57, Aaron Mildenstein notifications@github.comwrote:

It does work. The problem is the 0. You cannot use -d 0. I suppose that
should be better documented.

This was an intentional design choice so that you could not accidentally
delete the current index. If you want to delete the current index you will
have to use a curl statement: curl -XDELETE
localhost:9200/.marvel-2014.02.10?pretty

Perhaps we could correct the script to allow for this in the future, but
it was intentional.

Here's what it looks like for me:

curator --host blackbox -c 6 -b 1 -d 7 -p .marvel-
2014-02-10T06:55:27.821 INFO main:325 Job starting...
2014-02-10T06:55:27.821 INFO main:345 Deleting indices older than 7 days...
2014-02-10T06:55:27.822 INFO _new_conn:257 Starting new HTTP connection (1): blackbox
2014-02-10T06:55:27.829 INFO log_request_success:49 GET http://blackbox:9200/_settings [status:200 request:0.007s]
2014-02-10T06:55:27.833 INFO index_loop:290 Attempting to delete index .marvel-2014.01.31 because it is 3 days, 0:00:00 older than cutoff.
2014-02-10T06:55:28.040 INFO log_request_success:49 DELETE http://blackbox:9200/.marvel-2014.01.31 [status:200 request:0.206s]
2014-02-10T06:55:28.040 INFO index_loop:300 .marvel-2014.01.31: Successfully deleted.
2014-02-10T06:55:28.040 INFO index_loop:290 Attempting to delete index .marvel-2014.02.01 because it is 2 days, 0:00:00 older than cutoff.
2014-02-10T06:55:28.402 INFO log_request_success:49 DELETE http://blackbox:9200/.marvel-2014.02.01 [status:200 request:0.362s]
2014-02-10T06:55:28.402 INFO index_loop:300 .marvel-2014.02.01: Successfully deleted.
2014-02-10T06:55:28.402 INFO index_loop:290 Attempting to delete index .marvel-2014.02.02 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:29.118 INFO log_request_success:49 DELETE http://blackbox:9200/.marvel-2014.02.02 [status:200 request:0.716s]
2014-02-10T06:55:29.118 INFO index_loop:300 .marvel-2014.02.02: Successfully deleted.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.03 is 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.04 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.05 is 2 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.06 is 3 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.07 is 4 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.08 is 5 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.09 is 6 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO find_expired_indices:188 .marvel-2014.02.10 is 7 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.119 INFO index_loop:301 DELETE index operations completed.
2014-02-10T06:55:29.120 INFO main:350 Closing indices older than 6 days...
2014-02-10T06:55:29.122 INFO log_request_success:49 GET http://blackbox:9200/_settings [status:200 request:0.002s]
2014-02-10T06:55:29.122 INFO index_loop:290 Attempting to close index .marvel-2014.02.03 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:29.130 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.03&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.287 INFO log_request_success:49 POST http://blackbox:9200/.marvel-2014.02.03/_close [status:200 request:0.153s]
2014-02-10T06:55:29.287 INFO index_loop:300 .marvel-2014.02.03: Successfully closed.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.04 is 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.05 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.06 is 2 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.07 is 3 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.08 is 4 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.09 is 5 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO find_expired_indices:188 .marvel-2014.02.10 is 6 days, 0:00:00 above the cutoff.
2014-02-10T06:55:29.288 INFO index_loop:301 CLOSE index operations completed.
2014-02-10T06:55:29.289 INFO main:355 Disabling bloom filter on indices older than 1 days...
2014-02-10T06:55:29.291 INFO log_request_success:49 GET http://blackbox:9200/_settings [status:200 request:0.002s]
2014-02-10T06:55:29.291 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.03 because it is 6 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.299 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.03&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.303 INFO _bloom_index:267 Skipping index .marvel-2014.02.03: Already closed.
2014-02-10T06:55:29.303 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.04 because it is 5 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.312 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.04&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.376 INFO log_request_success:49 PUT http://blackbox:9200/.marvel-2014.02.04/_settings [status:200 request:0.059s]
2014-02-10T06:55:29.376 INFO index_loop:300 .marvel-2014.02.04: Successfully bloom filter disabled.
2014-02-10T06:55:29.376 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.05 because it is 4 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.385 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.05&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:29.446 INFO log_request_success:49 PUT http://blackbox:9200/.marvel-2014.02.05/_settings [status:200 request:0.057s]
2014-02-10T06:55:29.447 INFO index_loop:300 .marvel-2014.02.05: Successfully bloom filter disabled.
2014-02-10T06:55:29.447 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.06 because it is 3 days, 0:00:00 older than cutoff.
2014-02-10T06:55:29.455 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.06&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:31.644 INFO log_request_success:49 PUT http://blackbox:9200/.marvel-2014.02.06/_settings [status:200 request:2.185s]
2014-02-10T06:55:31.645 INFO index_loop:300 .marvel-2014.02.06: Successfully bloom filter disabled.
2014-02-10T06:55:31.645 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.07 because it is 2 days, 0:00:00 older than cutoff.
2014-02-10T06:55:31.653 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.07&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:35.807 INFO log_request_success:49 PUT http://blackbox:9200/.marvel-2014.02.07/_settings [status:200 request:4.150s]
2014-02-10T06:55:35.807 INFO index_loop:300 .marvel-2014.02.07: Successfully bloom filter disabled.
2014-02-10T06:55:35.807 INFO index_loop:290 Attempting to disable bloom filter for index .marvel-2014.02.08 because it is 1 day, 0:00:00 older than cutoff.
2014-02-10T06:55:35.816 INFO log_request_success:49 GET http://blackbox:9200/_cluster/state?filter_nodes=true&filter_routing_table=true&filter_indices=.marvel-2014.02.08&filter_index_templates=true&filter_blocks=true [status:200 request:0.008s]
2014-02-10T06:55:40.046 INFO log_request_success:49 PUT http://blackbox:9200/.marvel-2014.02.08/_settings [status:200 request:4.226s]
2014-02-10T06:55:40.047 INFO index_loop:300 .marvel-2014.02.08: Successfully bloom filter disabled.
2014-02-10T06:55:40.047 INFO find_expired_indices:188 .marvel-2014.02.09 is 0:00:00 above the cutoff.
2014-02-10T06:55:40.047 INFO find_expired_indices:188 .marvel-2014.02.10 is 1 day, 0:00:00 above the cutoff.
2014-02-10T06:55:40.047 INFO index_loop:301 DISABLE BLOOM FILTER FOR index operations completed.
2014-02-10T06:55:40.047 INFO main:365 Done in 0:00:12.228052.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/37#issuecomment-34627923
.

I prefer encrypted and signed messages. KeyID: B20A22F9
Fingerprint: 110A F423 3647 54E2 880F ADAD 1C52 85BF B20A 22F9

"It is not usually until you've built and used a version of the program
that you understand the issues well enough to get the design right." - Rob
Pike, Brian Kernighan.

No trees were harmed in the sending of this message, however, a number of
electrons were inconvenienced.

@untergeek
Copy link
Member

I'm closing this since I went through the code and found I do already have a message that indicates this:

messages.append('Values for --delete, --close, --bloom, --optimize must be > 0')

@dharrigan
Copy link
Contributor Author

May I suggest a slight grammar improvement,

If you have the sentence read "Values for --delete, --close, --bloom or --optimize must be > 0" then it's a bit easier to comprehend quickly :-)

-=david=-

On 10 February 2014 13:02, Aaron Mildenstein notifications@github.comwrote:

Closed #37 #37.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/37
.

I prefer encrypted and signed messages. KeyID: B20A22F9
Fingerprint: 110A F423 3647 54E2 880F ADAD 1C52 85BF B20A 22F9

"It is not usually until you've built and used a version of the program
that you understand the issues well enough to get the design right." - Rob
Pike, Brian Kernighan.

No trees were harmed in the sending of this message, however, a number of
electrons were inconvenienced.

@untergeek
Copy link
Member

Pull requests are cool 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dharrigan @untergeek