Curator not compatible with elasticsearch 8.0.0

[wargamez]

in settings.py there is a check that prohibits curator to run against elasticsearch >=8.0.0. Can we get a new release that bumps that to 8?

[soloyes]

Please could u speed up with the solution? Due to the Elastic 8 being released a long time ago.

[untergeek]

I cannot promise any rapid release of a version of Curator that supports ES 8. There are multiple API calls that have changed, and those will have to be addressed.

What is in Curator that is not in ILM and SLM that you require Curator for? If you're using ES 8, you have both ILM and SLM to cover a great many things, and while I know the feature set isn't perfectly aligned between the two, it seems unusual to me that you would require Curator, so you have piqued my curiosity.

[soloyes]

I am using a curator for the daily clearing of indices. I am rising it via cron and doing the cleaning.

[untergeek]

Does ILM not work for you? ILM's time-based policies should be able to accomplish this for you in a way that does not require cron.

The bottom line is that Curator was created long before ILM existed, and is effectively deprecated for most index management use cases in favor of ILM. Curator remains for use cases that ILM cannot address, and for older clusters that do not support ILM.

[soloyes]

I will try, i will be back.

[hamishforbes]

I'm still using Curator because of this limitation in ILM.
TBH my use case is pretty basic, (count indices with prefix, delete those over n) so if Curator isn't going to be supported in future a custom script shouldn't be hard.

It would be nice to just continue using Curator though!

edit: FWIW this is my 'fix' as none of the Elastic APIs i'm using have changed

sed -ri 's/return \(7, 99, 99\)/return \(8, 99, 99\)/g' /usr/local/lib/python3.6/site-packages/curator/defaults/settings.py

[khanhibus]

It would definitely be nice to have Curator be maintained to support Elasticsearch 8.x. Using ILM requires that an index template be created for each index, which is not always desirable or feasible. I manage 8 separate instances of Elasticsearch for various production, staging, and development workloads. To be able to simply delete (prune older) indices of certain types by editing a Curator yml file is significantly less cumbersome and more convenient than setting up index templates for each separate instance.

I hope you reconsider keeping this maintained, or consider providing a minor update to be able to delete indices on Elasticsearch 8.x.

Thank you.

[alex2013a90]

I'm still using Curator because of this limitation in ILM. TBH my use case is pretty basic, (count indices with prefix, delete those over n) so if Curator isn't going to be supported in future a custom script shouldn't be hard.

It would be nice to just continue using Curator though!

edit: FWIW this is my 'fix' as none of the Elastic APIs i'm using have changed

sed -ri 's/return \(7, 99, 99\)/return \(8, 99, 99\)/g' /usr/local/lib/python3.6/site-packages/curator/defaults/settings.py

Hey
Help please, I switched to ELC 8.1.2 and my Curator stopped working
i found the file /opt/elasticsearch-curator/lib/curator/defaults/settings.pyc

but it is binary
Tell me how you managed to make changes to the binary and run it. I will be grateful for help

[alex2013a90]

Can anyone help, I downloaded the archive:
curator-5.8.4.tar.gz
inside it, take the file:
curator-5.8.4.tar.gz\curator-5.8.4.tar\curator-5.8.4\curator\defaults\settings.py
open with notepad and correct version (return (7, 99, 99) to return (8, 99, 99)
Next, make the binary with the command python3 -m py_compile ss.py
replace the new binary in /opt/elasticsearch-curator/lib/curator/defaults/settings.pyc
After that, try to run the Curator again

[khanhibus]

@alex2013a90, I compiled the edited settings.py file and tested it on some Windows hosts running Elasticsearch 8.1.2 successfully. I don't have a Linux box to test against but feel free to use the attachment, which contains the recompiled settings.py file for Curator 5.8.4.

settings.zip

[AdminValida]

We are also using Curator to snapshot indeces and delete old ones.

We just upgraded to ES version 8 and run into the same problem here...

Regards

[archon810]

We rely on curator to avoid running out of space and were about to finally upgrade to ES v8. Thankfully, I found this ticket. How is curator still not compatible with v8+ yet?

[darootler]

Just for info: We used snapshot and delete indeces with Curator. I contacted ES support and they recommended to use ILM and SLM, so we just ditched Curator and switched to ILM and SLM.

Regards
Richard

[archon810]

We specifically used the curator instead of ILM because to my knowledge, ILM can't maintain an aggregate max index size.

So we're doing this:

    actions:
      1:
        action: delete_indices
        description: >-
          Delete indices if total size of logs-bla-* is greater than 400GB.
        options:
          ignore_empty_list: True
          timeout_override:
          continue_if_exception: False
          disable_action: false
          allow_ilm_indices: true
        filters:
        - filtertype: pattern
          kind: prefix
          value: logs-bla-
          exclude:
        - filtertype: space
          disk_space: 400

If this is possible now with ILM or SLM, I'm all ears.

[darootler]

Well, we don't have this use case. But check out the ILM docs, maybe this is doable with ILM.

Regards
Richard

[artificial-aidan]

It is not. ILM is time based, not size based.

[anjia0532]

My Gist https://gist.github.com/anjia0532/d8e779c97333d6fc0d045675e18e1224

Chinese Blog 074-Elastic Curator 支持ES8.x

docker pull anjia0532/curator:5.8.5

OR

docker build . -t curator:5.8.5

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch && \
     apk del .build-deps 

RUN cd /tmp/curator && python setup.py install

# Thanks for @arslanbekov
RUN  rm -rf /tmp/curator

[arslanbekov]

@anjia0532 Thanks!

But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator, and then run python setup.py

Correct script:

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

[anjia0532]

@anjia0532 Thanks!

But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator, and then run python setup.py

Correct script:

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

Thanks And Fixed it.

[Ekimik]

I would also appreciate if Curator will support Elasticsearch 8. I have created simple library, which helps us automatize indices and mappings changes. Our indices use index and component templates, but are not time-based neither size-based (so afaik ILM is not an option for us), but we have multiple environments and making changes manually is very annoying.

[kvokka]

@untergeek May I kindly ask if there is any movement about this issue?

ILM does NOT allow the removal of old indexes based on the disk used space, so if the cluster is running out of space it is throwing out the fresh logs instead of the old ones, and this is the only feature that I require from the curator.

Ofc it is possible to use the hacks advice above, but it feels so ridiculously wrong.

If the company is not willing to maintain this repo, then, please, allow someone else to do this job, officially.

Thank you

[Carmelly212]

Hi there, can make ILM work for me. Would thank you dearly if you can speed up the support for vesion 8 !

Also,
Anyone found an acceptable workaround for version 8 ?

[untergeek]

Update for all watching this issue:

Work on getting Curator to work with Elasticsearch 8 is currently underway, aggressively. You should know, however, that it will not be compatible with older configuration files, though action files should still work the same way. The reason for this is that Elasticsearch 8 has an updated python client which has been updated (majorly) to remove older methods, and to name functions, methods, classes, etc. to conform to Elasticsearch native patterns. Additionally, Elasticsearch 8 has not only deprecated (which was done in Elasticsearch 7) but fully removed doc_type as an argument from API calls. This requires extensive refactoring of the Curator code (though mostly in tests) in order to work. Other API calls are similarly changing.

The changes to the python Elasticsearch 8 client are so substantial that I'm completely removing the client connection code from Curator in favor of making that its own module, es_client. Work on that is nearly complete, though I have some more tests to write and more docs to update and/or write. This means that api keys, headers, and several other things will now just work out of the box. This also means, however, that Curator ceases to be reverse compatible with older versions. What will this mean for support for older versions?

Major version parity will be by major version names.

Work on both Curator 6 and Curator 7 is effectively complete, and they are in the 6.0 and 7.x branches already, respectively. No releases have been made yet, however, as documentation needs to be updated. As with Curator 8, they will only work within the same major version, i.e. Curator 7.x will only work with Elasticsearch 7.x clusters, though Curator 6 retains reverse compatibility with both Elasticsearch 6 and Elasticsearch 5 clusters. As both Elasticsearch 6 and Elasticsearch 7 are effectively EOL (though Elasticsearch 7 continues to receive security patches and other updates for a time in the 7.17.x branch), I don't expect Curator 6 and 7 to change much other than to fix execution bugs and to update dependencies to newer versions as needed for security patches and such.

It should also be noted that I will no longer be compiling and packaging Curator releases into DEB, RPM, or Windows MSI or Zip packages. Curator will be released to PyPi and Docker Hub (my personal repository there), and instructions will be shared for those who wish to build their own DEB/RPM/MSI/ZIP packages. The overhead of build-testing and maintaining the package repositories is part of the reason for slow release pacing, and I'd rather offer faster, more consistent updates without those luxuries than keep up the effort to maintain them. The Dockerfile in the root directory will be the one used for publication, so if you want to build your own you can use that, or alter it for your own purposes.

As Travis CI is also something of a luxury for me, I will be testing releases against local Docker images, and including the scripts necessary to build them so that it is easy for anyone to do likewise. If I can keep it working with Travis CI, I will, but it, too is a bit slower on the draw, so I'd just as soon keep this small, knowable, and nimble.

Other changes may follow. I was disappointed to learn that Python 3.10 has altered Collections such that I cannot easily make Curator work with it, though I haven't given up. I need to get Curator functional on Python 3.9 before I worry about 3.10+ support, so please be patient. The worry is that once I get it to work on 3.10, it may not work on 3.9, so there is that to consider.

Thank you all for your patience as I have kept this project on hold for so long. A lot has been going on in everyone's lives since the last Curator release, and I am no exception. I'm now in a place where I have more time and autonomy so I can dedicate my personal resources to this, and I'm excited to move it forward.

[untergeek]

UPDATE:

I have received some timely feedback from @sethmlarson and now believe I have a clean way forward to make it work with Python 3.10. You'll see it in the branch if I do get it there.

[wgebis]

@untergeek Hi there! Any updates in terms of new curator release?

[untergeek]

I just finished 2 weeks off work on vacation, followed immediately by my wife having surgery. I have fallen behind somewhat. No ETA yet. Tons of refactoring still to do.

[Carmelly212]

I just finished 2 weeks off work on vacation, followed immediately by my wife having surgery. I have fallen behind somewhat. No ETA yet. Tons of refactoring still to do.

Thank you for your effort, should we be expecting it by Xmas ?

[data-dude]

My backup plan is just to use a simple Python script to delete old indices.

/_cat/indices?format=json
for e in resp.json()
    if 'myindexprefix' in e['index']  && older than ndays 
        requests.delete

It's kind of surprising that no one has created a dirt simple curator in github

[lsoica]

@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator, and then run python setup.py
Correct script:

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

Thanks And Fixed it.

Does this patch give you a functional curator? On my side, for a delete indices action against ES8, I get this:

AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'

[anjia0532]

@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator, and then run python setup.py
Correct script:

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

Thanks And Fixed it.

Does this patch give you a functional curator? On my side, for a delete indices action against ES8, I get this:

AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'

@lsoica Test it by my docker image ?

docker pull anjia0532/curator:5.8.5

[slowquery]

@anjia0532 thank you.
I've solved things with your help! 🙂

curator-dn64c                                       0/1     Completed   0             86s

[untergeek]

I can tell you there is a huge push now to get Curator 8 out the door in rapid fashion, and I hope that it is fully available for the general public before the end of January now.

I can also tell you that refactoring all of the APIs (and I mean all of them) is tedious and time consuming. I have to change not only the API calls that are made, but how the tests are structured, too. These changes will not be able to be backported to Curator 7 or Curator 6 (which will also be officially released at the same time as Curator 8), nor will any future work on Curator be backported to earlier releases. The nice thing is that the work I'm doing now will make it easier to release a Curator 9 when Elasticsearch 9 is released.

What happened? Why did it take so much work to do this? In a nutshell, the elasticsearch python client was majorly refactored to more closely match the API of Elasticsearch itself. On the simple side, elasticsearch-py function calls which sent a JSON body (well, a Python dict that was automatically converted to JSON for the call) were frequently using body=... as a parameter. Guess what isn't used any more? And where settings were being passed as part of body= elsewhere, now there's dedicated transient= and permanent= options, which means completely changing the structures and how they were passed.

As stated in previous communications, I am not a full-time developer anymore, so all of my work has been slower, and a labor of love. It was hard to find time, and it was hard to find motivation when these changes are so tedious to chase down and fix. Now, however, there is internal pressure to get this ready for the general public. I'm excited that I get a few weeks to use nearly all of my work time to get Curator 8 ready. It's still tedious, but I've made great progress on the lower-hanging fruit. I have a ton of work to do for Reindex and Freeze, to say nothing of Snapshot and Restore ahead of me.

There will also be a few new features with regards to ILM, possibly SLM, and definitely about moving indices/shards through different tiers. I will also need to make Curator work with Searchable Snapshots, which is entirely new.

I appreciate the community's work in modifying older versions of Curator to ignore the <8.0 limitation in the current Curator 5.8.5 release. This is a workaround that will not pass all tests, unfortunately, but at least allows you the opportunity to get what you need working until such time as I can get a fully Curator 8 release to work.

NOTE: Curator 8 will NOT release on RPM/DEB packages any more. It will be available via source code, pip, and Docker image. I will try to build a Docker image for arm64 in addition to x86_64, but that will depend on some external testing. I will put the scripts to build Docker images in this Github repo so anyone can build their own.

[untergeek]

Also note that much of what I have had to refactor is the removal of the AWS-ES supporting code that resulted in using requests, which resulted in the error:

AttributeError: module 'elasticsearch' has no attribute 'RequestsHttpConnection'

A tremendous amount of code clean-up has happened. I should warn you all that your existing ~/.curator/curator.yml files will need to be modified. As a teaser, and for preparation, I have created a side project to handle Elasticsearch client object creation. I liked the work that I did in Curator to make this work, and wanted it to be available as a separate module, so I created it at https://github.com/untergeek/es_client. This now technically supports every client connection option that the official Elasticsearch Python client offers, including API keys, bearer tokens, and also allows the use of cloud_ids to connect to Elastic Cloud and/or ECE.

There is a simple example of how to build a CLI client which accepts a config file and will add and override that file with CLI-specified options here. Not all options are addressed, but the most common ones are, making this example usable out of the box for many applications.

[untergeek]

More info at https://es-client.readthedocs.io/en/v8.6.0/defaults.html

[rshiva777]

@anjia0532 Thanks!
But it's not working. First, in your script, the folder is deleted rm -rf /tmp/curator, and then run python setup.py
Correct script:

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

Thanks And Fixed it.

Hello @anjia0532 ,

I have used your docker image "anjia0532/curator:5.8.5" in the cronjob, but getting the below error, could you please assist

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  93s   default-scheduler  Successfully assigned efk-system-logging/curator-elasticsearch-curator-27903900--1-mlrc5 to bit-rat-tst-2-md-1-5b996849bf-hsrsl
  Normal   Pulled     93s   kubelet            Container image "registry.test.com/test/curator:5.8.5" already present on machine
  Normal   Created    93s   kubelet            Created container elasticsearch-curator
  Warning  Failed     93s   kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/curator/curator": permission denied: unknown
root@test:~$

My cronjob yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    meta.helm.sh/release-name: curator
    meta.helm.sh/release-namespace: efk-system-logging
  labels:
    app: elasticsearch-curator
    app.kubernetes.io/managed-by: Helm
    chart: elasticsearch-curator-2.2.3
    heritage: Helm
    release: curator
  name: curator-elasticsearch-curator
  namespace: efk-system-logging
spec:
  jobTemplate:
    metadata:
      labels:
        app: elasticsearch-curator
        release: curator
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: elasticsearch-curator
            release: curator
        spec:
          containers:
          containers:
          - args:
            - --config
            - /etc/es-curator/config.yml
            - /etc/es-curator/action_file.yml
            command:
            - /curator/curator
            image: registry.test.com/test/curator:5.8.5
            imagePullPolicy: IfNotPresent
            name: elasticsearch-curator
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/es-curator
              name: config-volume
          dnsPolicy: ClusterFirst
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext:
            runAsUser: 16
          terminationGracePeriodSeconds: 30
          volumes:
          - configMap:
              defaultMode: 420
              name: curator-elasticsearch-curator-config
            name: config-volume
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 3
  suspend: false

[untergeek]

Progress Report

I have a ton of work to do for Reindex and Freeze, to say nothing of Snapshot and Restore ahead of me.

Freeze is no longer supported in 8, so I don't have to worry about that.

Last night I finished initial work on Reindex. I have also finished Snapshot and Restore, and all of the tests are working for all of the refactoring work I've completed. I've had to update the es_client module a few times, so that is now at 8.6.0.post3 as a result. The two messiest and hardest ones to refactor so far were Reindex and Shrink, but they are working and passing tests now. Yay!

The Index Settings action is still working with the older body type calls. There are probably a few more calls like that which are deprecated but working still. I want to clean it all up, but can do some of that kind of clean-up later, so long as test are passing and the code is working.

What remains, if I'm this close? I have to finish es_repo_mgr and refactoring all of the click command-line work for the singletons. Most of that will be the same, but I do need to add new options for the client configuration portion of those scripts.

I will likely do a pre-release at that point. I have some Searchable Snapshot functionality to add, but that won't matter to Basic License users of Elasticsearch. I will likely also need to get data tiers working ASAP as well, though this functionality is still covered by the Shard Allocation Routing that is already in Curator, from what I can tell.

[anjia0532]

@anjia0532谢谢！但它不起作用。首先，在脚本中，删除文件夹，然后运行正确的脚本：rm -rf /tmp/curator``python setup.py

FROM python:3.9.4-alpine3.13

# Chinese mirror
#RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
#     pip install -i https://mirrors.ustc.edu.cn/pypi/web/simple pip -U && \
#     pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/web/simple

RUN  apk --no-cache add --virtual .build-deps git && \
     git clone https://github.com/elastic/curator.git /tmp/curator && \
     cd /tmp/curator && \
     git checkout -b tags/v5.8.4 && \
     wget -O /tmp/curator.patch https://gist.githubusercontent.com/anjia0532/d8e779c97333d6fc0d045675e18e1224/raw/5de8de9fd634afbc1fdbaf077f2e9e0c97638b43/elastic-curator-support-es-8-x.patch && \
     git apply --check /tmp/curator.patch && \
     git apply /tmp/curator.patch

WORKDIR /tmp/curator
RUN python setup.py install

谢谢并修复了它。

你好@anjia0532，

我在 cronjob 中使用了您的 docker 映像“anjia0532/curator：5.8.5”，但收到以下错误，请您协助

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  93s   default-scheduler  Successfully assigned efk-system-logging/curator-elasticsearch-curator-27903900--1-mlrc5 to bit-rat-tst-2-md-1-5b996849bf-hsrsl
  Normal   Pulled     93s   kubelet            Container image "registry.test.com/test/curator:5.8.5" already present on machine
  Normal   Created    93s   kubelet            Created container elasticsearch-curator
  Warning  Failed     93s   kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/curator/curator": permission denied: unknown
root@test:~$

我的 cronjob yaml：

apiVersion: batch/v1
kind: CronJob
metadata:
  annotations:
    meta.helm.sh/release-name: curator
    meta.helm.sh/release-namespace: efk-system-logging
  labels:
    app: elasticsearch-curator
    app.kubernetes.io/managed-by: Helm
    chart: elasticsearch-curator-2.2.3
    heritage: Helm
    release: curator
  name: curator-elasticsearch-curator
  namespace: efk-system-logging
spec:
  jobTemplate:
    metadata:
      labels:
        app: elasticsearch-curator
        release: curator
    spec:
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: elasticsearch-curator
            release: curator
        spec:
          containers:
          containers:
          - args:
            - --config
            - /etc/es-curator/config.yml
            - /etc/es-curator/action_file.yml
            command:
            - /curator/curator
            image: registry.test.com/test/curator:5.8.5
            imagePullPolicy: IfNotPresent
            name: elasticsearch-curator
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/es-curator
              name: config-volume
          dnsPolicy: ClusterFirst
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext:
            runAsUser: 16
          terminationGracePeriodSeconds: 30
          volumes:
          - configMap:
              defaultMode: 420
              name: curator-elasticsearch-curator-config
            name: config-volume
  schedule: '*/5 * * * *'
  successfulJobsHistoryLimit: 3
  suspend: false

remove command , change args to 'curator --config /path/to/config_file.yml /path/to/action_file.yml'

[untergeek]

This is resolved with the release of Curator v8.

[sastorsl]

This might be relevant for the docker image of curator in docker hub: docker/hub-feedback#2314

[untergeek]

Thank you for the heads-up. As Curator is hosted on my personal, individual account, I do not believe this will affect me. I will eventually get around to having Curator's docker images hosted with Elastic, but for now, this seems safe.

[Nurlan199206]

i'm trying to delete indices on es 8.3.3.

container shows exit status 0 , but indices still exists

docker run --network host --rm --name es-curator -v /etc/elasticsearch/curator:/.curator untergeek/curator:8.0.8 --config /.curator/client.yml /.curator/action.yml

actions:
  1:
    action: delete_indices
    description: Delete indices older than 4 days (based on index name)
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: k8s-production-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 4

elasticsearch:
client:
  hosts:
    - 127.0.0.1
  port: 9200
  http_auth: elastic:mypassword
logging:
  loglevel: INFO
  logfile: /var/log/curator
  logformat: default

[untergeek]

Your client YAML file raises a few questions. First, it doesn't look properly indented. Second, you have added port which appears to be a hold-over from a much older version of Curator. The hosts definition should now include the full URL + port, e.g. http://127.0.0.1:9200. Third, you're adding http_auth when it should be username and password underneath other_settings. This appears to be another holdover from an older release of Curator.

These changes are all documented in the official documentation. These breaking changes are documented in the Changelog.

Your client definition should look like this, instead:

elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9200
  other_settings:
    username: elastic
    password: mypassword
  logging:
    loglevel: INFO
    logfile: /var/log/curator
    logformat: default

I should also note that you should not be logging to /var/log/curator while using Docker. Given that you've mounted a path (even though you're keeping it in /etc locally) you could set logfile: /.curator/curator.log and then your log file will appear in /etc/elasticsearch/curator/curator.log.