[elasticsearch] Keystore integration #154

Crazybus · 2019-06-06T15:52:55Z

Chart version not bumped (the versions are all bumped and released at the same time)
README.md updated with any new values or changes
Updated template tests in ${CHART}/tests/*.py
Updated integration tests in ${CHART}/examples/*/test/goss.yaml

jordansissel · 2019-08-01T17:16:21Z

elasticsearch/README.md

-     path: /usr/share/elasticsearch/config/elasticsearch.keystore
-     subPath: elasticsearch.keystore
-   ```
+Take a look a the [config example](/elasticsearch/examples/config/values.yaml) which has a tested version of adding strings and files to the keystore.


Might be helpful to have a note explicitly stating that any external values (secrets, etc) need to be explicitly set as env vars (which is what the example does). Thoughts?

jordansissel

Approved as-is. Small comments inline for optional changes.

jordansissel · 2019-08-01T17:26:02Z

elasticsearch/examples/config/values.yaml

+  strings:
+    bootstrap.password: '${ELASTIC_PASSWORD}'
+    xpack.notification.slack.account.monitoring.secure_url: '${SLACK_URL}'
+  files:


cloud-on-k8s doesn't have a specific concept of file vs string keystore entries: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html

I don't recall why that is, but I wonder if we can have the same concepts for this on both the chart and the operator?

I remember a change was made (initiated by the cloud-on-k8s team) to have the Elasticsearch keystore treat all strings and files the same internally.

My original assumption was that the operator would be doing something special to take these secrets, generate a keystore and create that as a secret. But their approach is also fully init container based. All secrets are mounted as files and then the init script loops over them and adds them to the keystore.

I like this a lot more than my solution, mainly because you don't need to think about mapping the files and strings into the container yourself. And the other huge security advantage is that these secrets won't still exist as environment variables or mounts in the actual running Elasticsearch container.

One downside to this approach is that it looks like the spec only allows a single secret. The downside of this being that you now need to fully recreate your keystore secret everytime you want to add something new.

Making it a list would solve this so I'll ping the team and see what their thoughts on it are.

spec: secureSettings: - secretName: your-secure-settings-secret - secretName: other-secure-settings-secret

I'm going to take a stab at implementing this in the same way.

Crazybus · 2019-08-02T08:46:18Z

@jordansissel I have updated it to match how cloud-on-k8s is doing it (which is a much better approach). Issues have also been opened to implement the suggestions I had around allowing multiple secrets, and the ability to set all parts of the volumeMount spec to allow specifying a different path than the key of the secret.

elastic/cloud-on-k8s#1457
elastic/cloud-on-k8s#1458

Please take another look :)

jordansissel

Found an edge case on the bootstrap.password. Other comments are non-problem/style.

I like the approach to allow multiple secret mounts to provide keystore values. 👍

jordansissel · 2019-08-20T18:43:23Z

elasticsearch/templates/statefulset.yaml

+        - name: keystore
+          emptyDir: {}
+{{ end }}
+        {{- range .Values.keystore }}


curious, why not have this {{- range ...}} inside the {{ if ... block above? I I don't think there'd be any behavior changes, just curious about the style choice.

This is just a leftover from the original refactoring. But you are right that it doesn't change anything. I fixed it up to make it as explicit as possible.

jordansissel · 2019-08-20T18:43:58Z

elasticsearch/templates/statefulset.yaml

@@ -111,6 +111,14 @@ spec:
          configMap:
            name: {{ template "uname" . }}-config
        {{- end }}
+{{ if .Values.keystore }}


Did you want to indent this and use {{- if ... ? Not sure if we're concerned with indentation style.

Fixed! It would be nice to add a "white space detection" check in the pull request testing and do a one off PR to get things back in line. Right now its a bit of a mess. It's a shame that helm lint --strict doesn't already cover this.

jordansissel · 2019-08-20T18:47:24Z

elasticsearch/templates/statefulset.yaml

+          done
+
+          # Add the bootstrap password since otherwise the Elasticsearch entrypoint tries to do this on startup
+          [ ! -z "$ELASTIC_PASSWORD" ] && echo $ELASTIC_PASSWORD | elasticsearch-keystore add -x bootstrap.password


Needs quotation marks around the second var, echo "$ELASTIC_PASSWORD" otherwise a password with whitespace won't be set correctly.

It might be too complicated to add to the tests, but maybe we could add a test to cover this case (whitespace in a password, such as ELASTIC_PASSWORD="hello world "

Great catch! Fixed! I thought I had copy pasted this from the ELasticsearch docker file, but looks like my copy paste skills failed since it is properly quoted in there: https://github.com/elastic/elasticsearch/blob/31f6e78acdb8dfc7c042be4ad0bc0bdea003e055/distribution/docker/src/docker/bin/docker-entrypoint.sh#L88

Crazybus · 2019-08-22T06:45:14Z

Thanks for the review @jordansissel, all comments have been addressed. Please take another look :)

Closes: #90 Adds a kubernetes native way to add strings and files to the Elasticsearch keystore. Previously you needed to manually create the keystore and upload it as a secret. There were a couple of issues with this approach. 1. The Elasticsearch keystore has an internal version for the format. If this is changed it meant needing to recreate each keystore again. 2. If you wanted to add a single new value it meant recreating the entire keystore again

jordansissel

🎈 Ship it!

Thanks for being patient with my having forgotten to re-review this periodically :)

Crazybus · 2019-08-23T21:31:29Z

❤️

And thank you for the great suggestions and finding bugs!

dcvtruong · 2019-09-12T00:01:41Z

@Crazybus What is the correct usage for "keystore: []" in values.yaml? I try several different format for '[]' format and it was not working.

keystore: 
  - xpack.notification.slack.account.monitoring.secure_url: 'https://webhook_url.....'

Crazybus · 2019-09-18T12:48:37Z

The simpliest example is:

helm-charts/elasticsearch/examples/config/values.yaml

Lines 25 to 26 in a499955

    
           keystore: 
        
             - secretName: elastic-config-secret

With the secret created like this:

helm-charts/elasticsearch/examples/config/Makefile

Line 12 in a499955

    
           kubectl create secret generic elastic-config-slack --from-literal=xpack.notification.slack.account.monitoring.secure_url='https://hooks.slack.com/services/asdasdasd/asdasdas/asdasd'

dcvtruong · 2019-09-18T16:16:29Z

@Crazybus Thanks.

harsh4870 · 2020-07-28T07:37:06Z

@Crazybus

i have tried adding GCS service account key in keystone but getting error

created kubernetes secret and added file into it and added secret into keystore

but pod are crashing also my i am using custom docker image with plugin installed but getting error like :

csearch", "node.name": "elasticsearch-master-0",  "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [gcs_backup_key.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.2.jar:7.3.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.2.jar:7.3.2]",
"Caused by: java.lang.IllegalArgumentException: unknown secure setting [gcs_backup_key.json] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.2.jar:7.3.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.2.jar:7.3.2]",
"... 6 more"] }

created secret using

kubectl create secret generic gcs-backup-key --from-file=gcs_backup_key.json=gcs_backup_key.json

here

keystore:
  - secretName: gcs-backup-key

Update

Image running and ready 1/1.

secretMounts:
  - name: gcs-backup-key
    secretName: gcs-backup-key
    path: /usr/share/elasticsearch/config/gcs_backup_key.json
    subPath: gcs_backup_key.json

please correct me if missing anything or wrong way

i want to set cronjob for setting regular backup let me know if missing anything thanks in advance for help.

rimusz mentioned this pull request Jun 7, 2019

Elasticsearch cluster backup/restore #142

Closed

Crazybus mentioned this pull request Jul 18, 2019

Cannot mount elasticsearch keystore in pod, device busy #90

Closed

Crazybus force-pushed the configurator_mapinator branch 2 times, most recently from fd453a1 to efdcb6c Compare July 22, 2019 08:19

Crazybus changed the title ~~WIP: Proof of concept: Keystore integration~~ [elasticsearch] Keystore integration Jul 22, 2019

Crazybus requested a review from jordansissel July 22, 2019 08:48

Crazybus force-pushed the configurator_mapinator branch from efdcb6c to 9305a54 Compare August 1, 2019 15:04

jordansissel reviewed Aug 1, 2019

View reviewed changes

jordansissel previously approved these changes Aug 1, 2019

View reviewed changes

Crazybus dismissed jordansissel’s stale review via 3e99a26 August 2, 2019 08:41

Crazybus force-pushed the configurator_mapinator branch from 9305a54 to 3e99a26 Compare August 2, 2019 08:41

Crazybus requested a review from jordansissel August 2, 2019 08:46

jordansissel suggested changes Aug 20, 2019

View reviewed changes

Crazybus requested a review from jordansissel August 22, 2019 06:44

Crazybus added 2 commits August 22, 2019 16:47

Properly quote bootstrap password

d025a9b

Crazybus force-pushed the configurator_mapinator branch from 9583f93 to d025a9b Compare August 22, 2019 14:51

jordansissel approved these changes Aug 23, 2019

View reviewed changes

Crazybus merged commit 1ad3826 into master Aug 23, 2019

Crazybus deleted the configurator_mapinator branch August 23, 2019 21:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[elasticsearch] Keystore integration #154

[elasticsearch] Keystore integration #154

Crazybus commented Jun 6, 2019 •

edited

jordansissel Aug 1, 2019

jordansissel left a comment

jordansissel Aug 1, 2019

Crazybus Aug 2, 2019

Crazybus commented Aug 2, 2019

jordansissel left a comment

jordansissel Aug 20, 2019

Crazybus Aug 22, 2019

jordansissel Aug 20, 2019

Crazybus Aug 22, 2019

jordansissel Aug 20, 2019

jordansissel Aug 20, 2019

Crazybus Aug 22, 2019

Crazybus commented Aug 22, 2019

jordansissel left a comment

Crazybus commented Aug 23, 2019

dcvtruong commented Sep 12, 2019 •

edited

Crazybus commented Sep 18, 2019

dcvtruong commented Sep 18, 2019

harsh4870 commented Jul 28, 2020 •

edited

[elasticsearch] Keystore integration #154

[elasticsearch] Keystore integration #154

Conversation

Crazybus commented Jun 6, 2019 • edited

Choose a reason for hiding this comment

jordansissel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crazybus commented Aug 2, 2019

jordansissel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crazybus commented Aug 22, 2019

jordansissel left a comment

Choose a reason for hiding this comment

Crazybus commented Aug 23, 2019

dcvtruong commented Sep 12, 2019 • edited

Crazybus commented Sep 18, 2019

dcvtruong commented Sep 18, 2019

harsh4870 commented Jul 28, 2020 • edited

Crazybus commented Jun 6, 2019 •

edited

dcvtruong commented Sep 12, 2019 •

edited

harsh4870 commented Jul 28, 2020 •

edited