Add existing secrets to helm chart #397

lhaussknecht · 2023-11-19T10:20:13Z

This PR adds the possibility to use an existing kubernetes secret when installing sql-exporter with the helm chart.

Related to #396

burningalchemist · 2023-11-19T12:57:08Z

README.md

+helm repo add burningalchemist https://burningalchemist.github.io
+helm intall burningalchemist/sql-exporter


I've created a similar documentation at artifacthub.io - https://artifacthub.io/packages/helm/sql-exporter/sql-exporter#installing-the-chart. Let's put similar values everywhere just for consistency. There's a small typo, besides. 👍

Just a small reminder to address this little issue here. 👍

burningalchemist · 2023-11-19T13:00:04Z

helm/values.yaml

+# ---------------------------------------------------------------------
+# -- Option to provide an existing secret with all the values from the config section
+# ---------------------------------------------------------------------
+existingSecret: {}


I'd put existingSecret under the config section for better context, but I have no strong opinion here.

@allanger Do you mind checking the PR as well whenever you have some time? 🙂

Also, I've added helm-docs to generate README.md for chart values, and it's sensitive to the comments format (using them for description). So if you could remove delimiters (line 67 and 69), that'd be cool.

Actually, if you could run helm-docs --sort-values-order file command (from here) to regenerate README.md, that'd be amazing, so we could have a smooth release.

I'll add the notes to the contribution guide later. 🙂👍

@burningalchemist I'll check tomorrow

I also think that config related values should be under the config key. But the problem then is that currently we add the whole config object to the secret, I'd suggest to do something like this

config: existingSecret: SECRETNAME inline: global: ... ...

And then, if Values.config.existingSecret is not empty, we mount the existing one, otherwise, we put Values.config.inline to the templated secret

And it also would mean the major version bump, because it's a breaking change

I'm not concerned about breaking changes since we're under 0.x (zerover) so far, so breaking changes are intended, expected, but also should be announced. So we'd bump up the minor as usual.

I don't really like inline (although I understand the intention), let me think about it. 😉

Also we can do it like

config: from: kind: name: key: value: | ...

like here #399 (comment)

Oh, this one looks good. 👍

I'm not sure that value: | is a good idea, I guess it's better to handle value as an object in this case, and cast it to yaml in the template already, then users will be able to replace parts of it.

I'd say that it should be

config: from: kind: name: key: value: {} # global: ... ...

And then in the template it still can be {{ toYaml .Values.config.value }}

Fair enough, I was mostly looking at the structure. But yeah, agreed. 👍

burningalchemist · 2023-11-19T13:08:19Z

Hey @lhaussknecht, thank you for the contribution! 👍 I've left several comments above, but in general it's all good. We just need to agree on the existingSecret location in the values.yaml.

burningalchemist · 2023-11-21T09:34:53Z

Hey @lhaussknecht, could you please rebase the PR on the latest master whenever you're available? 🙂

lhaussknecht · 2023-11-21T09:53:33Z

Hey @burningalchemist . Will work on this today.

allanger · 2023-11-22T10:16:18Z

helm/templates/deployment.yaml

+            items:
+              - key: {{ .Values.config.from.key }}
+                path: "sql_exporter.yml"
+          {{- else }}


Hi, To me it looks like Values.config.value must have the highest priority, because it's something coming with a package, and external source should be used when the value is not provided.

if config.value then secret from the chart else if kind == secret use external secret else if kind == configmap use external configmap

And then the same applies to the secret.configuration.yaml, it's created if value is provided.

Or don't you think so?

Yes, good idea to look at config.value instead of config.from. But how do we override values when using an existing secret?

Hm, I guess you can set it like this:

config: from: kind: ... values: {}

But it doesn't look nice and user-friendly to me, so yeah, maybe it makes more sense to check for the from.kind in that case.
Maybe @burningalchemist has an idea about that

That's a shortcoming of Helm. You cannot override values with NULL.

Ok, you can do it with null I guess, not {}, but I still don't like it

I've checked like this:

# template apiVersion: v1 kind: Service metadata: {{- if .Values.check }} name: check {{- else }} name: not-check {{- end }} labels: {{- include "test.labels" . | nindent 4 }} spec: type: ClusterIP ports: - port: http targetPort: http protocol: TCP name: http selector: {{- include "test.selectorLabels" . | nindent 4 }}

default values:

check: check: check:

$ helm template . --- # Source: test/templates/service.yaml apiVersion: v1 kind: Service metadata: name: check labels: helm.sh/chart: test-0.1.0 app.kubernetes.io/name: test app.kubernetes.io/instance: release-name app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm spec: type: ClusterIP ports: - port: http targetPort: http protocol: TCP name: http selector: app.kubernetes.io/name: test app.kubernetes.io/instance: release-name

another value file: values-new.yaml

$ cat values-new.yaml check: null $ helm template . -f values-new.yaml --- # Source: test/templates/service.yaml apiVersion: v1 kind: Service metadata: name: not-check labels: helm.sh/chart: test-0.1.0 app.kubernetes.io/name: test app.kubernetes.io/instance: release-name app.kubernetes.io/version: "1.16.0" app.kubernetes.io/managed-by: Helm spec: type: ClusterIP ports: - port: http targetPort: http protocol: TCP name: http selector: app.kubernetes.io/name: test app.kubernetes.io/instance: release-name

So it either should be a _helper function, or we just leave it like that. And though I've suggested a change, after thinking about it, I guess, that it's better if your solution stays.

burningalchemist · 2023-11-22T17:58:01Z

Hey folks, I'll have a look tomorrow as well. 🙂👍

burningalchemist · 2023-12-04T12:30:52Z

Hey gents, apologies, I was a bit sick last week.

burningalchemist · 2023-12-20T11:46:40Z

Hey folks! I'd say, let's keep it as it is now (it can be a single comment in the end to clarify the precedence). Or we can get rid of config.value items completely for the time being.

The reason is there's a plan to implement a feature to have environment variables for config values (ideally, similar to how Grafana does it). This way, we address the precedence naturally: 'environment variables > config file'. It also might be easier to define parameters in the Helm template.

I'm ok with either. 🙂👍

allanger · 2024-01-08T14:58:54Z

Should I take care of it?

burningalchemist · 2024-01-08T15:28:58Z

Hey @allanger, if you have some time, could you please? 👍

Just pinging @lhaussknecht for visibility. 😃

lhaussknecht · 2024-01-08T16:14:18Z

Just arrived from holidays... @allanger are you already on it? I could work on it tomorrow.

allanger · 2024-01-08T16:27:45Z

Just arrived from holidays... @allanger are you already on it? I could work on it tomorrow.

Nope, not yet :)

…tion such as connection strings. Add hint in README.md to install the chart.

burningalchemist · 2024-01-15T11:12:19Z

hey @lhaussknecht, looks like the template is breaking the test with the following error:

ts=2024-01-10T12:55:48.711Z caller=klog.go:147 level=error func=Fatalf
msg="Error creating exporter: exactly one of `jobs` and `target` must be defined"

I guess we might need to update the test config as well.

burningalchemist · 2024-01-23T10:52:13Z

Hey @lhaussknecht, would you find some time in the upcoming weeks to figure out the remaining issues? 🙂 There are small things left, so we're close to merge the PR. 😉

burningalchemist · 2024-01-29T14:17:54Z

Hey @allanger, do you mind actually taking care of this PR? I guess we can check the PR out and recreate it under another request, so all the pushed commits are preserved.

If you don't have any capacity, I can take care myself, so just asking. 👍

allanger · 2024-01-29T14:24:00Z

@burningalchemist I'll be afk for about a week, and then I can take care of it.

burningalchemist · 2024-01-29T14:33:00Z

@allanger sounds good to me, let's see if there's any progress while you're away. 👍

allanger · 2024-02-07T14:08:27Z

So, I'm back, will try to fix it by the end of the week

allanger · 2024-02-09T15:19:11Z

@burningalchemist I can't understand one thing. Is there really a difference between this PR and this one #432?

To me, it looks like in both cases, we're just mounting secrets/configmaps to the sql_exported pod. But since I'm not using the tool atm, I'm not sure if intentions are the same.

I would basically just close them both by letting secret/configmap mount

burningalchemist · 2024-02-11T12:32:53Z

Hey @allanger, I guess the last sentence seems cut. 🤔 But I assume #432 wants to have a generic interface to mount arbitrary files to the container, whereas this PR wants a more specific parameter which just takes the secret name to be mounted. On the functional side, I think they perform the same operation, the goal for each might be different.

I'm curious to hear your idea. And thanks for looking into it. 👍

allanger · 2024-02-13T09:09:29Z

I think I would just let mount everything like that:

# values
mounts:
  - kind: Secret
    name: sql-exporter-config
    path: /etc/sql_exporter/
  - kind: ConfigMap
    name: big-json-config
    path: /etc/big-config

Then the whole configuration can be done with external secrets, and the templates/secret.configuration.yaml can become optional. Only if config is set explicitly in values, it should be created.

UPD: But I also need to think about environment variables, I guess it might make sense to mount whole secrets/configmaps as well, when they are created for sql_exporter env only

burningalchemist · 2024-02-15T16:12:48Z

@allanger Really I'd just make it as simple as possible, we can always tighten the interface later if it's too much confusion, but what you've described makes sense.

Users want to mount files next to the binary, be it the configuration file or any complimentary file.

At this stage, I think it's easier to explain that the configuration file needs to be prepared in advance (as a secret) and mounted respectively (like in your example snippet). It also implies that the config file's lifecycle is separate from the helm chart, which sounds pretty pragmatic. 👍

burningalchemist · 2024-02-24T19:38:58Z

Closed in favour of #446.

burningalchemist reviewed Nov 19, 2023

View reviewed changes

lhaussknecht force-pushed the add-existing-secrets-to-helm-chart branch from 0bf0ec1 to 009f8ec Compare November 22, 2023 09:42

allanger reviewed Nov 22, 2023

View reviewed changes

allanger mentioned this pull request Dec 11, 2023

[helm] Support getting data-source-name from an existing secret #396

Closed

lhaussknecht added 3 commits January 10, 2024 12:47

feat: Add possibility to use an existing secret for sensitive informa…

83e4d2c

…tion such as connection strings. Add hint in README.md to install the chart.

Reference existing ConfigMap or Secret for config values.

33b1c21

Add comment for config.from, bump version.

5e89203

lhaussknecht force-pushed the add-existing-secrets-to-helm-chart branch from 96e9454 to 5e89203 Compare January 10, 2024 11:54

run helm-template again

fb7dad6

allanger mentioned this pull request Jan 17, 2024

feat(helm): Add extraVolumes and extraVolumeMounts #432

Closed

allanger mentioned this pull request Feb 20, 2024

feat(helm): add an ability to mount extra volumes #446

Merged

burningalchemist closed this Feb 24, 2024

		helm repo add burningalchemist https://burningalchemist.github.io
		helm intall burningalchemist/sql-exporter

Add existing secrets to helm chart #397

Add existing secrets to helm chart #397

Conversation

lhaussknecht commented Nov 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burningalchemist Nov 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

allanger Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burningalchemist commented Nov 19, 2023 • edited Loading

burningalchemist commented Nov 21, 2023

lhaussknecht commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

burningalchemist commented Nov 22, 2023

burningalchemist commented Dec 4, 2023

burningalchemist commented Dec 20, 2023

allanger commented Jan 8, 2024

burningalchemist commented Jan 8, 2024

lhaussknecht commented Jan 8, 2024

allanger commented Jan 8, 2024

burningalchemist commented Jan 15, 2024

burningalchemist commented Jan 23, 2024

burningalchemist commented Jan 29, 2024

allanger commented Jan 29, 2024

burningalchemist commented Jan 29, 2024

allanger commented Feb 7, 2024

allanger commented Feb 9, 2024

burningalchemist commented Feb 11, 2024 • edited Loading

allanger commented Feb 13, 2024 • edited Loading

burningalchemist commented Feb 15, 2024 • edited Loading

burningalchemist commented Feb 24, 2024

burningalchemist Nov 19, 2023 •

edited

Loading

allanger Nov 20, 2023 •

edited

Loading

burningalchemist commented Nov 19, 2023 •

edited

Loading

burningalchemist commented Feb 11, 2024 •

edited

Loading

allanger commented Feb 13, 2024 •

edited

Loading

burningalchemist commented Feb 15, 2024 •

edited

Loading