Move config map to go #1268

RafalKorepta · 2024-05-05T21:51:06Z

`ec39960` Add ability to annotate functions with `ignore` statement

`99bb6e0` Fix typo from `interation` to `iteration`

`b016311` Improve rewrite error reporting

`6012dbb` Add `RegexReplaceAll` sprig function

`75f28bb` Generate values to helm template

Methods improves readability of Redpanda configuration.

IsAccessKeyReferenceValid and IsSecretKeyReferenceValid functions are not used.

`e0fad8c` Fix `for` loops using selector expression

Previous implementation would panic as type assertion was hardcoded for ast.Ident.

`2e38275` Change `tiered` helper function to `Storage` methods

`6df252f` Add Monitoring coreos.com custom resources to the scheme

When Prometheus Redpanda helm chart dependency is enabled, then DecodeYAMLFrom
is not working due to missing ServiceMonitor custom resource in scheme.

=== CONT  TestConfigMap/14-prometheus-no-tls-values.yaml
    chart_test.go:182:
        	Error Trace:	/Users/rafalkorepta/workspace/redpanda/helm-charts/charts/redpanda/chart_test.go:182
        	Error:      	Received unexpected error:
        	            	no kind "ServiceMonitor" is registered for version "monitoring.coreos.com/v1" in scheme "pkg/runtime/scheme.go:100"
        	Test:       	TestConfigMap/14-prometheus-no-tls-values.yaml
    --- FAIL: TestConfigMap/14-prometheus-no-tls-values.yaml (0.08s)

`28d2f13` Convert config maps to go base implementation

`97a80f2` Remove old config map implementation

Update Statefulset config map hash calculation

`9ef78e3` Handle overprovisioned differently from the desire implementation

Changes to the input values are anti-pattern and relay on order of execution.
Our test shows that overprovioned are incorrectly set when CPU is bellowed
1000m.

Reference
(overproviosned is set to false)[https://github.com/redpanda-data/helm-charts/blob/5e30ecbaf15da44691402f6128194d6639652845/charts/redpanda/testdata/ci/22-gke-tiered-storage-with-creds-values.yaml.tpl.golden#L481]

helm-charts/charts/redpanda/ci/22-gke-tiered-storage-with-creds-values.yaml.tpl

Line 33 in 5e30ecb

cores: 400m

helm-charts/charts/redpanda/templates/_helpers.tpl

Lines 167 to 169 in 5e30ecb

    
           Returns the SMP CPU count in whole cores, with minimum of 1, and sets 
        
           "resources.cpu.overprovisioned: true" when the "resources.cpu.cores" is less 
        
           than 1 core.

helm-charts/charts/redpanda/templates/_helpers.tpl

Line 174 in 5e30ecb

Follow up PRs

#1358
#1359

chrisseto

Huge PR and great work! Could we possibly split up some distinct pieces of this and merge it in one at a time? Extracting out all the gotohelm changes would be a great start and then we can make sure they get appropriate tests as well.

I'd like to talk about the proposed schema changes in a bit more depth as well. I think that would be a good candidate for being extracted into it's own commit/PR.

pkg/gotohelm/transpiler.go

pkg/gotohelm/helmette/sprig.go

charts/redpanda/chart_test.go

pkg/gotohelm/transpiler.go

charts/redpanda/configmap.tpl.go

charts/redpanda/values.go

charts/redpanda/chart_test.go

charts/redpanda/configmap.tpl.go

pkg/helm/helm.go

chrisseto · 2024-06-13T16:47:37Z

pkg/helm/helm.go

@@ -300,12 +304,18 @@ func (c *Client) Template(ctx context.Context, chart string, opts TemplateOption
 	// to set it ourselves.
 	client.ReleaseName = releaseName

+	previous := os.Getenv("HELM_CONFIG_HOME")
+	os.Setenv("HELM_CONFIG_HOME", path.Join(c.configHome, "helm-config"))


Do we need to set this? I don't see anything reading this value

I'm not sure why it isn't picked up by the install action. I agree it might affect other parallel runs. Using other helm commands this env var is passed into helm cli subproces

I was thinking that locate chart would work with

chart, err = client.ChartPathOptions.LocateChart(chart, &cli.EnvSettings{ RegistryConfig: envOr("HELM_REGISTRY_CONFIG", helmpath.ConfigPath("registry/config.json")), RepositoryConfig: envOr("HELM_REPOSITORY_CONFIG", helmpath.ConfigPath("repositories.yaml")), RepositoryCache: envOr("HELM_REPOSITORY_CACHE", helmpath.CachePath("repository")), Debug: true, })

apparently the c.configHome needs to be injected into tree of those environment variables.

Found the problem. Chart repository is relaying on env var. Here is the stack trace:

https://github.com/helm/helm/blob/15f76cf83c670a329b62c2b5ddeb0864ec99daec/pkg/repo/chartrepo.go#L78
https://github.com/helm/helm/blob/15f76cf83c670a329b62c2b5ddeb0864ec99daec/pkg/helmpath/home.go#L24
https://github.com/helm/helm/blob/15f76cf83c670a329b62c2b5ddeb0864ec99daec/pkg/helmpath/lazypath.go#L60

chrisseto

Tests are passing, let's merge and refactor from there!

Methods improves readability of Redpanda configuration.

Previous implementation would panic as type assertion was hardcoded for Ident.

When Prometheus Redpanda helm chart dependency is enabled, then DecodeYAMLFrom is not working due to missing ServiceMonitor custom resource in scheme. ``` === CONT TestConfigMap/14-prometheus-no-tls-values.yaml chart_test.go:182: Error Trace: /Users/rafalkorepta/workspace/redpanda/helm-charts/charts/redpanda/chart_test.go:182 Error: Received unexpected error: no kind "ServiceMonitor" is registered for version "monitoring.coreos.com/v1" in scheme "pkg/runtime/scheme.go:100" Test: TestConfigMap/14-prometheus-no-tls-values.yaml --- FAIL: TestConfigMap/14-prometheus-no-tls-values.yaml (0.08s) ```

Changes to the input values are anti-pattern and relay on order of execution. Our test shows that overprovioned are incorrectly set when CPU is bellowed `1000m`. Reference (overproviosned is set to false)[https://github.com/redpanda-data/helm-charts/blob/5e30ecbaf15da44691402f6128194d6639652845/charts/redpanda/testdata/ci/22-gke-tiered-storage-with-creds-values.yaml.tpl.golden#L481] https://github.com/redpanda-data/helm-charts/blob/5e30ecbaf15da44691402f6128194d6639652845/charts/redpanda/ci/22-gke-tiered-storage-with-creds-values.yaml.tpl#L33 https://github.com/redpanda-data/helm-charts/blob/5e30ecbaf15da44691402f6128194d6639652845/charts/redpanda/templates/_helpers.tpl#L167-L169 https://github.com/redpanda-data/helm-charts/blob/5e30ecbaf15da44691402f6128194d6639652845/charts/redpanda/templates/_helpers.tpl#L174

chrisseto · 2024-06-17T14:17:25Z

charts/redpanda/configmap.tpl.go

+
+	address := fmt.Sprintf("%s-%d", Fullname(dot), int(i))
+	if ptr.Deref(values.External.Domain, "") != "" {
+		// TODO: TPL DOES NOT WORK. Maybe tpl call could be removed


nit: tpl is now implemented

chrisseto · 2024-06-17T14:19:29Z

charts/redpanda/chart_test.go

+	client, err := helm.New(helm.Options{ConfigHome: testutil.TempDir(t)})
+	require.NoError(t, err)
+
+	// Downloading Redpanda helm chart release is required as client.Template


This feels fixable but we can worry about it later 👍

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch 2 times, most recently from 45f71be to e795535 Compare May 22, 2024 09:10

RafalKorepta marked this pull request as ready for review May 22, 2024 09:11

RafalKorepta requested a review from chrisseto as a code owner May 22, 2024 09:11

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch 2 times, most recently from 295c011 to 5704515 Compare May 22, 2024 09:29

RafalKorepta changed the title ~~[WIP] Move config map to go~~ Move config map to go May 22, 2024

chrisseto reviewed May 22, 2024

View reviewed changes

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch 4 times, most recently from b2f27bc to b6bfb03 Compare May 23, 2024 10:52

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch 7 times, most recently from 8767ad7 to 7cf3055 Compare June 13, 2024 16:14

chrisseto reviewed Jun 13, 2024

View reviewed changes

RafalKorepta enabled auto-merge (rebase) June 13, 2024 17:58

RafalKorepta disabled auto-merge June 13, 2024 18:01

chrisseto approved these changes Jun 14, 2024

View reviewed changes

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch 5 times, most recently from 54894a0 to 079fd76 Compare June 16, 2024 00:04

RafalKorepta mentioned this pull request Jun 17, 2024

Handle overprovisioned flag based on CPU values #1358

Merged

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch from 61579ed to 933db92 Compare June 17, 2024 09:13

RafalKorepta added 11 commits June 17, 2024 13:09

Add ability to annotate functions with ignore statement

ec39960

Fix typo from interation to iteration

99bb6e0

Improve rewrite error reporting

b016311

Add RegexReplaceAll sprig function

6012dbb

Generate values to helm template

75f28bb

Methods improves readability of Redpanda configuration.

Fix for loops using selector expression

e0fad8c

Previous implementation would panic as type assertion was hardcoded for Ident.

Change tiered helper function to Storage methods

2e38275

Convert config maps to go base implementation

28d2f13

Remove old config map implementation

97a80f2

RafalKorepta force-pushed the rk/gh-1202/move-configmap-to-go branch from 933db92 to 9ef78e3 Compare June 17, 2024 11:35

This was referenced Jun 17, 2024

redpanda: configmap refactors #1357

Closed

Handle secret reference check as method #1359

Merged

chrisseto approved these changes Jun 17, 2024

View reviewed changes

RafalKorepta merged commit 0082341 into main Jun 17, 2024
41 checks passed

RafalKorepta deleted the rk/gh-1202/move-configmap-to-go branch June 17, 2024 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move config map to go #1268

Move config map to go #1268

RafalKorepta commented May 5, 2024 •

edited

Loading

chrisseto left a comment

chrisseto Jun 13, 2024

RafalKorepta Jun 13, 2024 •

edited

Loading

RafalKorepta Jun 14, 2024

RafalKorepta Jun 14, 2024

chrisseto left a comment

chrisseto Jun 17, 2024

chrisseto Jun 17, 2024

	Returns the SMP CPU count in whole cores, with minimum of 1, and sets
	"resources.cpu.overprovisioned: true" when the "resources.cpu.cores" is less
	than 1 core.

Move config map to go #1268

Move config map to go #1268

Conversation

RafalKorepta commented May 5, 2024 • edited Loading

ec39960 Add ability to annotate functions with ignore statement

99bb6e0 Fix typo from interation to iteration

b016311 Improve rewrite error reporting

6012dbb Add RegexReplaceAll sprig function

75f28bb Generate values to helm template

e0fad8c Fix for loops using selector expression

2e38275 Change tiered helper function to Storage methods

6df252f Add Monitoring coreos.com custom resources to the scheme

28d2f13 Convert config maps to go base implementation

97a80f2 Remove old config map implementation

9ef78e3 Handle overprovisioned differently from the desire implementation

Follow up PRs

chrisseto left a comment

Choose a reason for hiding this comment

chrisseto Jun 13, 2024

Choose a reason for hiding this comment

RafalKorepta Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

RafalKorepta Jun 14, 2024

Choose a reason for hiding this comment

RafalKorepta Jun 14, 2024

Choose a reason for hiding this comment

chrisseto left a comment

Choose a reason for hiding this comment

chrisseto Jun 17, 2024

Choose a reason for hiding this comment

chrisseto Jun 17, 2024

Choose a reason for hiding this comment

RafalKorepta commented May 5, 2024 •

edited

Loading

`ec39960` Add ability to annotate functions with `ignore` statement

`99bb6e0` Fix typo from `interation` to `iteration`

`b016311` Improve rewrite error reporting

`6012dbb` Add `RegexReplaceAll` sprig function

`75f28bb` Generate values to helm template

`e0fad8c` Fix `for` loops using selector expression

`2e38275` Change `tiered` helper function to `Storage` methods

`6df252f` Add Monitoring coreos.com custom resources to the scheme

`28d2f13` Convert config maps to go base implementation

`97a80f2` Remove old config map implementation

`9ef78e3` Handle overprovisioned differently from the desire implementation

RafalKorepta Jun 13, 2024 •

edited

Loading