Globally injected variables do not work in subcharts/dependencies

[dvvanessastoiber]

Describe the bug
In the MRO pipeline, Jenkins sets the parameter imageTag during the helm install. If the deployment.yaml is located in a dependent chart (charts/ folder) and contains image: "{{ .Values.image.repository }}:{{ .Values.imageTag }}", helm cannot overwrite imageTag. This can only be achieved by accessing and setting the global parameter image: "{{ .Values.image.repository }}:{{ .Values.global.imageTag }}"; --set global.imageTag=x.x.x

To Reproduce
Steps to reproduce the behavior:

1. Clone Helm Chart Repo with nested chart structure
2. cd chart/
3. Access .Values.imageTag in deployment.yaml
4. Run helm template . -f values.yaml -f values.dev.yaml -f secrets.dev.yaml --set registry=image-registry.openshift-image-registry.svc:5000 --set componentId=<componentId> --set imageNamespace=<project>-dev --set imageTag=abc
5. Check imageTag used for deployment
6. Use .Values.global.imageTag in deployment.yaml
7. Run helm template . -f values.yaml -f values.dev.yaml -f secrets.dev.yaml --set registry=image-registry.openshift-image-registry.svc:5000 --set componentId=<componentId> --set imageNamespace=<project>-dev --set global.imageTag=abc
8. Check imageTag used for deployment

Expected behavior
imageTag should be set by MRO pipeline

Screenshots

Affected version:

• OpenDevStack v4.x

[serverhorror]

We likely need to go around here:

• ods-jenkins-shared-library/src/org/ods/component/HelmDeploymentStrategy.groovy

  Lines 115 to 152 in 88c3204

  	private void helmUpgrade(String targetProject) {
  	steps.dir(options.chartDir) {
  	jenkins.maybeWithPrivateKeyCredentials(options.helmPrivateKeyCredentialsId) { String pkeyFile ->
  	if (pkeyFile) {
  	steps.sh(script: "gpg --import ${pkeyFile}", label: 'Import private key into keyring')
  	}
  	// we add two things persistent - as these NEVER change (and are env independent)
  	options.helmValues['registry'] = context.clusterRegistryAddress
  	options.helmValues['componentId'] = context.componentId
  	// we persist the original ones set from outside - here we just add ours
  	Map mergedHelmValues = [:]
  	mergedHelmValues << options.helmValues
  	mergedHelmValues['imageNamespace'] = targetProject
  	mergedHelmValues['imageTag'] = options.imageTag
  	// deal with dynamic value files - which are env dependent
  	def mergedHelmValuesFiles = []
  	mergedHelmValuesFiles.addAll(options.helmValuesFiles)
  	options.helmEnvBasedValuesFiles.each { envValueFile ->
  	mergedHelmValuesFiles << envValueFile.replace('.env',
  	".${context.environment}")
  	}
  	openShift.helmUpgrade(
  	targetProject,
  	options.helmReleaseName,
  	mergedHelmValuesFiles,
  	mergedHelmValues,
  	options.helmDefaultFlags,
  	options.helmAdditionalFlags,
  	options.helmDiff
  	)
  	}
  	}
  	}

• ods-jenkins-shared-library/src/org/ods/orchestration/phases/DeployOdsComponent.groovy

  Lines 158 to 219 in 88c3204

  	private void applyTemplates(String startDir, Map deploymentMean = [:]) {
  	def jenkins = ServiceRegistry.instance.get(JenkinsService)
  	steps.dir(startDir) {
  	logger.info(
  	"Applying desired OpenShift state defined in " +
  	"${startDir}@${project.baseTag} to ${project.targetProject}, " +
  	"deploymentMean? ${deploymentMean}"
  	)
  	def secretName = startDir.startsWith('openshift') ?
  	project.tailorPrivateKeyCredentialsId :
  	project.helmPrivateKeyCredentialsId
  	def applyFunc = { String pkeyFile ->
  	// @ FIXME - which params should we take from the deploymentMean?
  	if (startDir.startsWith('openshift')) {
  	os.tailorApply(
  	project.targetProject,
  	deploymentMean.tailorSelectors as Map<String, String>,
  	project.environmentParamsFile,
  	// ensure that we don't pass duplicate parameters
  	(deploymentMean.tailorParams as Set<String>) as List<String>,
  	deploymentMean.tailorPreserve as List<String>,
  	pkeyFile,
  	true // verify
  	)
  	} else {
  	def helmValuesFiles =
  	(deploymentMean.helmValuesFiles) ?: ["values.yaml"]
  	// system values
  	Map<String, String> helmMergedValues = [
  	"imageTag": project.targetTag,
  	"imageNamespace": project.targetProject,
  	"componentId": deploymentMean.repoId
  	]
  	// take the persisted ones.
  	helmMergedValues << deploymentMean.helmValues
  	// deal with dynamic value files - which are env dependent
  	deploymentMean.helmEnvBasedValuesFiles.each { envValueFile ->
  	helmValuesFiles << envValueFile.replace('.env',
  	".${project.targetProject.split('-').last()}")
  	}
  	if (pkeyFile) {
  	steps.sh(script: "gpg --import ${pkeyFile}", label: 'Import private key into keyring')
  	}
  	os.helmUpgrade(
  	project.targetProject,
  	deploymentMean.helmReleaseName,
  	helmValuesFiles,
  	helmMergedValues,
  	deploymentMean.helmDefaultFlags,
  	deploymentMean.helmAdditionalFlags,
  	true)
  	}
  	}
  	jenkins.maybeWithPrivateKeyCredentials(secretName) { String pkeyFile ->
  	applyFunc(pkeyFile)
  	}
  	}
  	}

And

• we need to think about something like Support kustomize in Componentpipeline and Orchestrationpipeline #993 to allow "vendor provided charts", so we can use them "as-is", and

• as charts that are a bit more complex need to be able to deal with multiple references to the images built in the pipeline

Other useful references:

• https://helm.sh/docs/chart_template_guide/subcharts_and_globals/
  • specifically Overriding Values from a Parent Chart and
  • Global Chart Values

[michaelsauter]

FYI we actually set global.imageTag already for a while for exactly the same use case :) would be nice to have it out of the box!

[serverhorror]

Hi,

I believe a workaround for this could be, something like this in the Jenkinsfile:

odsComponentPipeline(
  imageStreamTag: 'ods/jenkins-agent-golang:4.x',
  branchToEnvironmentMapping: [
    'master': 'dev',
    // 'release/': 'test'
  ],
  resourceRequestMemory: '2Gi',
  resourceLimitMemory: '16Gi'
) { context ->

  // [...] typical pipeline code [...]
  odsComponentStageRolloutOpenShiftDeployment(context, [
    'selector': "app.kubernetes.io/instance=${context.componentId}",
    'helmValues': [
        'global.imageTag': context.shortGitCommit,
    ]
  ])
}

Given the following source:

(this lives in a subchart under templates/configMap.yaml as mentioned in the output)

kind: ConfigMap
metadata:
  name: my-name-{{ include "chart.fullname" . }}
  labels:
    {{- include "chart.labels" . | nindent 4 }}
data:
  example-configMap: |
    global.imageTag={{- .Values.global.imageTag }}

this leads to a resource like that:

---
# Source: chart/charts/mysubchart/templates/configMap.yaml
kind: ConfigMap
metadata:
  name: my-name-globally-injected-variables-do-not-work-in-subcharts-mysubchart
  labels:
    helm.sh/chart: mysubchart-0.1.0
    app.kubernetes.io/name: mysubchart
    app.kubernetes.io/instance: globally-injected-variables-do-not-work-in-subcharts
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
data:
  example-configMap: |
    global.imageTag=e44521d0

[serverhorror]

@dvvanessastoiber did you get a chance to give this a try?

[dvvanessastoiber]

@dvvanessastoiber did you get a chance to give this a try?

@serverhorror Thank you for providing this workaround and apologies for the delayed response! I have tested it and have a few open questions:

Prerequisites: We edited our base helm chart that now uses the global imageTag if it is set.

To my understanding, the DEV environment does not have any generated tags in the ImageStream and the actual version of the application image can be used. In contrast to that, QA and PROD only have ODS generated tags in the ImageStream (see screenshot below).

1. With the provided workaround in the Jenkinsfile, it is not possible to distinguish between stages, therefore the global value is set for dev as well, even if there is no need for a generated tag in this stage (please correct me if I am wrong)
2. The context.shortGitCommit is entirely different from the ODS generated tag (see screenshot above).

Please let me know what you think regarding those points and kindly let me know if we should have a call to discuss this in detail! Thank you!

[dvvanessastoiber]

@dvvanessastoiber did you get a chance to give this a try?

@serverhorror Thank you for providing this workaround and apologies for the delayed response! I have tested it and have a few open questions:

Prerequisites: We edited our base helm chart that now uses the global imageTag if it is set.

To my understanding, the DEV environment does not have any generated tags in the ImageStream and the actual version of the application image can be used. In contrast to that, QA and PROD only have ODS generated tags in the ImageStream (see screenshot below).

1. With the provided workaround in the Jenkinsfile, it is not possible to distinguish between stages, therefore the global value is set for dev as well, even if there is no need for a generated tag in this stage (please correct me if I am wrong)
2. The context.shortGitCommit is entirely different from the ODS generated tag (see screenshot above).

Please let me know what you think regarding those points and kindly let me know if we should have a call to discuss this in detail! Thank you!

@serverhorror Did you have time to look into this yet? Thank you for your support!