New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Charts with different set of values for each environment #2620

Closed
polasekr opened this Issue Jun 26, 2017 · 26 comments

Comments

Projects
None yet
@polasekr
Copy link

polasekr commented Jun 26, 2017

We have multiple environments (DEV, PROD) where we run same set of containers with slightly different setting. I didn’t want to duplicate charts for each environment and for that reason I created parametrized charts using subcharts. This is my directory structure:

storm - Chart.yaml
      - values.yaml
      - requirements.yaml
      - charts - storm-dev - Chart.yaml
                           - values.yaml
               - storm-prd - Chart.yaml
                           - values.yaml
      - templates - deployment.yaml
                  - _helpers.tpl
                  - job.yaml
                  - secret.yaml

The chart dependencies are defined as follows:

$ cat requirements.yaml 
dependencies:
  - name: storm-dev
    repository: "file://charts/storm-dev"
    version: ">= 0.0.1"
    tags:
      - dev-values
    import-values:
      - data

  - name: storm-prd
    repository: "file://charts/storm-prd"
    version: ">= 0.0.1"
    tags:
      - prd-values
    import-values:
      - data

Parent chart's values.yaml file contains only following values:

$ cat values.yaml 
# Default values
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

tags:
  dev-values: false
  prd-values: false

All the actual values for templates are coming from subchart values.yaml files. I control the subchart values assignment by defining either dev-values or prd-values tag as true on the command line as true

helm install -n v1 . --set tags.dev-values=true

This approach kind of works on helm version 2.4.2. The only issue is that it throws warnings which I am trying to eliminate.

2017/06/25 19:49:04 Warning: ImportValues missing table: no table named "storm-prd" (map[tags:map[prd-values:false dev-values:true] storm-dev:map[global:map[] exports:map[data:map[ … ]]]])

I tried adding a condition in requirements.yaml file, but it didn’t fix those warnings.

When I tried to use version 2.5.0, but it didn’t work at all. Subchart's values are not propagated to a main chart.

$ /usr/bin/helm.2.5.0 install -n legacy1 . --set tags.dev-values=true --debug --dry-run
[debug] Created tunnel using local port: '49381'

[debug] SERVER: "localhost:49381"

[debug] Original chart version: ""
[debug] CHART PATH: /home/robert/src/hubub-kubernetes/helm/storm

2017/06/25 19:56:30 Warning: ImportValues missing table: no table named "storm-dev" (map[tags:map[dev-values:false prd-values:false]])
2017/06/25 19:56:30 Warning: ImportValues missing table: no table named "storm-prd" (map[tags:map[dev-values:false prd-values:false]])
NAME:   legacy1
REVISION: 1
RELEASED: Sun Jun 25 19:56:32 2017
CHART: storm-0.1.0
USER-SUPPLIED VALUES:
tags:
  dev-values: true

COMPUTED VALUES:
tags:
  dev-values: true
  prd-values: false

HOOKS:
MANIFEST:

Regarding my problem I want to ask few questions:

  1. Is there a better approach to have parametrized charts than what I am doing?
  2. Am I hitting a bug in version 2.5.0 or I am using non supported method?
  3. Is there a way to get rid of those warning messages? In my case I have main chart which consists of those application charts where each application has subcharts with values for each environment and this setup generates about a page of warning messages which I am trying to get rid of.
@thomastaylor312

This comment has been minimized.

Copy link
Collaborator

thomastaylor312 commented Jun 27, 2017

@jascott1 any feedback on this?

@dominik-bln

This comment has been minimized.

Copy link

dominik-bln commented Jun 27, 2017

I'm not that experienced with Helm myself yet, but if I see this correctly, what you want to achieve could be done simpler with helm install -f your-prod-values.yml. Just put some good defaults in the Charts values.yml and move your environement specific values somewhere outside the Chart and change them to the values you want during your command line calls:

https://github.com/kubernetes/helm/blob/ecef026b68a23f0c1399d98425134ce837126831/docs/using_helm.md#customizing-the-chart-before-installing
https://github.com/kubernetes/helm/blob/master/docs/helm/helm_install.md

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Jun 27, 2017

@thomastaylor312 i will check this out but if behaving as described by @polasekr then this is a bug ( a regression in 2.5 preventing ImportValues from working). It also sounds like ImportValues doesn't respect tag/conditions and tries to import chart values for disabled charts (which is a problem).

@thomastaylor312

This comment has been minimized.

Copy link
Collaborator

thomastaylor312 commented Jun 28, 2017

@jascott1 Ok sounds good. Let me know what you find and I'll label appropriately

@bacongobbler

This comment has been minimized.

Copy link
Member

bacongobbler commented Jun 28, 2017

Not a native Helm tool, but it's worth noting that this is something Draft solves within its own CLI. Each "environment" can set its own set of chart values in draft.toml

Breakdown of that feature here: https://github.com/Azure/draft/blob/master/docs/user-guide.md

I would argue that this could be solved nicely through a Helm plugin.

@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Jun 28, 2017

@dominik-bln
Proposed approach would work for help chart that doesn't have subcharts that supposed to be environment dependent. I have a main chart with few subcharts and I need to pass values to all subcharts based on the environment I run the chart in. The option of including all the variable files on the command line is not well scalable.

@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Jun 28, 2017

@bacongobbler I like that build in feature of draft allowing to natively selecting environment and building and deploying chart accordingly. It would be +1 from me for either such feature or helm plugin.

@janwillies

This comment has been minimized.

Copy link

janwillies commented Jul 12, 2017

We have exactly the same problem as @polasekr and are looking for a solution as well. Wrapping environments into if-else or template helpers can't be the solution.
@dominik-bln this wouldn't work from a helm chart repository, because -f values-dev.yaml expects a local file (although values-dev.yaml is packaged into the chart), or would it?

@distorhead

This comment has been minimized.

Copy link
Contributor

distorhead commented Jul 12, 2017

There is another way of doing environment-specific values without using separate values files.

  1. Define values as:
mysql:
    user:
        _default: myproject
        staging: myproject_staging
        testing: myproject_testing
    database:
        _default: myproject
        staging: myproject_staging
        testing: myproject_testing
...
  1. Pass environment as value from cli: helm --set "global.env=staging" ...
  2. Use values in templates that way:
- name: DATABASE_URL
  value: "{{ printf "mysql2://%s:password@mysql-host/%s" (pluck .Values.global.env .Values.mysql.user | first | default .Values.mysql.user._default) (pluck .Values.global.env .Values.mysql.database | first | default .Values.mysql.database._default) }}"
@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Jul 12, 2017

I am wondering whether anybody tested my scenario and confirmed that it is a bug with version 2.5 or the problem is somewhere in my local setup.

@thomastaylor312

This comment has been minimized.

Copy link
Collaborator

thomastaylor312 commented Jul 14, 2017

@polasekr I think @jascott1 has been busy trying to nail down a nasty locking bug, so he hasn't had time to test this one

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Jul 25, 2017

@polasekr Can you please post your storm-prd and storm-dev values.yaml files? Those warnings suggest you don't have the requisite fields in your values files for importing.

@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Aug 1, 2017

This is a context of storm-dev valus.yaml file

exports:
  data:
    memcacheHost: "************.amazonaws.com:11211"
    cassandraDB: "***********"
    cassandraUser: "********"
    cassandraPass: "***********"
    postgresHost: "******************.amazonaws.com"
    postgresUser: '*******'
    postgresPass: '********'
    nexus_user: "***********"
    nexus_pass: "***********"
    nexus_url: "https://maven**********************************************.jar"
    nexus_artifact_md5: "31d7add829b588587a9f525fffafdcd2"
    storm_submit_class: "com.hubub.storm.topology.BaseSearchTopology"
    nimbus:
      ENVIRONMENT: "dev"
      nameOverride: "nimbus"
      replicaCount: 1
      annotations:
      image:
        pullSecret: secret-dockerhub-hubub
        repository: tenone/hubub-storm-docker
        #tag: "0.9.7-20170729"
        tag: "0.9.3-20170729"
        pullPolicy: Always
      env: 
        - name: ZOOKEEPER_SVC
          value: zookeeper-csvc-legacy
        - name: NIMBUS_SVC 
          value: localhost
        - name: STORM_LOCALDIR
          value: /srv
      volumes: []
      serviceType: ClusterIP
      service:
        - name: storm-nimbus
          port: 6627
          targetPort: 6627
      ingress:
        enabled: false
      args: [ "storm","nimbus" ]
      resources:
        limits:
          cpu: 1
          memory: 5000Mi
        requests:
          cpu: 10m
          memory: 500Mi
    supervisor:
      ENVIRONMENT: "dev"
      nameOverride: "nimbus"
      replicaCount: 1
      annotations:
      image:
        pullSecret: secret-dockerhub-hubub
        repository: tenone/hubub-storm-docker
        tag: "0.9.3-20170729"
        pullPolicy: Always
      env: 
        - name: ZOOKEEPER_SVC
          value: zookeeper-csvc-legacy
        - name: NIMBUS_SVC 
          value: nimbus-legacy
        - name: STORM_LOCALDIR
          value: /srv
      volumes:  []
      serviceType: ClusterIP
      service:
        - name: storm-nimbus
          port: 2181
          targetPort: 2181
      ingress:
        enabled: false
      args: [ "storm","supervisor" ]
      resources:
        limits:
          cpu: 1
          memory: 5000Mi
        requests:
          cpu: 10m
          memory: 128Mi
      volumes:
        - volume_source_name: storm-properties-legacy
          mnt_point: /etc/storm
          volume_type: secret
    ui:
      ENVIRONMENT: "dev"
      nameOverride: "storm-ui"
      replicaCount: 1
      annotations:
      image:
        pullSecret: secret-dockerhub-hubub
        repository: tenone/hubub-storm-docker
        tag: "0.9.3-20170729"
        pullPolicy: Always
      env: 
        - name: ZOOKEEPER_SVC
          value: zookeeper-csvc-legacy
        - name: NIMBUS_SVC 
          value: nimbus-legacy
        - name: STORM_LOCALDIR
          value: /srv
      volumes: []
      serviceType: ClusterIP
      service:
        - name: http
          port: 8080
          targetPort: 8080
      ingress:
        enabled: false
      args: [ "storm","ui" ]
      resources:
        limits:
          cpu: 1
          memory: 500Mi
        requests:
          cpu: 10m
          memory: 128Mi
    submit:
      ENVIRONMENT: "dev"
      nameOverride: "submit-job"
      annotations:
      image:
        pullSecret: secret-dockerhub-hubub
        repository: tenone/hubub-storm-docker
        tag: "0.9.3-20170729"
        pullPolicy: Always
      env: 
        - name: ZOOKEEPER_SVC
          value: zookeeper-csvc-legacy
        - name: NIMBUS_SVC 
          value: nimbus-legacy
        - name: STORM_LOCALDIR
          value: /srv
      volumes:  []
      service:
        name: http
        type: ClusterIP
        externalPort: 2181
        internalPort: 2181
      ingress:
        enabled: false
      args: [ "bash","/etc/storm/submit_script" ]
      resources:
        limits:
          cpu: 1
          memory: 1500Mi
        requests:
          cpu: 10m
          memory: 128Mi
      volumes:
        - volume_source_name: storm-properties-legacy
          mnt_point: /etc/storm
          volume_type: secret
@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Aug 1, 2017

This is a context of storm-prd valus.yaml file

exports:
  data:
    ENVIRONMENT: "prd"
@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Aug 25, 2017

Has been there any progress made on this case? I would like to upgrade to latest helm release, but this is a blocker. Any help appreciated.

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Aug 25, 2017

Sorry for the delay @polasekr this fell off my radar for some reason. Im going to take a look at it now.

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Aug 25, 2017

@polasekr Are the storm-dev and storm-prd charts the same other than values?

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Aug 26, 2017

@polasekr I created tests charts to reflect this and it is working for me on helm v2.6.x but I do see the warnings and have confirmed this is a bug. The ImportValues processor is downstream of EnableModules but isn't getting a filtered dependency list, hence the warnings when ImportValues looks for a disabled chart's table. Hopefully I can get a PR in early next week.

In the meanwhile do you think you can test with 2.6.0? Like I said although I get the warnings it is working for me.

@jascott1 jascott1 added bug and removed question/support labels Aug 26, 2017

@polasekr

This comment has been minimized.

Copy link
Author

polasekr commented Aug 27, 2017

Hi jascott1,

@polasekr Are the storm-dev and storm-prd charts the same other than values?

Yes, storm-dev and storm-prd have just differnet values, there is nothing else there.

I tested helm 2.6.0 as you suggested and it works fine in terms of main chart seeing values from sub-charts and templates of sub-charts are rendered correctly.
Unfortunately I also encounter different behavior from version 2.4.2.

My requirements.yaml are as follows:

dependencies:
  - name: hubub-dev
    repository: "file://charts/hubub-dev"
    version: ">= 0.0.1"
    tags:
      - dev-values
    import-values:
      - data

  - name: hubub-prd
    repository: "file://charts/hubub-prd"
    version: ">= 0.0.1"
    tags:
      - prd-values
    import-values:
      - data

  - name: secret-hubub-config
    version: ">= 0.0.1"
    repository: "file://../hubub-subcharts/secret-hubub-config"

  - name: secret-dockerhub
    version: ">= 0.0.1"
    repository: "file://../hubub-subcharts/secret-dockerhub"
    
    .
    .
    .

Tree structure looks like this:

hubub
├── charts
│   ├── hubub-dev
│   │   ├── Chart.yaml
│   │   └── values.yaml
│   └── hubub-prd
│       ├── Chart.yaml
│       └── values.yaml
├── Chart.yaml
├── requirements.lock
├── requirements.yaml
├── templates
│   ├── _configmap.yaml
│   ├── _deployment_multideployment.yaml
│   ├── _deployment_multi_elb.yaml
│   ├── _deployment.yaml
│   ├── _helpers.tpl
│   ├── _job_global_values_placeholder.yaml
│   ├── _pvc.yaml
│   ├── _secret-fromfiles.yaml
│   ├── _secret-fromtemplates.yaml
│   ├── _svc_internal.yaml
│   ├── _svc_multideployment_individual.yaml
│   ├── _svc_multideployment.yaml
│   ├── _svc_multi_elb.yaml
│   └── _svc.yaml
└── values.yaml

Sub-charts hubub-dev and hubub-prd are in charts subdirectory of hubub chart. When I do

helm dependency update --debug

I am getting following error:

Error: chart metadata (Chart.yaml) missing
robert@razer-laptop:~/src/hubub-kubernetes$ cd hubub
robert@razer-laptop:~/src/hubub-kubernetes/hubub$ helm.2.6.0 dependency update --debug
Repository from local path: file://charts/hubub-dev
Repository from local path: file://charts/hubub-prd
Repository from local path: file://../hubub-subcharts/secret-hubub-config
Repository from local path: file://../hubub-subcharts/secret-dockerhub
Repository from local path: file://../hubub-subcharts/api
Repository from local path: file://../hubub-subcharts/core
Repository from local path: file://../hubub-subcharts/graylog
Repository from local path: file://../hubub-subcharts/object
Repository from local path: file://../hubub-subcharts/reverse-proxy
Repository from local path: file://../hubub-subcharts/rss-bridge
Repository from local path: file://../hubub-subcharts/rss-gen
Repository from local path: file://../hubub-subcharts/search
Repository from local path: file://../hubub-subcharts/stream
Repository from local path: file://../hubub-subcharts/ui
Repository from local path: file://../hubub-subcharts/user
Repository from local path: file://../hubub-subcharts/zookeeper
Repository from local path: file://../hubub-subcharts/storm
Repository from local path: file://../hubub-subcharts/scrape
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
        Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: getsockopt: connection refused
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 18 charts
Archiving hubub-dev from repo file://charts/hubub-dev
Save error occurred:  directory /home/robert/src/hubub-kubernetes/hubub/charts/hubub-dev not found
Deleting newly downloaded charts, restoring pre-update state
Error: directory /home/robert/src/hubub-kubernetes/hubub/charts/hubub-dev not found

The directory exists and have correct permissions, just the problem seems that it is inside hubub chart.
Same command works fine in version 2.4.2. In order to make it working, I had to move those two sub-charts out of hubub/chart directory and update references accordingly in requirements.yaml to make it work.
I am not sure whether it is a change of behavior or it is a bug.

@bacongobbler

This comment has been minimized.

Copy link
Member

bacongobbler commented Nov 1, 2017

That was a bug which was fixed in helm v2.6.1: https://github.com/kubernetes/helm/releases/tag/v2.6.1

I'd suggest upgrading and trying again, @polasekr :)

@jascott1

This comment has been minimized.

Copy link
Collaborator

jascott1 commented Dec 4, 2017

Hi @polasekr

Did @bacongobbler 's suggestion resolve this for you?

@DanielCorral

This comment has been minimized.

Copy link

DanielCorral commented Dec 18, 2017

I'm on version 2.7.2 and I'm still hitting the issues outlined by @polasekr.
I have an almost identical setup, where the parent chart's requirements.yaml is referencing subcharts and using import-values according to which tags has been specified.
Warning: ImportValues missing table: no table named "exports" (map[global:map[]])

jascott1 added a commit to jascott1/helm that referenced this issue Jan 9, 2018

fix(helm): fix importValues warnings from disabled charts
ImportValues incorrectly processes charts that should be disabled. This
patch excludes those charts.

Closes #2620
@sgandon

This comment has been minimized.

Copy link

sgandon commented Jan 31, 2018

Hello there, any idea when the PR for fixing this will be merged ?

@esmet

This comment has been minimized.

Copy link

esmet commented Feb 2, 2018

+1 for the proposed solution to allow certain values files to be selected from inside the chart based on a new concept of "environment" - I am currently working around this with custom tooling, like most others.

@vishal-biyani

This comment has been minimized.

Copy link

vishal-biyani commented Apr 12, 2018

Seems to be broken in 2.8.x as well. A short note with example charts: https://github.com/vishal-biyani/helm-subchart-env-values

@johngmyers

This comment has been minimized.

Copy link

johngmyers commented Apr 20, 2018

We handle this by putting an additional yaml file into the chart and installing with a wrapper that extracts the relevant section of that yaml file and feeds it to helm with -f.

We use two levels: cluster and namespace. We find the term "environment" to be ambiguous through overuse.

splisson added a commit to splisson/helm that referenced this issue Dec 6, 2018

fix(helm): fix importValues warnings from disabled charts
ImportValues incorrectly processes charts that should be disabled. This
patch excludes those charts.

Closes helm#2620
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment