New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Support using environment variables inside deployment yaml file #52787

Closed
mahdix opened this Issue Sep 20, 2017 · 16 comments

Comments

Projects
None yet
@mahdix
Contributor

mahdix commented Sep 20, 2017

Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST

/kind feature

What happened: When you write a deployment yaml file, everything must be specified statically. In cases where something is not specified at time of writing, the only choice is using a 3rd-party tool and use yaml file as a template which will later be processed by the tool to replace some markers with actual values. This is not efficient.

What you expected to happen: Deployment yaml file format should support some kind of marker which specifies values which will be provided at the time of deployment via command line arguments (e.g. kubectl create -f myfile.ymml --var1=DATA --var2=gcr.io/image121212) or feed values from environment variables.

How to reproduce it (as minimally and precisely as possible):
Example (just a suggestion, actual implementation could be different):

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: {{IMAGE_URL}}

Then run: kubectl create -f file.yaml --IMAGE_URL=nginx.1.9

Anything else we need to know?: This will make life easier for DevOps.

Environment:

  • Kubernetes version (use kubectl version): Latest
  • Cloud provider or hardware configuration**:
  • OS (e.g. from /etc/os-release): OS X
  • Kernel (e.g. uname -a): Darwin
  • Install tools:
  • Others:
@mahdix

This comment has been minimized.

Contributor

mahdix commented Sep 20, 2017

@kubernetes/sig-cli-feature-request

@resouer

This comment has been minimized.

Member

resouer commented Sep 20, 2017

I am not sure about this. Sounds more like a PaaS layer feature to me ...

@zjj2wry

This comment has been minimized.

Member

zjj2wry commented Sep 29, 2017

@mahdix kubectl create command now just add image, because we want it enough simple. but now 1.8 support kubectl set env command.

you can use kubectl create deploy a --image=xxx --dry-run -o yaml | kubectl set env --local -f - 'env=prod' -o yaml . hope this can help you

@fengh0409

This comment has been minimized.

fengh0409 commented Dec 8, 2017

Hey, I like this feature,and I had met this situation actually.How is this going?

@hex0cter

This comment has been minimized.

hex0cter commented Dec 14, 2017

I would absolutely love it.

@kminehart

This comment has been minimized.

kminehart commented Dec 29, 2017

Nobody has mentioned envsubst.

For a small project I put everything in a folder, kubernetes/.

Anything ending in .yaml is ignored, and anything ending in .tmpl is environment-variable substituted and placed into ./bin/kubernetes.

The deploy step just runs a kubectl apply -f on ./bin/kubernetes

###############################################################################
# Build Kubernetes yaml files
###############################################################################
BUILD_DIR = ./bin
K8S_DIR = ./kubernetes
K8S_TMPL := $(notdir $(wildcard $(K8S_DIR)/*.tmpl))
build_yaml:
ifneq ("","$(wildcard $(K8S_DIR))")
	@mkdir -p $(BUILD_DIR)/kubernetes
	@cp $(K8S_DIR)/*.yaml $(BUILD_DIR)/kubernetes
	$(info $(M) building yaml...)
	for tmpl in $(K8S_TMPL); do \
		VERSION=$(VERSION) envsubst < $(K8S_DIR)/$$tmpl > $(BUILD_DIR)/kubernetes/"$${tmpl/.tmpl/}" ; \
	done
endif

deploy: build_yaml
ifneq ("","$(wildcard $(K8S_DIR))")
	$(info $(M) deploy...)
	kubectl apply -f $(BUILD_DIR)/kubernetes
endif

And in my deployment.yaml.tmpl:

        image: example/image:${VERSION}
@daniel-kun

This comment has been minimized.

daniel-kun commented Jan 5, 2018

I face a similar problem with yaml template files that include CPU architecture, such as amd64. I often had to change those manually to arm when deploying on a raspberry pi for testing purposes. One popular example is metrics-server, where amd64 is hard-coded in the yaml:
https://github.com/kubernetes-incubator/metrics-server/blob/master/deploy/metrics-server-deployment.yaml

@kminehart

This comment has been minimized.

kminehart commented Jan 5, 2018

@daniel-kun I'm going to assume that you're using Go, but i guess that doesn't have to be true for this to work.

envsubst is a great tool here, especially since you're already compiling your code using $GOOS if you're using Go.

Template:

apiVersion: extensions/v1beta1
kind: Deployment
# ...
    architecture: ${GOOS}
GOOS=amd64 envsubst < mytemplate.tmpl > mydeployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
# ...
    architecture: amd64

I think there are plenty of tools available for this; it doesn't have to be embedded into kubectl. :)

@liggitt

This comment has been minimized.

Member

liggitt commented Jan 6, 2018

See discussion and resolution of this using external templating/creation tools in #23896 (comment)

@liggitt liggitt closed this Jan 6, 2018

@hamshif

This comment has been minimized.

hamshif commented Feb 13, 2018

kubctl should enable templates and insertion to avoid DRY(don't repeat yourself)

One simple example are DEV, QA, PROD environments where sanboxing images is mission critical and the image value of containers should vary with arguments.

Another is using similar deployments for minikube development and cloud deployment where the only differences are mounting volumes and replicas

Another example is deploying to a feature branch with less CPU

@hamshif

This comment has been minimized.

hamshif commented Feb 13, 2018

Thanks for the envsubst tip
it is a good solution on mac I would incorporate it to kubectl

@jonahbron

This comment has been minimized.

jonahbron commented Feb 22, 2018

Another good solution would be to allow value overwriting, in the same way that Helm does. Something like:

kubectl create -f deployment.yaml --set spec.template.spec.containers[0].image=image_url
@daniel-kun

This comment has been minimized.

daniel-kun commented Feb 23, 2018

@jonahbron I think this is a great idea, with great flexibility and without re-inventing the wheel. It's a bit similar to docker build args in a way. (Although I'd still find it useful to have build args just like docker for kubectl.)

@jonahbron

This comment has been minimized.

jonahbron commented Feb 28, 2018

@daniel-kun Yes, build args would be another nice option.

@jonahbron

This comment has been minimized.

jonahbron commented Feb 28, 2018

For the record, the simple solution I settled on was to feed the file through sed before kubectl.

cat deployment.yaml.tmpl | \
sed 's/\$BRANCH_NAME'"/$BRANCH_NAME/g" | \
sed 's/\$COMMIT_ID'"/$COMMIT_ID/g" | \
kubectl apply -f -

In practice, I put the replacements into a bash script file, so invocation looks more like:

cat deployment.yaml.tmpl | config.sh | kubectl apply -f -

Which works really well in my Jenkins pipeline. And it's helpful to not depend on any external packages like envsubst in environments where it's not immediately available.

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented Mar 1, 2018

kubectl will never support variable substitution.

Background document: https://goo.gl/T66ZcD

I suggest building an extension for Helm and participating in the Helm v3 discussions. My aim is to make Helm more flexible, to address such cases: https://docs.google.com/presentation/d/10dp4hKciccincnH6pAFf7t31s82iNvtt_mwhlUbeCDw/edit#slide=id.p

Please participate in SIG Apps and the Application Definition Working Group.

@kubernetes kubernetes locked as resolved and limited conversation to collaborators Mar 1, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.