https://cloud.google.com/kubernetes-engine/kubernetes-comic/
You can interact with the Kubernetes environment in the following ways -
- GitHub - commits to master branch are continuously deployed to the relevant environment. See .travis.yaml for the continuous deployment configuration and deployed environments.
- Google Cloud Shell - The recommended and easiest way for running management commands. Just setup a Google Cloud account and enable billing (you get 300$ free, you can setup billing alerts to avoid paying by mistake). You can use the cloud shell file editor to edit files, just be sure to configure it to indentation of 2 spaces (not tabs - because they interfere with the yaml files).
- Any modern PC / OS should also work, you will just need to install some basic dependencies like Docker and Google Cloud SDK (possibly more). The main problem with working from local PC is the network connection, if you have a stable, fast connection and know how to install the dependencies, you might be better of running from your own PC.
- Docker + Google Cloud service account - for automation / CI / CD. See the sk8s-ops sub-chart for more details.
Ensure you have permissions on the relevant Google Project. Permissions are personal, so once you authenticate with your google account, you will have all permissions granted for you by the different Google Cloud projects.
To interact with the environment locally, install Google Cloud SDK and run gcloud auth login
to authenticate.
On Google Cloud Shell you are already authenticated and all dependencies are installed.
Clone the relevant repo
git clone <K8S_REPO_URL>
All following commands should run from the k8s repo root directory
cd <CLONED_K8S_REPO_DIRECTORY>
The main environments should be committed to the k8s repo under environments
directory
Each directory under environments
corresponds to an environment name which you can connect to:
source switch_environment.sh ENVIRONMENT_NAME
Make sure you are connected to the correct environment before running any of the following commands.
See the environments documentation for more details about environments and how to create a new environment / Kubernetes cluster.
Helm manages everything for us.
When deploying to the main, shared environments (e.g. staging
/ production
), check the following points first:
- The preferred way to deploy is by opening and merging a pull request - this prevents infrastructure deployment risks and is generally more secure.
- If you intend to do some infrastructure development, consider creating your own personal environment and testing on that, it's very easy to replicate an existing environment, see the environments documentation for details.
- If you want to update an attribute of a specific deployment, see the section below - Patching configuration values without Helm
If you still want to deploy directly, just make sure you are the only one working on the environment and/or update with the master branch to prevent infrastructure conflicts.
Make sure you have the latest helm installed on both client and server:
kubectl create -f https://raw.githubusercontent.com/OriHoch/sk8s/master/rbac-config.yaml
helm init --service-account tiller --upgrade
Deploy:
./helm_upgrade.sh
When helm upgrade command completes successfully it doesn't necesarily mean deployment is complete (although it often does) - it only updates the desired state.
Kubernetes / Helm have a desired state of the infrastructure and they will do their best to move to that state.
You can add arguments to ./helm_upgrade.sh
which are forwarded to the underlying helm upgrade
command.
Check the Helm documentation for more details.
Some useful arguments:
- For initial installation you should add
--install
- Depending on the changes you might need to add
--recreate-pods
or--force
- For debugging you can also use
--debug
and--dry-run
Additionally, you can to use force_update.sh
to force an update on a specific deployment.
The default values are at values.yaml
at the root of the k8s repo. These are used in the chart template files (under templates
and charts
directories)
Each environment can override these values using environments/ENVIRONMENT_NAME/values.yaml
Finally, automation scripts write values to environments/ENVIRONMENT_NAME/values.auto-updated.yaml
using the helm_update_values.sh script.
After updating a secret you should update the affected deployments, you can use ./force_update.sh
to do that
Secrets shoule be optional so you can run the environment without any secretes and will use default values similar to dev environments.
Each environment may include a script to create the environment secrets under environments/ENVIRONMENT_NAME/secrets.sh
- this file is not committed to Git.
The helm_update_values.sh script can be used to update Helm configuration values, usually from automation / CI scripts.
This method works under the following conditions:
- You want to make changes to a main / shared environment, otherwise, just do a helm upgrade.
- You want to modify a specific value in a specific resource (usually a deployment image, but other values are possible too)
- This value is represented in the Helm configuration values
This script accepts a single required argument - a json string containing the required update
It updates the values in the current environment's auto updated values file - environments/ENVIRONMENT_NAME/values.auto-updated.yaml
for example, the following command will update the image value under the spark key.
./helm_update_values.sh '{"spark":{"image":"'${IMAGE_TAG}'"}}'
Json values require quotes which may interfere with bash scripts, so you can provide it base64 encoded
B64_UPDATE_VALUES=`echo '{"spark":{"image":"'${IMAGE_TAG}'"}}' | base64 -w0`
./helm_update_values.sh $B64_UPDATE_VALUES
After ther values were updated, they should be pushed to GitHub
It's important to commit the changes to Git first and only then patch the deployment - this prevents infrastrcuture conflicts.
This script also supports updating Git from CI tools
Create a GitHub machine user and give this user write permissions to the k8s repo.
Run the script with the full parameters:
./helm_update_values.sh <YAML_OVERRIDE_VALUES_JSON> [GIT_COMMIT_MESSAGE] [GIT_REPO_TOKEN] [GIT_REPO_SLUG] [GIT_REPO_BRANCH]
Where GIT_REPO_TOKEN is the machine user's token
After GitHub was updated, patch the deployment, for example, in case of image update:
kubectl set image deployment/spark spark=${IMAGE_TAG}
You can use the following to set the token on travis:
travis env set --private K8S_OPS_GITHUB_REPO_TOKEN "*****"
External services / apps can be self-deploying using the above method for patching configurations
The continuous deployment flow is based on:
- Travis - runs the deployment script on each app's repo on commit to master branch (AKA merge of PR).
- Sk8s Ops - provides a consistent deployment environment and to securely authenticate with the service account secret.
- GitHub - for persistency of deployment environment values - GitHub maintains the state of the environment. Each app commits deployment updates to the k8s repo.
Install the Travis CLI
Enable Travis for the external repo (run travis enable
from the repo directory)
Create a travis.yml
and related scripts depending on your continuous deployment requirements
You can copy from other projects:
- Suitable for k8s repositories - it does deployment of charts and external sub-charts - https://github.com/Midburn/midburn-k8s/blob/master/.travis.yml
- Suitable for external app repositories - it builds a docker image and updates image values - https://github.com/Midburn/spark/blob/master/.travis.yml
- https://github.com/Beit-Hatfutsot/mojp-dbs-pipelines/blob/master/.travis.yml
- https://github.com/Midburn/volunteers/blob/master/.travis.yml
Modify the values / script according to your app requirements
Create a service account secret, see Sk8s Ops - Creating a new service account with full permissions and related key file
The service account secret should be encrypted for use by Travis
Assuming you have the secret-k8s-ops.json
file available at the external app's root directory:
travis encrypt-file secret-k8s-ops.json secret-k8s-ops.json.enc
Copy the openssl
command output by the above command and modify in the .travis-yml
You can use the following procudure on both Google Cloud Shell and from local PC to enable pushing changes to GitHub
Create an SSH key -
[ ! -f .ssh/id_rsa.pub ] && ssh-keygen -t rsa -b 4096 -C "${USER}@cloudshell"
cat ~/.ssh/id_rsa.pub
Add the key in github - https://github.com/settings/keys
Clone the repo
git clone git@github.com:midburn/midburn-k8s.git
see https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
See example usage here
To expose google repo publically, follow these steps