Skip to content
Browse files

Expands info on config with kustomize (#1093)

* Moved the kustomize doc to the config section.

* Expanded the kustomize guide and fixed links.

* More content tweaks.

* Still more content tweaks.

* Added punctuation and indents as requested in review comments.

* Added a section on using the init command.

* WIP need more detail on how to edit KfDef.

* Added a better customisation example.
  • Loading branch information...
sarahmaddox authored and k8s-ci-robot committed Aug 28, 2019
1 parent 466c106 commit 47388737fcd09fd3c913c2271d590a0385b1ae76
@@ -87,11 +87,13 @@ docs/started/requirements/ /docs/started/getting-started/
/docs/started/getting-started-gke/ /docs/started/cloud/getting-started-gke/
/docs/started/getting-started-icp/ /docs/started/cloud/getting-started-icp/

/docs/other-guides/job-scheduling/ /docs/use-cases/job-scheduling/
/docs/other-guides/upgrade/ /docs/upgrading/upgrade/

/docs/use-cases/kubeflow-on-multinode-cluster/ /docs/other-guides/kubeflow-on-multinode-cluster/

# Remove Kubeflow installation on existing EKS cluster
/docs/aws/deploy/existing-cluster/ /docs/aws/deploy/install-kubeflow/
/docs/aws/deploy/existing-cluster/ /docs/aws/deploy/install-kubeflow/

# Move the kustomize guide to the config section
/docs/components/misc/kustomize/ /docs/other-guides/kustomize/

This file was deleted.

@@ -189,7 +189,7 @@ Your Kubeflow app directory **${KFAPP}** contains the following files and direct

* **kustomize** is a directory that contains the kustomize packages for Kubeflow
applications. See
[how Kubeflow uses kustomize](/docs/components/misc/kustomize/).
[how Kubeflow uses kustomize](/docs/other-guides/kustomize/).

* The directory is created when you run `kfctl generate`.
* You can customize the Kubernetes resources by modifying the manifests and
@@ -0,0 +1,259 @@
title = "Configuring Kubeflow with kfctl and kustomize"
description = "The basics of Kubeflow configuration with kfctl and kustomize"
weight = 10

Kfctl is the Kubeflow command-line interface (CLI) that you can use to
install and configure Kubeflow.

Kubeflow makes use of [kustomize]( to help customize YAML
configurations. With kustomize, you can traverse a Kubernetes manifest to add,
remove, or update configuration options without forking the manifest. A
_manifest_ is a YAML file containing a description of the applications that you
want to include in your Kubeflow deployment.

## Overview of kfctl and kustomize

This section describes how kfctl works with kustomize to set up your
Kubeflow deployment.

### The Kubeflow installation process

Kfctl is the Kubeflow CLI that you can use to set up a Kubernetes cluster with
Kubeflow installed, or to deploy Kubeflow to an existing Kubernetes cluster.
See the [Kubeflow getting-started guide](/docs/started/getting-started/) for
installation instructions based on your deployment scenario.

The deployment process consists of three steps, _init_, _generate_, and
_apply_, so that you can modify your configuration before actually deploying

* `kfctl init` - one time set up.
* `kfctl generate` - creates config files defining the various resources.
* `kfctl apply` - creates or updates the resources.

### Specifying a configuration file when initializing your deployment

When you install Kubeflow, the deployment process uses one of a few possible
YAML configuration files to bootstrap the configuration.

Typically, you specify the configuration file with a `--config` parameter
when you run `kfctl init`. For example:

export KFAPP="<your choice of application directory name>"
export CONFIG="{{% kf-latest-version %}}/bootstrap/config/kfctl_existing_arrikto.yaml"
kfctl init ${KFAPP} --config=${CONFIG} -V

*For details of the above deployment, see the guide to deployment using the
[kfctl_existing_arrikto configuration](/docs/started/k8s/kfctl-existing-arrikto/).*

Some deployment processes use a default config file and you don't need to
add the `--config` argument. For example, the
Google Cloud Platform (GCP) initialization command looks like this:

export PROJECT="<your GCP project ID>"
export KFAPP="<your choice of application directory name>"
kfctl init ${KFAPP} --platform gcp --project ${PROJECT}

*For details of the above deployment, see the guide to deployment
[on GCP using the CLI](/docs/gke/deploy/deploy-cli/).*

### Your Kubeflow directory layout

Your Kubeflow app directory is the directory where you choose to store
your Kubeflow configurations during deployment. This guide refers to the
directory as `${KFAPP}`. The directory contains the following files and

* **app.yaml** stores your primary Kubeflow configuration in the form of a
`KfDef` Kubernetes object.

* The values are set when you run `kfctl init`.
* The values are snapshotted inside `app.yaml` to make your app
self contained.
* The YAML defines each Kubeflow application as a kustomize package.

* **[platform-name]_config** is a directory that contains
configurations specific to your chosen platform or cloud provider. This
directory may or may not be present, depending on your setup.

* The directory is created when you run `kfctl generate platform`.
* To customize these configurations, you can modify parameters in the `KfDef`
object in your `app.yaml` file, and then re-run `kfctl generate` and
`kfctl apply`.

* **kustomize** is a directory that contains Kubeflow application manifests.
That is, the directory contains the kustomize packages for the Kubeflow
applications that are included in your deployment.

* The directory is created when you run `kfctl generate`.
* To customize these configurations, you can modify parameters in the `KfDef`
object in your `app.yaml` file, and then re-run `kfctl generate` and
`kfctl apply`.

### How your configuration is generated

The content of your `app.yaml` is the result of running kustomize
on the base and overlay `kustomization.yaml` files in the
[Kubeflow config](
directory. The overlays reflect the options that you choose when calling
`kfctl init`.

Below are some examples of `KfDef` configuration files:

* [kfctl_k8s_istio.yaml](
to install Kubeflow on an existing Kubernetes cluster.
* [kfctl_gcp_basic_auth.yaml](
to set up a Google Kubernetes Engine (GKE) cluster with Kubeflow using basic

The kustomize package manager in kfctl uses the information in your
`app.yaml` to traverse the directories under the
[Kubeflow manifests]( and to
create kustomize build targets based on the manifests.

## Installing kustomize

Make sure that you have the minimum required version of kustomize:
<b>{{% kustomize-min-version %}}</b> or later.

1. Follow the [kustomize installation
choosing the relevant options for your operating system. For example, if
you're on Linux:

* Set some variables for the operating system:

export opsys=linux

* Download the kustomize binary:

curl -s |\
grep browser_download |\
grep $opsys |\
cut -d '"' -f 4 |\
xargs curl -O -L

* Move the binary:

mkdir -p ${HOME}/bin
mv kustomize_*_${opsys}_amd64 ${HOME}/bin/kustomize
chmod u+x ${HOME}/bin/kustomize

1. Include the `kustomize` command in your path:

export PATH=$PATH:${HOME}/bin/kustomize

## Modifying configuration before deployment

Kustomize lets you customize raw, template-free YAML files for multiple
purposes, leaving the original YAML untouched and usable as is.

You can use the following command to build and apply kustomize directories:

kustomize build <kustomization_directory> | kubectl apply -f -

The [Kubeflow manifests repo]( contains
kustomize build targets, each with a `base` directory. You can use kustomize to
generate YAML output and pass it to kfctl. You can also make
changes to the kustomize targets in the manifests repo as needed.

## Modifying the configuration of an existing deployment

To customize the Kubeflow resources running within the cluster, you can modify
parameters in the `KfDef` object in your `${KFAPP}/app.yaml` file, where
`KFAPP` is the directory where you stored your Kubeflow configurations during
deployment. Then re-run `kfctl generate` and `kfctl apply`.

For example, to modify settings for the Spartakus usage reporting tool within
your Kubeflow deployment:

1. Edit the configuration file at `${KFAPP}/app.yaml`.

1. Find and replace the parameter values for `spartakus` to suit your

- kustomizeConfig:
- initRequired: true
name: usageId
value: <randomly-generated-id>
- initRequired: true
name: reportUsage
value: "true"
name: manifests
path: common/spartakus
name: spartakus

1. Due to
[Kubeflow issue #3810](,
you currently need to tear down your Kubeflow cluster before applying the
configuration change. This command **deletes all your Kubeflow resources**:

cd ${KFAPP}
kfctl delete all -V

1. Regenerate and deploy your Kubeflow resources:

cd ${KFAPP}
kfctl generate all -V
kfctl apply all -V

### More examples

For examples of customizing your deployment, see the guide to [customizing
Kubeflow on GKE](/docs/gke/customizing-gke/).

For information about how Kubeflow uses Spartakus, see the guide to
[usage reporting](/docs/other-guides/usage-reporting/).

## More about kustomize

Below are some useful kustomize terms, from the
[kustomize glossary](

* **base:** A combination of a kustomization and resource(s). Bases can be
referred to by other kustomizations.

* **kustomization:** Refers to a `kustomization.yaml` file, or more generally to
a directory containing the `kustomization.yaml` file and all the relative file
paths that the YAML file references.

* **overlay:** A combination of a kustomization that refers to a base, and a
patch. An overlay may have multiple bases.

* **patch:** General instructions to modify a resource.

* **resource:** Any valid YAML file that defines an object with a kind and a
metadata/name field.

* **target:** The argument to `kustomize build`. For example,
`kustomize build $TARGET`. A target must be a path or a URL to a
kustomization. A target can be a base or an overlay.

* **variant:** The outcome of applying an overlay to a base.

Read more about kustomize in the
[kustomize documentation](

0 comments on commit 4738873

Please sign in to comment.
You can’t perform that action at this time.