This is a project intended to document and automate the process required for a PKS + NSX-T deployment on vSphere.
These documents are not public yet. They are linked here for VMware internal users, but should be converted over time into publicly consumable documents.
- Read Niran's step-by-step NSX-T deploy for a step-by-step manual deploy using some automation
- Use the PoC Planning Reference document to vet the planned deployment
- Use the PKS Configuration Worksheet to identify and track needed configuration that must be captured as YAML for the pipelines to work properly
High level end-to-end PKS deploment
The overall process for a PKS and NSX-T deployment is
- Start with a new vcenter, or a new cluster in an existing vcenter
- Deploy a PKS deployment server. This has Pivotal Concourse for running pipelines, and all the needed binaries and tools to do an automated deploy of PKS and NSX-T
- Use a default configuration YAML or create a new one for NSX-T and another for PKS. These describe what the final deployment will look like
- Apply the pipelines to your configuration
- Connect to the Concourse UI
- Trigger pipelines to:
- deploy NSX-T
- deploy PKS
To get the inital OVA, you must bootstrap. That process looks is:
- Start with a machine with access to a vCenter
- Download this code as described below
- Create a container with tools needed to operate on vCenter
- Deploy a ubuntu 16.04 cloud image into vCenter
- Boot the stock VM using cloudinit to set usernames/passwords/ssh keys
- Run ansible playbooks against the VM to provision everything needed to make a deploy server, including:
- install concourse
- download and host needed binaries
- host container images needed by concourse
At this point, you have two choices:
- export the VM as an OVA for a future deployment
- use the running VM to perform a deploy now
Assuming you want to do these things, continue into the details of this process below:
Get the code
Do not clone this repository. Instead, install Google Repo.
Here's a quick google repo install for the impatient.
# Validate python python2.7 -c "print 'Python OK'" || echo 'Need python 2.7!' python --version | grep "Python 2" || echo 'Warning: python 3 is default!' mkdir ~/bin PATH=~/bin:$PATH curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo chmod a+x ~/bin/repo # If you get a warning that about python 3, you might run this: # After repo is installed: sed -ri "1s:/usr/bin/env python:/usr/bin/python2.7:" ~/bin/repo
Once you've installed Google Repo, you will use it to download and assemble all the component git repositories.
This process is as follows:
mkdir pks-deploy-testing cd pks-deploy-testing repo init -u https://github.com/vmware/vmware-pks-deploy-meta.git # or, with ssh: (you will have first had to register an SSH key with Github) repo init -u email@example.com:vmware/vmware-pks-deploy-meta.git # Then sync, which pulls down the code. repo sync
After pulling down all the code as described above, go into
and you'll see there are several directories. These are each a git repository.
We'll focus on the
This directory contains code that will create a VM in vCenter, install Concourse, ansible, and other tools into that VM.
You can use an existing OVA captured after doing this process once, or you can go into the bootstrap directory and follow the readme there to create the VM directly in vCenter.
This should take about 15 minutes.
Ssh into the jumpbox
Get the ip of the vm created in the bootstrap step above. If you set up ssh keys, you can ssh right now, otherwise use:
On the jumpbox, there is also a copy of the source you used to bootstrap at
Download VMware bits
If you passed the following variables into the bootstrap process above,
the required binaries will be downloaded as part of the automation:
If you did not pass those in, then you'll need to run this step manually as described below.
Go into the jumpbox directory
and follow the readme there to pull needed bits from http://my.vmware.com and pivnet.
You can see an online version in downloads.
The downloaded files will be hosted via s3 by minio and
can be accessed at
Apply various pipelines
On the jumpbox, the pipelines exist at
/home/vmware/deployroot and concourse is running on
http://jumpbox-ip:8080 with the same credentials as ssh to log in.
You can use fly from the jumpbox to apply the pipelines. To log in try
fly --target main login -c http://localhost:8080 and
fly pipelines --target main
/home/vmware/deployroot/nsx-t-gen and follow the guide from sparameswaran/nsx-t-gen.
Anther good guide is from Sabha
A sample config file is at
/home/vmware/deployroot/deploy-params/one-cloud-param.yaml on the jumpbox, or live here.
There is also good coverage of the config file needed in Niran's guide from above starting in section 4.b.
Once you have the config file correct:
cd /home/vmware/deployroot/nsx-t-gen fly --target main login -c http://localhost:8080 -u vmware -p 'VMware1!' fly -t main set-pipeline -p deploy-nsx -c pipelines/nsx-t-install.yml -l ../pks-deploy/one-cloud-nsxt-param.yaml fly -t main unpause-pipeline -p deploy-nsx
Install PAS and/or PKS
/home/vmware/deployroot/nsx-t-ci-pipeline and follow the guide from sparameswaran/nsx-t-ci-pipeline
cd /home/vmware/deployroot/nsx-t-ci-pipeline fly --target main login -c http://localhost:8080 -u vmware -p 'VMware1!' fly -t main set-pipeline -p deploy-pks -c pipelines/install-pks-pipeline.yml -l ../pks-deploy/pks-params.sample.yml fly -t main unpause-pipeline -p deploy-pks
The vmware-pks-deploy project team welcomes contributions from the community. Before you start working with vmware-pks-deploy, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.
For development, you will clone this repository and submit PRs back to upstream. This is intended to be used as a sub project pulled together by a meta-project called vmware-pks-deploy-meta. You can get the full set of repositories by follow the prep section above.
Copyright © 2018 VMware, Inc. All Rights Reserved.