vmware-pks-deploy

This is a project intended to document and automate the process required for a PKS + NSX-T deployment on vSphere.

Extra documents

These documents are not public yet. They are linked here for VMware internal users, but should be converted over time into publicly consumable documents.

• Read Niran's step-by-step NSX-T deploy for a step-by-step manual deploy using some automation
• Use the PoC Planning Reference document to vet the planned deployment
• Use the PKS Configuration Worksheet to identify and track needed configuration that must be captured as YAML for the pipelines to work properly

High level end-to-end PKS deploment

The overall process for a PKS and NSX-T deployment is

• Start with a new vcenter, or a new cluster in an existing vcenter
• Deploy a PKS deployment server. This has Pivotal Concourse for running pipelines, and all the needed binaries and tools to do an automated deploy of PKS and NSX-T
• Use a default configuration YAML or create a new one for NSX-T and another for PKS. These describe what the final deployment will look like
• Apply the pipelines to your configuration
• Connect to the Concourse UI
• Trigger pipelines to:
  • deploy NSX-T
  • deploy PKS

To get the inital OVA, you must bootstrap. That process looks is:

• Start with a machine with access to a vCenter
• Download this code as described below
• Create a container with tools needed to operate on vCenter
• Deploy a ubuntu 16.04 cloud image into vCenter
• Boot the stock VM using cloudinit to set usernames/passwords/ssh keys
• Run ansible playbooks against the VM to provision everything needed to make a deploy server, including:
  • install concourse
  • download and host needed binaries
  • host container images needed by concourse

At this point, you have two choices:

• export the VM as an OVA for a future deployment
• use the running VM to perform a deploy now

Assuming you want to do these things, continue into the details of this process below:

Get the code

Do not clone this repository. Instead, install Google Repo.

Here's a quick google repo install for the impatient.

# Validate python
python2.7 -c "print 'Python OK'" || echo 'Need python 2.7!'
python --version | grep "Python 2" || echo 'Warning: python 3 is default!'
mkdir ~/bin
PATH=~/bin:$PATH
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo
# If you get a warning that about python 3, you might run this:
# After repo is installed:
sed -ri "1s:/usr/bin/env python:/usr/bin/python2.7:" ~/bin/repo

Once you've installed Google Repo, you will use it to download and assemble all the component git repositories.

This process is as follows:

mkdir pks-deploy-testing
cd pks-deploy-testing
repo init -u https://github.com/vmware/vmware-pks-deploy-meta.git
# or, with ssh: (you will have first had to register an SSH key with Github)
repo init -u git@github.com:vmware/vmware-pks-deploy-meta.git
# Then sync, which pulls down the code.
repo sync

After pulling down all the code as described above, go into pks-deploy-testing and you'll see there are several directories. These are each a git repository.

We'll focus on the pks-deploy repository.

Bootstrapping

Go into pks-deploy/bootstrap. This directory contains code that will create a VM in vCenter, install Concourse, ansible, and other tools into that VM.

You can use an existing OVA captured after doing this process once, or you can go into the bootstrap directory and follow the readme there to create the VM directly in vCenter.

This should take about 15 minutes.

Ssh into the jumpbox

Get the ip of the vm created in the bootstrap step above. If you set up ssh keys, you can ssh right now, otherwise use:

• Username: vmware
• Password: VMware1!

On the jumpbox, there is also a copy of the source you used to bootstrap at /home/vmware/deployroot.

Download VMware bits

If you passed the following variables into the bootstrap process above, the required binaries will be downloaded as part of the automation: PIVNET_API_TOKEN, MY_VMWARE_USER, and MY_VMWARE_PASSWORD. If you did not pass those in, then you'll need to run this step manually as described below.

Go into the jumpbox directory /home/vmware/deployroot/pks-deploy/downloads, and follow the readme there to pull needed bits from http://my.vmware.com and pivnet. You can see an online version in downloads.

The downloaded files will be hosted via s3 by minio and can be accessed at http://bootstrap-box-ip:9091.

Apply various pipelines

On the jumpbox, the pipelines exist at /home/vmware/deployroot and concourse is running on http://jumpbox-ip:8080 with the same credentials as ssh to log in. You can use fly from the jumpbox to apply the pipelines. To log in try fly --target main login -c http://localhost:8080 and fly pipelines --target main

Install NSX-T

cd /home/vmware/deployroot/nsx-t-gen and follow the guide from sparameswaran/nsx-t-gen.

Anther good guide is from Sabha

A sample config file is at /home/vmware/deployroot/deploy-params/one-cloud-param.yaml on the jumpbox, or live here.

There is also good coverage of the config file needed in Niran's guide from above starting in section 4.b.

Once you have the config file correct:

cd /home/vmware/deployroot/nsx-t-gen
fly --target main login -c http://localhost:8080 -u vmware -p 'VMware1!'
fly -t main set-pipeline -p deploy-nsx -c pipelines/nsx-t-install.yml -l ../pks-deploy/one-cloud-nsxt-param.yaml
fly -t main unpause-pipeline -p deploy-nsx

Install PAS and/or PKS

cd /home/vmware/deployroot/nsx-t-ci-pipeline and follow the guide from sparameswaran/nsx-t-ci-pipeline

In particular, this is the pipeline and here is a sample param file.

cd /home/vmware/deployroot/nsx-t-ci-pipeline
fly --target main login -c http://localhost:8080 -u vmware -p 'VMware1!'
fly -t main set-pipeline -p deploy-pks -c pipelines/install-pks-pipeline.yml -l ../pks-deploy/pks-params.sample.yml
fly -t main unpause-pipeline -p deploy-pks

Contributing

The vmware-pks-deploy project team welcomes contributions from the community. Before you start working with vmware-pks-deploy, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.

Development

For development, you will clone this repository and submit PRs back to upstream. This is intended to be used as a sub project pulled together by a meta-project called vmware-pks-deploy-meta. You can get the full set of repositories by follow the prep section above.

License

Copyright © 2018 VMware, Inc. All Rights Reserved.

SPDX-License-Identifier: MIT