Skip to content

[Deprecated] Creating a vSphere 2012 R2 Windows Stemcell

Jump to bottom
Kartik Lunkad edited this page Aug 16, 2019 · 1 revision

Introduction

These instructions describe how to build a BOSH stemcell based on Windows Server 2012 R2 only. For building Windows Stemcells for any other versions, please refer to Create a vSphere Windows Stemcell on bosh.io

Windows 2012R2 support is end-of-life. Please refer to Create a vSphere Windows Stemcell on bosh.io for stemcell creation needs.

In order to create a vSphere stemcell by hand, you must first begin with an ISO or other VM image. This document describes using VMware Workstation, VMware Fusion, and vCenter to install the BOSH dependencies and then create a .tgz file that can be uploaded to your BOSH director and used with Cloud Foundry.

Quick overview

  1. Start from a Windows ISO in your environment of choice.
  2. Install Windows Updates
  3. Clone the VM. (Save the original VM to run on updates in the future.)
  4. Install CF binaries and other required software
  5. Sysprep the VM and Applying Security Policies
  6. Export the VM's VMDK file
  7. Convert the VMDK into a BOSH stemcell
  8. Monthly Patch Tuesday Updates

About Windows Updates and Producing New Stemcells

This process starts with an operator creating a base Windows VM from a volume-licensed ISO and subsequently maintaining that base template with all Windows recommended security updates, but without the BOSH dependencies.

This VM with security updates will serve as the base for all future stemcells, produced from clones of that base VM. This way, you can build new stemcells without having to run Windows Updates from scratch each time.

You can determine if your image needs updates by creating a VM with the image and going to control panel. If any critical or important updates are available we recommend installing updates, then rebuilding the stemcell from a clone of that original VM.

Stemcell Versions

Stemcells are versioned with a MAJOR.MINOR format, e.g. 1200.35. Take note of the version you want to build, as it will be needed to download the proper packages and in the final stembuild step 7 that packages the stemcell.

Known Issues

  • Windows2012R2 based stemcells do not support ephemeral disk. We recommend 128GB or greater for the root disk.
  • There is a known issue using special characters in the password during the sysrep step 5. We recommend only using the "!" as a special character until this issue is resolved.

Dependencies

You will need:

  • Windows Server 2012 R2 ISO from MSDN or VLSC. (Note that you can use an evaluation copy for testing, but we do not recommend this for production, as the licensing expires.)
  • Access to a vSphere account, VMware Workstation, or VMware Fusion.
  • ovftool Ensure the ovftool command is available from your command-line path on your workstation (not the Windows VM). On Windows desktop, it is installed by default in C:\Program Files\VMware\VMware OVF Tool.
  • Download the latest stembuild release
  • lgpo.exe from the Microsoft Security Toolkit.
  • Download OpenSSH v7.7.2.0p1 from OpenSSH-Win64
  • Download latest BOSH PS Modules and BOSH Agent for your target stemcell versions (1200.*)

Stemcell Build Steps

Step 1: Create base VM for stemcell

These are instructions for installing windows from a Windows installation disk ISO (either a volume licensed copy or a retail copy from MSDN). We recommend maintaining a separate, updated Windows VM based on this ISO to serve as the basis for the installation steps below. This way, you may apply Windows Updates and create new stemcells without having to reinstall all updates from scratch.

Disk Size to 128GB+ for 2012R2 stemcells

Currently, Windows 2012R2 based stemcells do not yet support ephemeral and persistent disks. Without this feature, Windows VMs on vSphere are created reflecting the size of the original virtual disk used to build the stemcell, regardless of what is specified in the deployment manifest.

To ensure you don't run into disk size problems especially on Cloud Foundry (as the VM disks cannot be resized on vSphere without this support), set your VM disk size to 128GB or greater.

Setup in vCenter:

  • Upload the Windows ISO to your datastore.
    • Click "Storage" in the front vCenter menu.
    • Choose a datastore and click on (or create) the directory where you want the ISO.
    • Click "Upload a file to datastore" (harddisk icon with green plus), and upload ISO.
      • You may need to install the vSphere client web plugin to upload through your browser, or scp the file directly to the datastore server. Please see VMware documentation for more help.
  • Create and customize a new virtual machine.
    • If you are using an existing template, select the creation type "Deploy from template" and select a template.
    • In "Select compatibility," ensure that you choose "ESXi 5.5 and later" if it's 2012R2.
    • For "Guest OS Family," select "Windows."
    • For "Guest OS version," select "Microsoft Windows Server 2012"
    • In "Customize hardware":
      • Under New Hard disk,
      • Under "New CD\DVD Drive":
        • Select "Datastore ISO File"
        • Expand the menu and select "Connect At Power On"
        • Click "Browse" and select the ISO you uploaded to your datastore
        • For "Virtual Device Node", select "IDE." XXX
    • For Windows Server 2012 R2: Use hardware version 10 (ESXi 5.5 and later)
  • Install Windows Server.
    • After creating VM, click "Power On" in the "Actions" tab for your VM.
      • For Windows Server 2012 R2: Select server with GUI
    • Select "Custom installation."
    • Follow along the installation process, and enter a password for the Administrator user. NOTE: Upon BOSH deployment, the Administrator password will be randomized.
  • Install VMware Tools.
    • In the vCenter web client, pick "Install VMware Tools" in the VM "Summary". tab.
    • For Windows Server 2012 R2: the installer should pop up automatically.
    • Restart the VM as required to finish the install.

Setup in VMware Fusion:

  • Create the virtual machine.
    • File => New
    • Select the Installation Method: Create a custom virtual machine
    • Choose Operating System: Microsoft Windows => Windows Server 2012
    • Choose a Virtual Disk: Create a new virtual disk.
  • Customize the VM.
    • Select "Customize Settings." Save the VM before continuing and pick a name for your reference.
    • A "Settings" window will pop up for your new VM
    • In CD/DVD:
      • Check the box 'Connect CD/DVD Drive'
      • From "This drive is configured to use the following:", select "Choose a disc or disc image" and select your base ISO.
    • In Hard Disk:
      • Click "Advanced options"
      • Uncheck the box "Split into multiple files."
      • Click "Apply"
      • In Camera:
        • Click "Remove Camera"
      • In Compatibility:
        • Click "Advanced options"
        • From "Use Hardware Version", select 10
        • Click "Apply"
  • Install Windows Server.
    • Start the VM.
    • For Windows2012R2: Select Server with GUI
    • Select "Custom installation."
    • Follow along the installation process, and enter a password for the Administrator user. NOTE: Upon BOSH deployment, the Administrator password will be randomized.
  • Install VMware Tools.
    • Install VMware Tools by navigating in the menu bar to "Virtual Machine" => "Install VMWare Tools"
    • For Windows Server 2012 R2: the installer should pop up automatically.
    • Restart the VM as required to finish the install.

Setup in VMware Workstation (requires v12 Pro or later):

  • Create the virtual machine.
    • Custom Advanced
    • For Windows Server 2012 R2: Select Worksation 9.x Compatibility
    • Select "Use ISO image" and browse for the correct ISO.
  • Customize the VM.
    • For Windows Server 2012 R2: Select the Windows version 2012
    • Choose a name for the image.
    • Select the BIOS Firmware Type
    • Adjust the number of cores and amount of memory for this session. (This will not affect the BOSH-deployed VMs.)
    • Select NAT network type.
    • Select LSI logic SAS controller type.
    • Select SCSI disk type.
    • Create a new virtual disk.
    • Select "Store virtual disk as a single file."
  • Install Windows Server.
    • Start the VM.
    • For Windows Server 2012 R2: Select server with GUI
    • Select "Custom installation."
    • Follow along the installation process, and enter a password for the Administrator user. NOTE: Upon BOSH deployment, the Administrator password will be randomized.
  • Install VMware Tools.
    • After the VM has started successfully, right-click the machine name in Workstation and click Install VMware Tools.
    • For Windows Server 2012 R2: the installer should pop up automatically.
    • Restart the VM as required to finish the install.

Step 2: Install Windows Updates

For 2012R2, installing Windows Updates could take a long time (8+ hours)

Note for "Meltdown" Security Patch

Until Windows 2012 R2 update channels contain a Meltdown patch, after applying all available updates you will need to manually download and apply patch "2018-01 Security Only Quality Update for Windows Server 2012 R2 for x64-based Systems (KB4056898)" Microsoft, which can be downloaded here.

NOTE

  • See Microsoft's Known Issues pertaining to KB4056898 prior to installing the patch.
  • Please follow Microsoft's instructions for enabling and testing the protections provided by the patch here.

.msu patchfiles can be applied by running from the command line, e.g.,

wusa.exe C:\patch-file.msu

This patch also requires a reboot to complete.

For both 2012R2, make sure the following registry keys are set to enable meltdown mitigation:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management" /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization" /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d "1.0" /f

REG ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat /f /v cadca5fe-87d3-4b96-b7fb-a231484277cc /t REG_DWORD /d 0

Step 3: Clone the VM

We recommend to clone the VM that has all the Windows updates installed. Please save the original VM so that you can run updates on it in the future.

For vSphere

  • In vCenter, you can right+click the current Windows VM
  • Select a Clone to Virtual Machine
  • Post that you can create the Virtual Machine that can be used to create a stemcell for the next Patch Tuesday Monthly Updates.

Do not create a snapshot of the VM, as this will break the stemcell created by stembuild later.

Step 4: Install CF binaries and other required software

4.1 Download and Install the BOSH PS Modules

Download latest BOSH PS Modules and BOSH Agent for your target stemcell versions (1709.* or 1200.*)

For building the BOSH PS Modules from source, see this guide.

  • Transfer the bosh-psmodules.zip to your Windows VM
  • If you are running Workstation or Fusion, you can just drag and drop files into the VM window if you have installed VMware tools.
  • Start "Powershell" in the Windows VM and run Unblock-File <path-to-bosh-psmodules.zip> where <path-to-bosh-psmodules.zip> is the full path to the location you put the bosh-psmodules.zip file in your Windows VM.
  • Unzip the archive and copy the BOSH.* folders to C:\Program Files\WindowsPowerShell\Modules
  • For Windows Server version 1709/1803: you can use Expand-Archive in PowerShell

4.2 Note For Remaining Steps

Depending on the ISO you use, you may need to run the PowerShell commands in this guide by specifying an explicit execution policy, like this:

powershell -ExecutionPolicy Bypass -Command <cmdlet>

where <cmdlet> is the command you need to run. For example, powershell -ExecutionPolicy Bypass -Command Install-CFFeatures.

**Note: For 2012R2, please ensure you have upgraded your Powershell version to 5 using the command Upgrade-PSVersion**

4.3 Install Cloud Foundry Diego Cell requirements

This step installs the Windows features necessary for the most common operation on Cloud Foundry (e.g. .NET Framework), as well as disabling services by default that could present security risks.

  • On your Windows VM, start PowerShell, and run Install-CFFeatures. The machine will restart automatically.
  • Apply the recommended ingress and service configuration by running Protect-CFCell.

4.4 Download and Install the BOSH Agent

The BOSH Agent is the core component used to communicate with the BOSH director to manage the VM. It is managed by a Windows Service.

  • Transfer agent.zip to your Windows VM.
  • On your Windows VM, start PowerShell:
    • Run Unblock-File <path-to-agent.zip> where <path-to-agent.zip> is the full path to the location you put the agent.zip file in your Windows VM.
    • Run Install-Agent -IaaS vsphere -agentZipPath <path-to-agent.zip> where <path-to-agent.zip> is the full path to the location you put the agent.zip file in your Windows VM.

For building the BOSH Agent from source, see this guide.

4.5 Install OpenSSH

The bosh ssh command can be used on BOSH-deployed Windows VMs if the OpenSSH dependency is installed on the Windows VM and later enabled during deploy time. This enables an operator to enter into a CMD or PowerShell session on the VM as a user with administrator privileges.

  • Transfer OpenSSH-Win64.zip to the Windows VM to C:\provision\OpenSSH-Win64.zip
  • Run Unblock-File 'C:\provision\OpenSSH-Win64.zip'.
  • Run the following PowerShell command Install-SSHD -SSHZipFile 'C:\provision\OpenSSH-Win64.zip'

In order to actually use the bosh ssh command, there are additional steps you'll need to perform at deployment time. For details, please see this GitHub repo.

4.6 Optimize and Compress the Disk

Windows Server stemcells can often be quite large, sometimes greater than the 10GB upload limit imposed by default in a BOSH director (they're also rather unwieldy to manage).

In order to reduce the stemcell size so you don't have to change this limit (through the nginx config on the director), you can do the following:

  • Restart the VM.
  • In Powershell, run:
    • Optimize-Disk - Runs dism to clear unnecessary files.
    • Compress-Disk - Defragments and "zeroes out" the disk.

PowerShell processes running in a command window can be inadvertently paused by left clicking on the PowerShell window. This can be verified by observing the prompt to see if it is in the select mode. You can read more about it here. If the PowerShell process appears to have stalled, please verify that the process hasn't inadvertently been paused.

Step 5: Sysprep and Applying Security Policies

NOTE Security Policies currently only supported for Windows Server 2012 R2

This step "syspreps" the system, which ensures each BOSH VM has a unique identity and applies the appropriate startup configuration at boot time.

The included policies help ensure the up-time and the secure operations of the stemcell's VMs, especially when deployed on Cloud Foundry.

One notable behavior this sets is to disable services that could cause restarts, like Windows Automatic Updates. OS restarts are not supported on BOSH-deployed Windows VMs, and the BOSH director will "resurrect" (i.e. destroy and repave) the VM.

  • Copy lgpo.exe to the Windows VM at location C:\Windows\lgpo.exe
  • Run the following command in PowerShell:
Invoke-Sysprep -IaaS vsphere -NewPassword <NEW_PASSWORD> -ProductKey <PRODUCT_KEY> -Owner <OWNER> -Organization <ORGANIZATION>
  • This will power off the VM.
  • Do not turn the VM back on before exporting in Step 6 below.

Notes about Invoke-Sysprep

Arguments: If you are using a volume licensed copy of Windows, you do not need to provide -ProductKey <PRODUCT_KEY>, as activation typically occurs via a KMS server. The OWNER and ORGANIZATION flags are always optional. Set them if your organization requires it.

Known issue regarding passwords: Do not use any special character in the password other than "!". For example, "Example12!" is fine, but "Example#12" is not. This is a known issue that will be fixed in future versions.

PowerShell tip: If you are invoking this sysprep command by specifying an execution policy, i.e.

powershell -ExecutionPolicy Bypass -Command Invoke-Sysprep...

make sure to wrap the entire sysprep invocation in double quotes, for example:

powershell -ExecutionPolicy Bypass -Command "Invoke-Sysprep -IaaS vsphere -NewPassword Foobar123! -Owner Bob -Organization 'Bob Company Org'"

Regarding Security Policies

While it is not recommended, it is possible to sysprep the image without applying the included security policies. As of version 1200.6, the included policies have been hardened against the CIS MS-L1 security benchmark.

Skipping these policies is typically acceptable for development setups or testing scenarios that don't require such security, or if you are using your own local policies.

To do this, use the -SkipLGPO flag when invoking the Invoke-Sysprep command. For example:

Invoke-Sysprep -IaaS vsphere -NewPassword <NEW_PASSWORD> -SkipLGPO

The security policies that ship with the stemcell packages are tested with Cloud Foundry. Using your own policies, while possible, may interfere with the operation of Cloud Foundry components.

Step 6: Export the VM's VMDK file

Find the .VMDK file associated with the VM you just powered off in Step 5.

vCenter

  • Right-click the VM and select "Template -> Export to OVF Template".
  • Download the OVA to your workstation. (You do not need to include files in the floppy or CD Drive.) You can also download the standalone vSphere client and click "File -> Export -> Export OVF Template".
  • Rename the OVA you download to have a .tar extension. The VMDK file will be in this tar archive.

Fusion and Workstation

  • The default directory for VM disk images is in Documents in your home directory. For example:
    • macOS: ~/Documents/Virtual Machines/vm-name.vmwarevm/Virtual Disk.vmdk
    • Windows: C:\Users\Administrator\Documents\Virtual Machines\vm-name.vmwarevm\Virtual Disk.vmdk
    • Linux: ~/vmware/vm-name/vm-name.vmdk
  • Locate the VMDK for the VM you created and take note of its location.

Step 7: Convert the VMDK to a BOSH Stemcell

This final step typically takes about 10-20 minutes to complete.

Note We are introducing improvements to stembuild, and have created a new command for converting a VMDK into a Windows Stemcell. Refer to the Usage document of stembuild.

  • Download the latest release of the stembuild utility.
  • Place the stembuild executable in your path.
  • Make note of the path to the VMDK from step 6 and the version of the stemcell you're building.
  • Build the stemcell with the following command:
stembuild package -vmdk <path-to-vmdk> -stemcell-version <stemcell stemcellVersion> -os <os stemcellVersion>```

Regarding the stemcellVersion, make sure to specify the version that maps to the stemcell release version.

For example

  • If you downloaded bits from the 1200.7 release, then specify that in the option.

stembuild will create the stemcell in the directory you execute it in.

stembuild requires ovftool to be in your system's PATH. It invokes ovftool to convert the disk image to the appropriate stemcell format and apply the proper configuration.

The stemcell is ready for use in conjunction with your bosh deployment.

Step 8: Monthly Patch Tuesday Updates

On Patch Tuesday, run Windows Updates on the base image, then repeat steps 3 through 7.

Troubleshooting

Garden-Windows logs suggest Windows features not installed

In case you see the below error in your garden-windows job while deploying Windows 2012R2, 1709 or 1803,

Missing required Windows Features: Web-Webserver, Web-WebSockets, AS-Web-Support, AS-NET-Framework, Web-WHC, Web-ASP.  Please use the most recent stemcell.

Please run the below steps on Powershell on your Windows VM to verify whether Install-CFFeatures ran successfully or not.

  • For Windows 2012R2
Get-WindowsFeature "Web-Webserver" | Where InstallState -Eq "Installed"
Get-WindowsFeature "Web-WebSockets"  | Where InstallState -Eq "Installed"
Get-WindowsFeature "AS-Web-Support"  | Where InstallState -Eq "Installed"
Get-WindowsFeature "AS-NET-Framework"  | Where InstallState -Eq "Installed"
Get-WindowsFeature "Web-WHC"  | Where InstallState -Eq "Installed"
Get-WindowsFeature "Web-ASP"  | Where InstallState -Eq "Installed"
Clone this wiki locally