Automate a multi machine ArcGIS Enterprise deployment on Windows using Chef Server

Scott M. MacDonald edited this page Oct 15, 2018 · 6 revisions

This workflow describes how to deploy ArcGIS Enterprise 10.6 on a set of Windows machines using Chef Server.

To follow this workflow for Linux machines, see Automate a multi machine ArcGIS Enterprise deployment on Linux using Chef Server.

Note: The main goal of this workflow is to demonstrate the deployment process. Chef cookbook can configure any ArcGIS Enterprise deployment pattern. Please refer to the ArcGIS Enterprise documentation to pick the ideal deployment architecture for your organization.

Before beginning this workflow, we have:

  • Five Windows machines in the same domain with access to the internet.
  • Setups for ArcGIS Enterprise 10.6. Note the name of your file. For setup files named "Setup.exe," you should define them in the JSON configuration files using the setups variable; setup files named in a format such as "ArcGIS_Server_1061_163983.exe" are self-extracting and should be defined using the setups_archive variable.
  • Software authorization files for ArcGIS Enterprise 10.6 (Portal for ArcGIS and ArcGIS GIS Server) and for ArcGIS GeoAnalytics Server 10.6.
  • A SSL certificate file for the domain name of the base ArcGIS Enterprise machine.
  • Starter Kit for Chef Server. See the options below under Prepare Chef Server.

Below is the intended mapping for our five machines and deployment roles.

  1. Machine 1 -> Chef workstation
  2. Machine 2 -> Base ArcGIS Enterprise (baseenterprise)
  3. Machine 3 -> Big Data Store (bds)
  4. Machine 4 -> File server for GeoAnalytics (fileserver)
  5. Machine 5 -> GeoAnalytics Server (gaserver)

Prerequisites

Machines

  1. Make sure WinRM service is running and already set up for remote management on all machines participating in this deployment. By default, WinRM service should be running on Windows machines. You can verify this on individual machines by running the command below in your command prompt:

    winrm quickconfig -q
    

If WinRM service is not configured on any machine, the above command will configure it.

  1. Make sure you have one Windows user (local or domain) with administrator privileges on each machine. This user is used in subsequent PowerShell scripts for deployment purposes.

Chef Server

Most organizations are using Chef Server to automate deployment. If you don’t have a Chef Server, then there are multiple options available to set it up. You can download the setup from the Chef website and install it on your machine. Alternatively, you can use Hosted Chef Server provided by Chef (SaaS) or Chef Automate. If you have an AWS account, then you can use AWS opsworks for Chef to set up Chef Server.

For this workflow, we are using Hosted Chef Server.

  1. Sign up for a free trial of Hosted Chef if you don't already have an account.
  2. Sign in to the Chef Server management console, create an organization and download the starter kit for the organization (Administration > Organizations > Starter Kit).

Chef workstation

If you don’t have a Chef workstation set up, the following steps can be used to set it up as a reference.

1. Log in to the Chef workstation machine (Machine 1) with Remote Desktop.
2. On the Chef workstation machine, install ChefDK.
3. Unzip the 'chef-repo' directory from your Chef Server starter kit into your C:\ drive.
  • On the Chef Workstation machine, unzip the starter kit starter-kit.zip
  • After unzipping, the path should be C:\chef-repo.
4. Configure your SSH key:
  • Copy your SSH key and paste it inside folder C:\chef-repo\.chef.
6. Start the ChefDK PowerShell console.
Click Windows button and start 'Chef Development Kit'.   
7. In the PowerShell console, change the current directory to C:\chef-repo.
8. Install the knife-windows plugin using command below:
chef gem install knife-windows
9. Retrieve your server certificate and verify your workstation can communicate with the Chef server:
cmd.exe /c "knife ssl fetch"
cmd.exe /c "knife ssl check"
cmd.exe /c "knife client list"

Prepare Chef Server for ArcGIS Enterprise Deployment

1. Copy cookbooks onto your Chef workstation machine.

Download the arcgis-3.2.0-cookbooks.zip and arcgis-enterprise-windows-roles.zip. Unzip the contents of the arcgis-3.2.0-cookbooks.zip file into the C:\chef-repo\cookbooks directory on your Chef workstation machine.

2. Upload cookbooks to the Chef server. Run the command below in the PowerShell console.
cmd.exe /c "knife cookbook upload --all"
3. Copy the 'roles' and 'environments' folders on your Chef workstation machine.

Unzip your arcgis-enterprise-windows-roles.zip file and move the two folders into C:\chef-repo\roles and C:\chef-repo\environments folders respectively.

4. Modify the C:\chef-repo\environments\demo.json file.

Open the file in a text editor like Notepad++. The file has a number of sample values meant to be replaced. Substitute values for these nodes specific to your deployment, like REPOMACHINE name, file paths, user names, and passwords.

5. Upload the 'demo' environment to the Chef server.
cmd.exe /c "knife upload environments\demo.json"

Install Base ArcGIS Enterprise (baseenterprise node)

1. Bootstrap the base ArcGIS Enterprise node (Machine 2).

A bootstrap is a process that installs the chef-client on a target system so that it can run as a chef-client and communicate with a Chef server. Replace username and password in the command below with those specific to your deployment. Replace HOSTNAME2 with system/host name of your Machine 2.

cmd.exe /c "knife bootstrap windows winrm HOSTNAME2 --winrm-user 'username' --winrm-password 'password' --node-name baseenterprise --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret"

username refers to the local or domain Windows administrator user available on Machine 2. password refers to the password of your Windows administrator user.

2. Modify the C:\chef-repo\roles\baseenterprise.json file on your Chef workstation machine.

Update values for nodes like REPOMACHINE, yourdomainname, file paths, user names, and passwords, and upload it to the Chef server.

cmd.exe /c "knife upload roles\baseenterprise.json"
3. Set the role of 'baseenterprise' node to 'baseenterprise.'
cmd.exe /c "knife node run_list set baseenterprise 'role[baseenterprise]'"
4. Set the environment of 'baseenterprise' node to 'demo.'
cmd.exe /c "knife node environment set baseenterprise demo"
5. Run chef-client on nodes in the 'baseenterprise' role. Replace username and password as mentioned in step 1.
cmd.exe /c "knife winrm 'role:baseenterprise' 'chef-client' --winrm-shell elevated -x 'username' -P 'password'"

The Chef run takes about 80 minutes. After the Chef run finishes, you have Portal, Server and Data Store installed on Machine 2 with the appropriate configuration, as depicted in the above architecture diagram. You can access your ArcGIS Enterprise portal using the URL format https://<yourdomainname>/portal/home.

Install Spatiotemporal Big Data Store (bds node)

1. Bootstrap the big data store node (Machine 3).

Replace username and password in the command below with those specific to your deployment. Replace HOSTNAME3 with system/host name of Machine 3.

cmd.exe /c "knife bootstrap windows winrm HOSTNAME3 --winrm-user 'username' --winrm-password 'password' --node-name bds --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret"

username refers to the local or domain Windows administrator user available on Machine 3. password refers to the password of your Windows administrator user.

2. Modify node values in the file 'C:\chef-repo\roles\bds.json' available on your Chef workstation machine with values specific to your deployment, and upload it to the Chef server.
cmd.exe /c "knife upload roles\bds.json"
3. Set the role of the big data store node to 'bds.'
cmd.exe /c "knife node run_list set bds 'role[bds]'"
4. Set the environment of the big data store node to 'demo.'
cmd.exe /c "knife node environment set bds demo"
5. Run chef client on nodes in the 'bds' role. Replace username and password as mentioned in step 1.
cmd.exe /c "knife winrm 'role:bds' 'chef-client' --winrm-shell elevated -x 'username' -P 'password'"

After the Chef run finishes, a spatiotemporal big data store is installed on Machine 3 and registered with the ArcGIS Server installed on Machine 2, as depicted in the architecture diagram.

Create File Server for GeoAnalytics (fileserver node)

Most organizations probably already have a file server in place. If you do, then you can use your existing file server by providing its file path to the GeoAnalytics Server you set up. Refer to the Chef parameter list on how to define it. The following steps are just an example to configure a file server for you if you don’t already have one.

1. Bootstrap the File Server node (Machine 4).

Replace username and password in command below with those specific to your deployment. Replace HOSTNAME4 with system/host name of Machine 4.

cmd.exe /c "knife bootstrap windows winrm HOSTNAME4 --winrm-user 'username' --winrm-password 'password' --node-name fileserver --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret"

username refers to the local or domain Windows administrator user available on Machine 4. password refers to the password of your Windows administrator user.

2. Modify node values in 'C:\chef-repo\roles\fileserver.json' file on your Chef workstation machine with values specific to your deployment and upload them to the Chef server.
 cmd.exe /c "knife upload roles\fileserver.json"
3. Set the role of the file server node to 'fileserver.'
cmd.exe /c "knife node run_list set fileserver 'role[fileserver]"
4. Set the environment of the file server node to 'demo.'
cmd.exe /c "knife node environment set fileserver demo"
5. Run chef-client on nodes in the 'fileserver' role. Replace username and password as mentioned in step 1.
cmd.exe /c "knife winrm 'fileserver' 'chef-client' --winrm-shell elevated -x 'username' -P 'password'"

After the Chef run finishes, a file server is configured on Machine 4, as depicted in the architecture diagram.

Install GeoAnalytics Server (gaserver node)

1. Bootstrap the primary GeoAnalytics Server node (Machine 5).

Replace username and password in command below with those specific to your deployment. Replace HOSTNAME5 with system/host name of Machine 5.

cmd.exe /c "knife bootstrap windows winrm HOSTNAME5 --winrm-user 'username' --winrm-password 'password' --node-name gaserver --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret"

username refers to the local or domain Windows administrator user available on Machine 4. password refers to the password of your Windows administrator user.

2. Modify node values in 'C:\chef-repo\roles\gaserver.json' file on your Chef workstation machine with values specific to your deployment and upload them to the Chef server.
 cmd.exe /c "knife upload roles\gaserver.json"
3. Set the roles of the primary GeoAnalytics Server node to 'gaserver'.
cmd.exe /c "knife node run_list set gaserver 'role[gaserver]'"
4. Set the environment of the primary GeoAnalytics Server node to 'demo'.
cmd.exe /c "knife node environment set gaserver demo"
5. Run chef-client on nodes in the 'gaserver' role. Replace username and password as mentioned in step 1.
cmd.exe /c "knife winrm 'gaserver' 'chef-client' --winrm-shell elevated -x 'username' -P 'password'"

After the Chef run finishes, ArcGIS Server is installed on Machine 5, as depicted in the architecture diagram.

You can verify deployments by accessing URLs https://yourdomainname/portal and https://yourdomainname/server.

How to uninstall ArcGIS Enterprise software and delete content directories

1. Modify 'C:\chef-repo\roles\cleanarcgisenterprise.json' file on your Chef workstation,and upload it to your Chef server.
cmd.exe /c "knife upload roles\cleanarcgisenterprise.json"
2. Set the role of all above created nodes to 'cleanarcgisenterprise'.
cmd.exe /c "knife node run_list set baseenterprise 'role[cleanarcgisenterprise]'"
cmd.exe /c "knife node run_list set fileserver 'role[cleanarcgisenterprise]'"
cmd.exe /c "knife node run_list set gaserver 'role[cleanarcgisenterprise]'"
cmd.exe /c "knife node run_list set bds 'role[cleanarcgisenterprise]'"

3. Run chef-client on the nodes identified with the 'cleanarcgisenterprise' role.
cmd.exe /c "knife winrm 'role:cleanarcgisenterprise' 'chef-client' --winrm-shell elevated -x 'username' -P 'password'"

The Chef run takes about 30 minutes. After the Chef run finishes, ArcGIS Server, Portal for ArcGIS, ArcGIS Data Store, and ArcGIS Web Adaptor for IIS will be uninstalled from respective nodes. It also cleans up content directories like arcgisserver, arcgisportal and arcgisdatastore.

Production Deployment workflow

The above workflow was designed to make the deployment process easier to understand and recover from mistakes. In production deployments, instead of deploying one machine at a time, it is recommended to split and script the process into three phases:

1. Define all the environments and roles;
2. Provision all the nodes (machines) and assign roles and environments to the nodes;
3. Run chef-client on all the roles in a specific order.

A sample PowerShell script would be:

knife bootstrap windows winrm HOSTNAME2 --winrm-user 'AVWORLD\USER' --winrm-password 'password' --node-name baseenterprise --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret
knife bootstrap windows winrm HOSTNAME3 --winrm-user 'AVWORLD\USER' --winrm-password 'password' --node-name bds --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret
knife bootstrap windows winrm HOSTNAME4 --winrm-user 'AVWORLD\USER' --winrm-password 'password' --node-name fileserver --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret
knife bootstrap windows winrm HOSTNAME5 --winrm-user 'AVWORLD\USER' --winrm-password 'password' --node-name gaserver  --secret-file C:/chef-repo/.chef/encrypted_data_bag_secret

knife upload roles\baseenterprise.json
knife upload roles\bds.json
knife upload roles\fileserver.json
knife upload roles\gaserver.json

knife node run_list set baseenterprise 'role[baseenterprise]'
knife node run_list set bds 'role[bds]'
knife node run_list set fileserver 'role[fileserver]'
knife node run_list set gaserver 'role[gaserver]'

knife node environment set baseenterprise demo
knife node environment set bds demo
knife node environment set fileserver demo
knife node environment set gaserver demo

knife winrm 'role:baseenterprise' 'chef-client' --winrm-shell elevated -x 'AVWORLD\USER' -P 'password'
knife winrm 'role:bds' 'chef-client' --winrm-shell elevated -x 'AVWORLD\USER' -P 'password'
knife winrm 'role:fileserver' 'chef-client' --winrm-shell elevated -x 'AVWORLD\USER' -P 'password'
knife winrm 'role:gaserver' 'chef-client' --winrm-shell elevated -x 'AVWORLD\USER' -P 'password'
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.