Deploy ArcGIS Data Pipelines Server

Deploying and Configuring ArcGIS DataPipelines Server with PowerShell DSC ArcGIS Module

Overview

This guide provides step-by-step instructions for deploying and configuring ArcGIS DataPipelines Server using the PowerShell DSC ArcGIS Module. The approach leverages Desired State Configuration (DSC) to automate installation, licensing, and integration with ArcGIS Enterprise environments, ensuring repeatable and reliable deployments.

Prerequisites

Before you begin, ensure the following prerequisites are met:

• Supported Operating System: Microsoft Windows Server (see Esri documentation for supported versions).
• ArcGIS Enterprise Software: DataPipelines Server, Web Adaptor installers (and patches, if required).
• PowerShell DSC: Windows PowerShell 5.1 or later with DSC enabled.
• PowerShell DSC ArcGIS Module: This repository/module, imported on the deployment host.
• Service Account: A Windows account (domain or local) with permissions to install and run ArcGIS components.
• Licenses: Valid DataPipelines Server license file.
• Configuration File: A JSON configuration file describing your deployment (see below for structure).
• Network & Firewall: Required ports open between all ArcGIS components (see Esri documentation).

Tip: Place all installers and configuration files on accessible local or network paths. Use secure methods for handling credentials and license files.

Best Practices and Summary

• Use Secure Credential Handling: Prefer environment variables or encrypted files for passwords.
• Version Control Configurations: Store your JSON configuration files in version control for traceability.
• Test in Staging: Validate deployments in a non-production environment before rolling out to production.
• Monitor Logs: Regularly review deployment and ArcGIS logs for early detection of issues.
• Stay Updated: Keep ArcGIS software and the DSC module up to date with the latest patches and releases.
  • Upgrade PowerShell DSC Module for ArcGIS
  • Download and Install patches for ArcGIS Enterprise

---

By following this guide, you can automate and standardize the deployment of ArcGIS DataPipelines Server using the PowerShell DSC ArcGIS Module, ensuring consistency and reducing manual effort. For advanced scenarios or troubleshooting, refer to the module documentation and Esri’s official resources.

Deployment Steps

Follow these steps to deploy and configure ArcGIS DataPipelines Server:

1. Prepare the Environment

• Ensure all prerequisites are met (see above).
• Place all required installers, license files, and your JSON configuration on the deployment host.

2. Install the PowerShell DSC ArcGIS Module

You can install the module using one of the following methods:

• From GitHub Artifacts:
  Follow the instructions in the Getting Started guide (Option 1) to download and import the module directly from the GitHub repository.

• From PowerShell Gallery:
  Refer to Getting Started guide (Option 2) for steps to install the module from the PowerShell Gallery.

Note: Ensure the module resides on each target node and orchestrating node.

3. Run the Deployment Command

Use the Invoke-ArcGISConfiguration command to start the deployment. Example:

$configFile = "C:\Path\To\GISServer-DataPipelines.json"
$cred = Get-Credential  # Enter the service account credentials
Invoke-ArcGISConfiguration -ConfigurationParametersFile $configFile -Mode InstallLicenseConfigure -Credential $cred -DebugSwitch

Parameters:

• -ConfigurationParametersFile: Path to your JSON configuration file.
• -Mode: Use InstallLicenseConfigure for initial deployment.
• -Credential: Service account credentials (used for remote execution).
• -DebugSwitch: Enables verbose logging for troubleshooting.

Tip: For comprehensive details on available command-line parameters, refer to the PowerShell DSC ArcGIS Module Command-Line Parameters documentation.

4. Monitor Progress

Verbose output will be shown in the console. Detailed logs are written to the Logs/<node>/ directory for each node and configuration phase.

5. Post-Deployment

• Validate the deployment (see next section).
• If required, complete any additional configuration steps as described in your JSON file or Esri documentation.

Configuration File Structure

Deployment is driven by a JSON configuration file. Below is a summary of the key sections and fields, based on the provided sample:

{
	"AllNodes": [
		{
			"NodeName": "[Machine Name]",
			"Role": ["Server", "WebAdaptor"],
			"WebAdaptorConfig": [
				{ "Role": "Server" }
			]
		}
	],
	"ConfigData": {
		"Version": "[ArcGIS Version]",
		"ServerRole": "DataPipelinesServer",
		"ServerContext": "datapipelines",
		"Credentials": { ... },
		"Server": {
			"LicenseFilePath": "[License File Path]",
			"Installer": {
				"Path": "[Server Installer Path]",
				"InstallDir": "[Server Install Dir]"
			},
			"ServerDirectoriesRootLocation": "C:\\datapipelinesserver\\directories",
			"ConfigStoreLocation": "C:\\datapipelinesserver\\config-store",
			"ServerDirectories": [ ... ],
			"ServerLogsLocation": "C:\\datapipelinesserver\\logs",
			"PrimarySiteAdmin": { ... }
		},
		"WebAdaptor": { ... },
		"Federation": { ... }
	}
}

Key Fields:

• AllNodes: List of target machines and their roles.
• ConfigData.Version: ArcGIS Enterprise version.
• ConfigData.ServerRole: Must be DataPipelinesServer for this deployment.
• ConfigData.ServerContext: Should be set to datapipelines.
• ConfigData.Credentials: Service account and admin credentials.
• ConfigData.Server: DataPipelines Server install, license, directories, and admin details.
• ConfigData.WebAdaptor: Web Adaptor installer and settings.
• ConfigData.Federation: Portal federation details (host, port, context, admin credentials).

Note: See SampleConfigs/v5/v5.1.0/Gis Servers/GISServer-DataPipelines.json for a complete example.

Validation and Troubleshooting

Validating the Deployment

After deployment, verify the following:

• DataPipelines Server Service: Confirm the Windows service is running.
• ArcGIS Server Site: Access the ArcGIS Server Manager (typically https://<server>:6443/arcgis/manager) and verify the DataPipelines capability is present.
• Portal Federation: If federated, check the Portal for ArcGIS to ensure the DataPipelines utility service is registered and available.
• Web Adaptor: Confirm the Web Adaptor endpoint is accessible and properly routes requests.
• Logs: Review logs in the Logs/<node>/ directory for errors or warnings.

Troubleshooting Tips

• Deployment Fails:
  • Check PowerShell output and verbose logs for error messages.
  • Ensure all installer paths and credentials are correct in the JSON configuration.
  • Verify required ports are open and firewall rules allow communication between components.
• DataPipelines Not Available:
  • Confirm the ServerRole is set to DataPipelinesServer and ServerContext is set to datapipelines.
• Federation Issues:
  • Double-check Portal host, port, and admin credentials in the configuration.
  • Review Portal logs for federation errors.
• Web Adaptor Problems:
  • Ensure the Web Adaptor installer path is correct and the IIS site is running.

For further assistance, consult the ArcGIS Enterprise documentation and the module's internal logs.