Skip to content
A guide and samples for working with the ArcGIS Insights scripting environment.
Jupyter Notebook Python Dockerfile R
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Cert Initial commit Apr 10, 2019
Docker Update docker file comments Oct 24, 2019
Graphics Initial commit Apr 10, 2019
Py Update README.md Jul 29, 2019
R
.gitignore
LICENSE Initial commit Apr 10, 2019
README.md Doc edit recommend python 3.7, given geopandas Nov 18, 2019

README.md

ArcGIS Insights scripting guide

Welcome to the ArcGIS Insights scripting guide.

This repo contains information on getting started with scripting in ArcGIS Insights. This feature enables the execution of Python and R code using a bring-your-own scripting environment. Among many topics, we outline how to deploy a Jupyter Kernel Gateway, how to connect to a Jupyter Kernel Gateway, and tips and tricks for using the ArcGIS Insights scripting environment with other ArcGIS Insights features. In addition, the repo serves as a place to find and share useful Python and R scripts and creates a community around those who prefer to write code to advance data science and knowledge sharing.

We welcome any contributions for improving this guide. If you find a bug or would like to report a problem in the ArcGIS Insights scripting environment, please log an issue in this repo.

Overview

  • Learn how to configure a Jupyter Kernel Gateway (this is a requirement for using the ArcGIS Insights scripting environment).
  • Pick a Windows or macOS machine, such as a desktop or laptop, to host your Jupyter Kernel Gateway.
  • Generate Transport Layer Security (TLS) or Secure Sockets Layer (SSL) certificates.
  • Follow the guide for installing and deploying a Jupyter Kernel Gateway.
  • Learn tips about keyboard shortcuts and important scripting options.
  • Search for Python and R code.
  • Contribute Python and R code.
  • Read ArcGIS Insights use cases and product documentation.
  • Try ArcGIS Insights in Enterprise.
  • Use the ArcGIS Insights scripting environment in Enterprise.

Required packages to export data from the scripting environment to the data pane

To export data from the ArcGIS Insights scripting environment to the ArcGIS Insights data pane, the following Python and R packages are used:

  • The Python Kernel uses pandas, geopandas, numpy, matplotlib, msgpack, shapely and requests.
  • The R Kernel uses jsonlite, data.table, and itertools.

Install and deploy a Jupyter Kernel Gateway

The ArcGIS Insights scripting environment uses a Jupyter Kernel Gateway. This section includes guidance on how to configure a Jupyter Kernel Gateway. It requires using either Docker or Anaconda and enabling TLS or SSL for security. TLS and SSL permit ArcGIS Insights to access your gateway over HTTPS and WSS.

Windows instructions

Create a TLS or SSL certificate

The instructions below will create a self-signed certificate. If you are able to create a trusted certificate in your enterprise, you can skip these steps and use the trusted enterprise certificate.

  1. Clone the insights-scripting-guide repo to your local machine.
  2. Use Anaconda to install openssl for creating a self-signed certificate.
  3. Open the Anaconda Command Prompt window.
  4. Create a folder called insightsgw on your local machine. For example, C:\insightsgw.
  5. Copy and paste the insightsgw.cnf file from this repo's Cert folder into the insightsgw folder and edit each placeholder value (host_name and domain) within the file.

Example:

[dn]
CN=insights.esri.com
[req]
distinguished_name=dn
[EXT]
subjectAltName=DNS:insights.esri.com
keyUsage=digitalSignature
extendedKeyUsage=serverAuth

When editing the insightsgw.cnf file, if you are unsure of your host_name run hostname from a command prompt to get the correct value. If you are unsure of your domain run set user from a command prompt to get the correct value.

  1. Using the Anaconda Command Prompt, change directories to insightsgw. For example, cd C:\insightsgw.
  2. Change the placeholders for <host_name> and <dns_name> and run the following command:

openssl req -x509 -days 365 -out <host_name>.crt -keyout <host_name>.key -newkey rsa:2048 -nodes -sha256 -subj "/CN=<host_name>.<dns_name>" -extensions EXT -config insightsgw.cnf

Example:

openssl req -x509 -days 365 -out insights.crt -keyout insights.key -newkey rsa:2048 -nodes -sha256 -subj "/CN=insights.esri.com" -extensions EXT -config insightsgw.cnf

  1. Close the Anaconda Command Prompt when the command has finished running. There should now be a <host_name>.key and a <host_name>.crt file in your insightsgw folder.
  2. Using administrative privileges, import the .crt file in your insightsgw folder into the Windows Certificate Store's Trusted Root Certification Authorities. Use the following steps:
    • Open Manage computer certificates.
    • Open Trusted Root Certification Authorities.
    • Right click Certificates.
    • Click All Tasks > Import.
    • Select the .crt file created using openssl.
    • Flush DNS using by running ipconfig /flushdns from the command prompt.
  3. Add new Inbound and Outbound rules for port 9999 using Windows Defender Firewall with Advanced Security.

Create a gateway

When creating a Jupyter Kernel Gateway choose either the Docker or Anaconda section below (not both).

Create a gateway using Docker

  1. Install Docker.
  2. Create a folder called insightsgw on your local machine for example, c:\insightsgw.
  3. Copy and paste Dockerfile from this repo's Docker folder into the insightsgw folder.
  4. Edit Dockerfile to fit your requirements.

The ENV KG_ALLOW_ORIGIN and ENV KG_LIST_KERNELS parameters are required for the Jupyter Kernel Gateway

If you want to load data into the container to work with inside the ArcGIS Insights console, create a local folder and add the data files to it and then set that folder to be copied from the local machine to the container in the Dockerfile. For example, create a folder in your local project folder insightsgw named data and add working data files to it. Then, add COPY /data/* /data/ to the Dockerfile to copy the local data folder with the files to the container. Adding WORKDIR /data to the Dockerfile will set the container working folder to the data folder.

  1. Run docker build -t insightsgw . from a command prompt in the folder created in step 2. For example, c:\insightsgw>docker build -t insightsgw .
  2. Run docker run -p 9999:9999 insightsgw from a command prompt in the folder created in step 2. For example, c:\insightsgw>docker run -p 9999:9999 insightsgw

If there is a database on the local host machine with data to be used in the ArcGIS Insights console, a connection to it can be created in the docker run command. For example, connecting to PostgreSQL in Windows docker run -p 9999:9999 -e DB_PORT=5432 -e DB_HOST=docker.for.win.host.internal insightsgw

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

  1. Leave the command prompt open while the Jupyter Kernel Gateway is running.
  2. To test that the gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

    You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

The default configuration of Docker is limiting so it is best practice to use Docker's Advanced Setting tab to configure additional CPU and memory resources.

In addition to configuring resources for the container from the Docker application, they can be configured within Dockerfile as well. See the Docker documentation for configuration details.

Create a gateway using Anaconda

  1. Install Anaconda.
  2. Open the Anaconda Command Prompt window.
  3. Create an ArcGIS Insights Python environment using the command conda create -n my_insights_env python=3.7.
  4. Activate the ArcGIS Insights Python environment using the command conda activate my_insights_env.
  5. Install Jupyter Kernel Gateway using the command conda install -c anaconda jupyter_kernel_gateway=2.1.0.
  6. Install pandas and numpy packages using the command conda install -c anaconda numpy pandas.
  7. Install geopandas and matplotlib package using the command conda install -c conda-forge geopandas matplotlib.
  8. Install msgpack package using the command conda install -c conda-forge msgpack-python.
  9. Install shapely package using the command conda install -c conda-forge shapely.
  10. Install requests package using the command conda install -c conda-forge requests.
  11. Add the ArcGIS API for Python using the command conda install -c esri arcgis.
  12. Install essential R packages using the command conda install -c r r-essentials.
  13. Add the itertools package using the command conda install -c conda-forge r-itertools.

Consider what directory to use when launching the Jupyter Kernel Gateway. If you have a C:\insightsgw\data directory which contains .csv files, launching the Jupyter Kernel Gateway from this folder will enable easy access to those files within scripts by allowing the use of relative paths for file and folder access.

The KernelGatewayApp.allow_origin and JupyterWebsocketPersonality.list_kernels parameters are required.

  1. Launch the Jupyter Kernel Gateway using the following command, changing the placeholders for <host_name>
jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=C:\insightsgw\<host_name>.crt --keyfile=C:\insightsgw\<host_name>.key 

Example:

jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=C:\insightsgw\insights.crt --keyfile=C:\insightsgw\insights.key 

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

  1. Leave the command prompt open while the Jupyter Kernel Gateway is running.

  2. To test that the Gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

    You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

If you are experiencing issues connecting to the console after creating a self-signed certificate, check the browser to make sure an exception does not need to be added for the site created in the certificate. In the example above, the site is https://insights.esri.com. Navigate to https://insights.esri.com in the browser. If an exception is required for the self-signed certificate, a warning page will be shown:

Click Advanced

Click Add Exception...

Click Confirm Security Exception

macOS instructions

Create a TLS or SSL cert

The instructions below will create a self-signed certificate. If you are able to create a trusted certificate in your enterprise, you can skip these steps and use the trusted enterprise certificate.

  1. Clone the insights-scripting-guide repo to your local machine.
  2. Create a directory called insightsgw on your local machine. For example, insightspc:insightsgw insightsuser$ .
  3. Copy and paste the insightsgw.cnf file from this repo's Cert directory into the insightsgw directory and edit each placeholder value (host_name and domain) within the file.

Example:

[dn]
CN=insights.esri.com
[req]
distinguished_name=dn
[EXT]
subjectAltName=DNS:insights.esri.com
keyUsage=digitalSignature
extendedKeyUsage=serverAuth

When editing the insightsgw.cnf file, if you are unsure of your host_name run sudo scutil --get LocalHostName from the terminal to get the correct value. If you are unsure of your domain run sudo scutil --get HostName from the terminal to get the correct value.

  1. Using the terminal, change directories to insightsgw. For example, insightspc:Documents insightsuser$ cd insightsgw.

    openssl is installed on macOS by default.

  1. Change the placeholders for <host_name> and <dns_name> and run the following command:

openssl req -x509 -days 365 -out <host_name>.crt -keyout <host_name>.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=<host_name>.<dns_name>' -extensions EXT -config ../insightsgw.cnf

Example:

openssl req -x509 -days 365 -out insights.crt -keyout insights.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=insights.esri.com' -extensions EXT -config ../insightsgw.cnf

    The terminal can be closed when finished.

    There should now be a <host_name>.key and a <host_name>.crt file in your insightsgw directory.

  1. Using administrative privileges, import the .crt file in your insightsgw directory into the Mac Keychain. Use the following steps:
    • Open Keychain
    • Choose System from Keychains
    • Choose Certificates from Category
    • Click File > Import Items
    • Choose .crt file created using openssl
    • Flush the DNS using sudo killall -HUP mDNSResponder from terminal

Create a gateway

When creating a Jupyter Kernel Gateway choose either the Docker or Anaconda section below (not both).

Create a gateway using Docker

  1. Install Docker.
  2. Create a directory called insightsgw on your local machine. For example, insightspc:insightsgw insightsuser$.
  3. Copy and paste Dockerfile from this repo's Docker directory into the insightsgw directory.
  4. Edit Dockerfile to fit your requirements.

The ENV KG_ALLOW_ORIGIN and ENV KG_LIST_KERNELS parameters are required for the Jupyter Kernel Gateway.

If you want to load data into the container to work with inside the ArcGIS Insights console, create a local directory and add the data files to it and then set that directory to be copied from the local machine to the container in Dockerfile. For example, create a directory in your local project directory insightsgw named data and add working data files to it. Then, add COPY /data/* /data/ to Dockerfile to copy the local data directory with the files to the container. Adding WORKDIR /data to Dockerfile will set the container working directory to the data directory.

  1. Run docker build -t insightsgw . from the terminal in the directory created in step 2. For example, insightspc:insightsgw insightsuser$docker build -t insightsgw .
  2. Run docker run -p 9999:9999 insightsgw from the terminal in the folder created in step 2. For example, insightspc:insightsgw insightsuser$docker run -p 9999:9999 insightsgw

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

    Leave the terminal open while the Jupyter Kernel Gateway is running.

  1. To test that the Gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.
        You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

The default configuration of Docker is limiting so it is highly recommended to use Docker's Advanced Setting tab to configure additional CPU and memory resources.

In addition to configuring resources for the container from the Docker application, they can be configured within the Dockerfile as well. See the Docker documentation for configuration details.

Create a gateway using Anaconda

  1. Install Anaconda.
  2. Open the terminal window.
  3. Create an ArcGIS Insights Python environment using the command conda create -n my_insights_env python=3.6.
  4. Activate the ArcGIS Insights Python environment using the command source activate my_insights_env.
  5. Install Jupyter Kernel Gateway using the command conda install -c anaconda jupyter_kernel_gateway.
  6. Install pandas and numpy packages using the command conda install -c anaconda numpy pandas.
  7. Install geopandas and matplotlib package using the command conda install -c conda-forge geopandas matplotlib.
  8. Install msgpack package using the command conda install -c conda-forge msgpack-python.
  9. Install shapely package using the command conda install -c conda-forge shapely.
  10. Install requests package using the command conda install -c conda-forge requests.
  11. Add the ArcGIS API for Python using the command conda install -c esri arcgis.
  12. Install essential R packages using the command conda install -c r r-essentials.
  13. Add the itertools package using the command conda install -c conda-forge r-itertools.

Consider what directory to use when launching the Jupyter Kernel Gateway. If you have a /Users/insightsuser/Documents/insightsgw/data directory which contains .csv files, launching the Jupyter Kernel Gateway from this directory will enable easy access to those files within scripts by allowing the use of relative paths for file and directory access.

The KernelGatewayApp.allow_origin and JupyterWebsocketPersonality.list_kernels parameters are required.

  1. Launch the Jupyter Kernel Gateway using the following command, changing the placeholders for <host_name>
jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=../certs/<host_name>.crt --keyfile=../certs/<host_name>.key

Example:

jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=../certs/insights.crt --keyfile=../certs/insights.key

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

    Leave the terminal open while the Jupyter Kernel Gateway is running.

  1. To test that the gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

  2. You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}

If you are experiencing issues connecting to the console after creating a self-signed certificate, check the browser to make sure an exception does not need to be added for the site created in the certificate. In the example above, the site is https://insights.esri.com. Navigate to https://insights.esri.com in the browser. If an exception is required for the self-signed certificate, a warning page will be shown:

Click Advanced

Click Add Exception...

Click Confirm Security Exception

Use the console in ArcGIS Insights

The ArcGIS Insights Console is available within workbooks only in the Enterprise version, not in Online. You can launch the console by clicking the Console button next to the Basemap button in the workbook toolbar.

Learn more about the scripting console in the ArcGIS Insights documentation.

Connecting your Jupyter Kernel Gateway

  1. To connect ArcGIS Insights to your Jupyter Kernel Gateway, click the Console button .

    The New Jupyter Kernel Gateway connection window opens.

  1. Enter the URL and web socket connection information.

    If using the examples in this guide, the URL parameter will be https://<host_name>.<dns_domain>:9999 and the Web socket parameter will be wss://<host_name>.<dns_domain>:9999.

Example:

  • URL

      https://insights.esri.com:9999

  • Web Socket

      wss://insights.esri.com:9999

Shortcuts and ArcGIS Insights magic commands

The ArcGIS Insights console uses keyboard shortcuts and magic commands so that routine tasks can be performed quickly and efficiently.

  • Use the %insights_return(<R data frame or Pandas DataFrame>) magic command to add an R data frame or Pandas DataFrame to the ArcGIS Insights data pane.

    The %insights_return magic command must be run on a single line in a single cell in the ArcGIS Insights Console

  • Ctrl/control + Alt/option + B Add %insights_return magic command to cell.
  • Ctrl/control + / Comment line.
  • Ctrl/control + Spacebar Enable IntelliSense.
  • Shift/shift + Enter/return Execute the code in the current cell.
  • Shift/shift + Up Arrow or Down Arrow Access the history of executed cells.

What is ArcGIS Insights?

Part of the Esri Geospatial Cloud, ArcGIS Insights is web-based data analytics made for advanced location intelligence. Answer questions you didn’t know to ask, analyze data completely, and tell powerful data stories. Connect to your data directly, then use maps, charts and tables to perform basic to complex analyses that scale based on skill level and business need.

Why Get Involved?

Send feedback to us concerning the ArcGIS Insights scripting console. Found an issue or have questions? Feel free to post questions or comments and report bugs to the issues section of this repo.

Start using ArcGIS Insights with a Free Trial

Sign-up to start a free trial.

Licensing

Copyright 2019 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

You can’t perform that action at this time.