Skip to content
A collection of scripts and tips to help analysts and data scientists work with the Scripting Console in Insights for ArcGIS.
Branch: master
Clone or download
Latest commit 9502761 Apr 10, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Cert Initial commit Apr 10, 2019
Docker Initial commit Apr 10, 2019
Graphics Initial commit Apr 10, 2019
Py Initial commit Apr 10, 2019
R Initial commit Apr 10, 2019
.gitignore Initial commit Apr 10, 2019
LICENSE Initial commit Apr 10, 2019
README.md Initial commit Apr 10, 2019

README.md

Insights scripting guide - Beta release

Welcome to the Insights scripting guide.

This repo contains information to get started with scripting in an Insights workbook. The Beta Release enables the execution of Python and R code using a bring-your-own scripting environment. Among many topics, we outline how to deploy a Jupyter Kernel Gateway, how to connect to a Jupyter Kernel Gateway, and tips and tricks for using the Insights console with other Insights features. In addition, the repo serves as a place to find and share useful Python and R scripts and creates a community around those who prefer to write code to advance data science and knowledge sharing.

We welcome any contributions for improving this guide. If you find a bug or would like to report a problem in the Beta release of the Insights Console, please log an issue in this repo.

Overview

  • Learn how to configure a Jupyter Kernel Gateway environment (this is a requirement for using the Insights console).
  • Pick a Windows or macOS machine, such as a desktop or laptop, to host your Jupyter Kernel Gateway.
  • Generate Transport Layer Security (TLS) or Secure Sockets Layer (SSL) certificates.
  • Follow the guide for installing and deploying a Jupyter Kernel Gateway.
  • Learn tips about keyboard shortcuts and important console tools.
  • Search for Python and R code.
  • Contribute Python and R code.
  • Read Insights use cases and product documentation.
  • Try Insights in Enterprise.
  • Use the Insights console in Enterprise.

Client data limitations

The following limitations exist for the Insights console. These limitations are temporary during the Beta release and will be removed once the Insights console is out of Beta. In the Beta Release, there are limitations on the size of data that can be exported from the Insights console to the Insights data pane. These limitations do not exist when bringing data from the Insights data pane into the Insights console. These limitations are temporary during the Beta Release and will be removed once the Insights Console is out of Beta.

  • Exporting data from the Insights Console to the Insights data pane.
    • Current limit of ~ 8 MB for data frame size.
    • ~ 27000 rows X 10 columns of mixed data types.

Required packages to export data from the console to the data pane

To export data from the Insights console to the Insights data pane, the following Python and R packages are used:

  • The Python Kernel uses pandas and NumPy.
  • The R Kernel uses jsonlite, data.table, and itertools.

Insights will install these packages if they are not available in the respective kernels.

Install and deploy a Jupyter Kernel Gateway

The Insights scripting console uses Jupyter Kernel Gateways. This section includes guidance on how to configure a Jupyter Kernel Gateway environment. It requires using either Docker or Anaconda and enabling TLS or SSL for security. TLS and SSL permit Insights to access your gateway over HTTPS and WSS.

Windows instructions

Create a TLS or SSL certificate

  1. Clone the insights-scripting-guide repo to your local machine.
  2. Use Anaconda to install openssl for creating a self-signed certificate.
  3. Open the Anaconda Command Prompt window.
  4. Create a folder called insightsgw on your local machine. For example, C:\insightsgw.
  5. Copy and paste the insightsgw.cnf file from this repo's Cert folder into the insightsgw folder and edit each placeholder value (host_name and domain) within the file.

Example:

[dn]
CN=insights.esri.com
[req]
distinguished_name=dn
[EXT]
subjectAltName=DNS:insights.esri.com
keyUsage=digitalSignature
extendedKeyUsage=serverAuth

When editing the insightsgw.cnf file, if you are unsure of your host_name run hostname from a command prompt to get the correct value. If you are unsure of your domain run set user from a command prompt to get the correct value.

  1. Using the Anaconda Command Prompt, change directories to insightsgw. For example, cd C:\insightsgw.
  2. Change the placeholders for <host_name> and <dns_name> and run the following command:

openssl req -x509 -days 365 -out <host_name>.crt -keyout <host_name>.key -newkey rsa:2048 -nodes -sha256 -subj "/CN=<host_name>.<dns_name>" -extensions EXT -config insightsgw.cnf

Example:

openssl req -x509 -days 365 -out insights.crt -keyout insights.key -newkey rsa:2048 -nodes -sha256 -subj "/CN=insights.esri.com" -extensions EXT -config insightsgw.cnf

  1. Close the Anaconda Command Prompt when the command has finished running. There should now be a <host_name>.key and a <host_name>.crt file in your insightsgw folder.
  2. Using administrative privileges, import the .crt file in your insightsgw folder into the Windows Certificate Store's Trusted Root Certification Authorities. Use the following steps:
    • Open Manage computer certificates.
    • Open Trusted Root Certification Authorities.
    • Right click Certificates.
    • Click All Tasks > Import.
    • Select the .crt file created using openssl.
    • Flush DNS using by running ipconfig /flushdns from the command prompt.
  3. Add new Inbound and Outbound rules for port 9999 using Windows Defender Firewall with Advanced Security.

If you are experiencing issues connecting to the console after creating a self-signed certificate, check the browser to make sure an exception does not need to be added for the site created in the certificate. In the example above, the site is https://insights.esri.com. Navigate to https://insights.esri.com in the browser. If an exception is required for the self-signed certificate, a warning page will be shown:

Click Advanced

Click Add Exception...

Click Confirm Security Exception

Create a gateway

When creating a Jupyter Kernel Gateway choose either the Docker or Anaconda section below (not both).

Create a gateway using Docker

  1. Install Docker.
  2. Create a folder called insightsgw on your local machine for example, c:\insightsgw.
  3. Copy and paste Dockerfile from this repo's Docker folder into the insightsgw folder.
  4. Edit Dockerfile to fit your requirements.

The ENV KG_ALLOW_ORIGIN and ENV KG_LIST_KERNELS parameters are required for the Jupyter Kernel Gateway

If you want to load data into the container to work with inside the Insights console, create a local folder and add the data files to it and then set that folder to be copied from the local machine to the container in the Dockerfile. For example, create a folder in your local project folder insightsgw named data and add working data files to it. Then, add COPY /data/* /data/ to the Dockerfile to copy the local data folder with the files to the container. Adding WORKDIR /data to the Dockerfile will set the container working folder to the data folder.

  1. Run docker build -t insightsgw . from a command prompt in the folder created in step 2. For example, c:\insightsgw>docker build -t insightsgw .
  2. Run docker run -p 9999:9999 insightsgw from a command prompt in the folder created in step 2. For example, c:\insightsgw>docker run -p 9999:9999 insightsgw

If there is a database on the local host machine with data to be used in the Insights console, a connection to it can be created in the docker run command. For example, connecting to PostgreSQL in Windows docker run -p 9999:9999 -e DB_PORT=5432 -e DB_HOST=docker.for.win.host.internal insightsgw

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

  1. Leave the command prompt open while the Jupyter Kernel Gateway is running.
  2. To test that the gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

    You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

The default configuration of Docker is limiting so it is best practice to use Docker's Advanced Setting tab to configure additional CPU and memory resources.

In addition to configuring resources for the container from the Docker application, they can be configured within Dockerfile as well. See the Docker documentation for configuration details.

Create a gateway using Anaconda

  1. Install Anaconda.
  2. Open the Anaconda Command Prompt window.
  3. Create an Insights Python environment using the command conda create -n my_insights_env python=3.6.
  4. Activate the insights Python environment using the command conda activate my_insights_env.
  5. Install Jupyter Kernel Gateway using the command conda install -c anaconda jupyter_kernel_gateway=2.1.0.
  6. Install pandas and NumPy packages along with any other packages you require using the command conda install -c anaconda numpy pandas.
  7. Add the ArcGIS API for Python using the command conda install -c esri arcgis.
  8. Install essential R packages using the command conda install -c r r-essentials.
  9. Add the itertools package using the command conda install -c conda-forge r-itertools.

Consider what directory to use when launching the Jupyter Kernel Gateway. If you have a C:\insightsgw\data directory which contains .csv files, launching the Jupyter Kernel Gateway from this folder will enable easy access to those files within scripts by allowing the use of relative paths for file and folder access.

The KernelGatewayApp.allow_origin and JupyterWebsocketPersonality.list_kernels parameters are required.

  1. Launch the Jupyter Kernel Gateway using the following command, changing the placeholders for <host_name>
jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=C:\insightsgw\<host_name>.crt --keyfile=C:\insightsgw\<host_name>.key 

Example:

jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=C:\insightsgw\insights.crt --keyfile=C:\insightsgw\insights.key 

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

  1. Leave the command prompt open while the Jupyter Kernel Gateway is running.

  2. To test that the Gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

    You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

macOS instructions

Create a TLS or SSL cert

  1. Clone the insights-scripting-guide repo to your local machine.
  2. Create a directory called insightsgw on your local machine. For example, insightspc:insightsgw insightsuser$ .
  3. Copy and paste the insightsgw.cnf file from this repo's Cert directory into the insightsgw directory and edit each placeholder value (host_name and domain) within the file.

Example:

[dn]
CN=insights.esri.com
[req]
distinguished_name=dn
[EXT]
subjectAltName=DNS:insights.esri.com
keyUsage=digitalSignature
extendedKeyUsage=serverAuth

When editing the insightsgw.cnf file, if you are unsure of your host_name run sudo scutil --get LocalHostName from the terminal to get the correct value. If you are unsure of your domain run sudo scutil --get HostName from the terminal to get the correct value.

  1. Using the terminal, change directories to insightsgw. For example, insightspc:Documents insightsuser$ cd insightsgw.

    openssl is installed on macOS by default.

  1. Change the placeholders for <host_name> and <dns_name> and run the following command:

openssl req -x509 -days 365 -out <host_name>.crt -keyout <host_name>.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=<host_name>.<dns_name>' -extensions EXT -config ../insightsgw.cnf

Example:

openssl req -x509 -days 365 -out insights.crt -keyout insights.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=insights.esri.com' -extensions EXT -config ../insightsgw.cnf

    The terminal can be closed when finished.

    There should now be a <host_name>.key and a <host_name>.crt file in your insightsgw directory.

  1. Using administrative privileges, import the .crt file in your insightsgw directory into the Mac Keychain. Use the following steps:
    • Open Keychain
    • Choose System from Keychains
    • Choose Certificates from Category
    • Click File > Import Items
    • Choose .crt file created using openssl
    • Flush the DNS using sudo killall -HUP mDNSResponder from terminal

If you are experiencing issues connecting to the console after creating a self-signed certificate, check the browser to make sure an exception does not need to be added for the site created in the certificate. In the example above, the site is https://insights.esri.com. Navigate to https://insights.esri.com in the browser. If an exception is required for the self-signed certificate, a warning page will be shown:

Click Advanced

Click Add Exception...

Click Confirm Security Exception

Create a gateway

When creating a Jupyter Kernel Gateway choose either the Docker or Anaconda section below (not both).

Create a gateway using Docker

  1. Install Docker.
  2. Create a directory called insightsgw on your local machine. For example, insightspc:insightsgw insightsuser$.
  3. Copy and paste Dockerfile from this repo's Docker directory into the insightsgw directory.
  4. Edit Dockerfile to fit your requirements.

The ENV KG_ALLOW_ORIGIN and ENV KG_LIST_KERNELS parameters are required for the Jupyter Kernel Gateway.

If you want to load data into the container to work with inside the Insights console, create a local directory and add the data files to it and then set that directory to be copied from the local machine to the container in Dockerfile. For example, create a directory in your local project directory insightsgw named data and add working data files to it. Then, add COPY /data/* /data/ to Dockerfile to copy the local data directory with the files to the container. Adding WORKDIR /data to Dockerfile will set the container working directory to the data directory.

  1. Run docker build -t insightsgw . from the terminal in the directory created in step 2. For example, insightspc:insightsgw insightsuser$docker build -t insightsgw .
  2. Run docker run -p 9999:9999 insightsgw from the terminal in the folder created in step 2. For example, insightspc:insightsgw insightsuser$docker run -p 9999:9999 insightsgw

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

    Leave the terminal open while the Jupyter Kernel Gateway is running.

  1. To test that the Gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.
        You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}.

The default configuration of Docker is limiting so it is highly recommended to use Docker's Advanced Setting tab to configure additional CPU and memory resources.

In addition to configuring resources for the container from the Docker application, they can be configured within the Dockerfile as well. See the Docker documentation for configuration details.

Create a gateway using Anaconda

  1. Install Anaconda.
  2. Open the terminal window.
  3. Create an Insights Python environment using the command conda create -n my_insights_env python=3.6.
  4. Activate the insights Python environment using the command source activate my_insights_env.
  5. Install Jupyter Kernel Gateway using the command conda install -c anaconda jupyter_kernel_gateway.
  6. Install pandas and NumPy packages along with any other packages you require using the command conda install -c anaconda numpy pandas.
  7. Add the ArcGIS API for Python using the command conda install -c esri arcgis.
  8. Install essential R packages using the command conda install -c r r-essentials.
  9. Add the itertools package using the command conda install -c conda-forge r-itertools.

Consider what directory to use when launching the Jupyter Kernel Gateway. If you have a /Users/insightsuser/Documents/insightsgw/data directory which contains .csv files, launching the Jupyter Kernel Gateway from this directory will enable easy access to those files within scripts by allowing the use of relative paths for file and directory access.

The KernelGatewayApp.allow_origin and JupyterWebsocketPersonality.list_kernels parameters are required.

  1. Launch the Jupyter Kernel Gateway using the following command, changing the placeholders for <host_name>
jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=../certs/<host_name>.crt --keyfile=../certs/<host_name>.key

Example:

jupyter kernelgateway --KernelGatewayApp.ip=0.0.0.0 --KernelGatewayApp.port=9999 --KernelGatewayApp.allow_origin='*' --KernelGatewayApp.allow_credentials='*' --KernelGatewayApp.allow_headers='*' --KernelGatewayApp.allow_methods='*' --JupyterWebsocketPersonality.list_kernels=True --certfile=../certs/insights.crt --keyfile=../certs/insights.key

    Upon success the prompt will show the following output:

      [KernelGatewayApp] Jupyter Kernel Gateway at https://0.0.0.0:9999

    Leave the terminal open while the Jupyter Kernel Gateway is running.

  1. To test that the gateway is running as expected, open a web browser and navigate to https://<host_name>.<dns_domain>:9999/api. For example, https://insights.esri.com:9999/api.

  2. You should see JSON text on the web page showing the api version information. For example, {"version": "5.7.4"}

Use the console in Insights

The Insights Console is available within workbooks only in the Enterprise version, not in Online. You can launch the console by clicking the Console button next to the Basemap button in the workbook toolbar.

Learn more about the scripting console in the Insights documentation.

Connecting your Jupyter Kernel Gateway

  1. To connect Insights to your Jupyter Kernel Gateway, click the Console button .

    The New Jupyter Kernel Gateway connection window opens.

  1. Enter the URL and web socket connection information.

    If using the examples in this guide, the URL parameter will be https://<host_name>.<dns_domain>:9999 and the Web socket parameter will be wss://<host_name>.<dns_domain>:9999.

Example:

  • URL

      https://insights.esri.com:9999

  • Web Socket

      wss://insights.esri.com:9999

Shortcuts and Insights magic commands

The Insights console uses keyboard shortcuts and magic commands so that routine tasks can be performed quickly and efficiently.

  • Use the %insights_return(<R data frame or Pandas DataFrame>) magic command to add an R data frame or Pandas DataFrame to the Insights data pane.

    The %insights_return magic command must be run on a single line in a single cell in the Insights Console

  • Ctrl/control + Alt/option + B Add %insights_return magic command to cell.
  • Ctrl/control + / Comment line.
  • Ctrl/control + Spacebar Enable IntelliSense.
  • Shift/shift + Enter/return Execute the code in the current cell.
  • Shift/shift + Up Arrow or Down Arrow Access the history of executed cells.

What is Insights for ArcGIS?

Part of the Esri Geospatial Cloud, Insights for ArcGIS is web-based data analytics made for advanced location intelligence. Answer questions you didn’t know to ask, analyze data completely, and tell powerful data stories. Connect to your data directly, then use maps, charts and tables to perform basic to complex analyses that scale based on skill level and business need.

Why Get Involved?

Send feedback to us concerning the Beta Release of the Insights scripting console. Found an issue or have questions? Feel free to post questions or comments and report bugs to the issues section of this repo.

Start using Insights for ArcGIS with a Free Trial

Sign-up to start a free trial.

Licensing

Copyright 2019 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

You can’t perform that action at this time.