Skip to content
Permalink
master
Switch branches/tags
Go to file
collapsible section for old versions
6 contributors

Users who have contributed to this file

@scw @cleebp @guillochon @sandeepgadhwal @juli7579 @scdub

Deep Learning Libraries Installers for ArcGIS

ArcGIS Pro, Server and the ArcGIS API for Python all include tools to use AI and Deep Learning to solve geospatial problems, such as feature extraction, pixel classification, and feature categorization. This installer includes a broad collection of components, such as PyTorch, TensorFlow, Fast.ai and scikit-learn, for performing deep learning and machine learning tasks, a total collection of 99 packages. These packages can be used with the Deep Learning Training tools, interactive object detection, by using the arcgis.learn module within the ArcGIS API for Python, and directly imported into your own scripts and tools. Most of the tools in this collection will work on any machine, but common deep learning workflows require a recent NVIDIA graphics processing unit (GPU), and problem sizes are bound by available GPU memory, see the requirements section.

This installer adds all the included packages to the default arcgispro-py3 environment that Pro and Server both ship with, and no additional environments are necessary in order to get started using the tools. If you do create custom environments, these packages will also be included so you can use the same tools in your own custom environments as well.

For an example of the kinds of workflows this installer and ArcGIS enables, see the AI & Deep Learning in the UC 2020 Plenary video

Download

GitHub All Releases

Downloads for Previous Releases

Once you've downloaded the archive for your product, extract the Zip file to a new location, and run the Windows Installer (MSI, e.g. ProDeepLearning.msi) on Windows. Note that this will install the deep learning frameworks into your currently active Pro Python environment; if you wish to install in an environment aside from the default arcgispro-py3 environment, switch to that environment before running the MSI using either ArcGIS Pro's Python manager UI or on the command line with proswap. You'll need to extract the file (not just open the .MSI from within the Zip file) or the installer won't be able to find its contents. On Linux, extrac the .tar.gz archive, e.g. with tar xvf <file>.tar.gz, then run the DeepLearning-Setup.sh script. After installation, the archive and installer files can be deleted.

Manual Installation

If you cannot use the Pro installer, you can install the libraries manually using these instructions:

Developer install steps:

  • Clone arcgispro-py3 and activate your clone before installing (see the manual installation instructions for cloning steps)
  • From the Python Command Prompt run conda install deep-learning-essentials to install the latest developer package set for any released version of Pro.
    • Note: this will install an experimental package set that has not been verified with Pro yet

Additional Installation for Disconnected Environment

If you will be working in a disconnected environment, download the arcgis_dl_backbones package and follow the instructions under the Steps to Install listed on the package page. The package places backbones for deep learning models in the specified install location, eliminating the need for internet access when training deep learning models in ArcGIS.

Next Steps

Once you've installed the deep learning libraries, you can use the Deep Learning Tools to train geospatial deep learning models. You can also find out more about the capabilities of the arcgis.learn module which provides specialized access to many geospatial models beyond those directly available as Geoprocessing tools. Finally, you can add any of the above libraries to your own workflows, by importing the packages listed below.

A collection of recent User Conference 2020 Technical Workshops on Deep Learning:

Requirements

Most of the packages included in the Deep Learning Libraries installer will work out of the box on any machine configuration. For example, PyTorch optionally can take advantage of a GPU, but will fall back to running its calculations on the CPU if a GPU is not available. However, GPU computation is significantly faster, and some packages such as TensorFlow in this distribution only will work with a supported GPU. CUDA, or Compute Unified Device Architecture, is a general purpose computing platform for GPUs, a requirement for current GPU backed deep learning tools.

GPU requirement Supported
GPU Type NVIDIA with CUDA Compute Capability* 3.5 minimum, 6.1 or higher recommended
Dedicated graphics memory minimum: 2GB
recommended: 8GB, depending on the deep learning model architecture and the batch size being used

* NVIDIA provides a list of CUDA enabled products and their compute capability.

† GPU memory, unlike system memory, cannot be accessed 'virtually'. If a model training consumes more GPU memory than you have available, it will fail. GPU memory is also shared across all uses of the machine, so open Pro projects with maps and other applications can limit the available memory for use with these tools.

Manifest of included packages in Pro 2.8 / Server 10.9.0

Library Name Version Description
absl-py 0.12.0 Abseil Python Common Libraries, see https://github.com/abseil/abseil-py.
ase 3.19.1 Set of tools for atomistic simulations
astor 0.8.1 Read, rewrite, and write Python ASTs nicely
beautifulsoup4 4.9.3 Python library designed for screen-scraping
boost 1.73.0 Free peer-reviewed portable C++ source libraries.
cachetools 4.2.2 Extensible memoizing collections and decorators
catalogue 1.0.0 Super lightweight function registries for your library
cloudpickle 1.6.0 Extended pickling support for Python objects
cudatoolkit 10.1.243 NVIDIA CUDA toolkit
cudnn 7.6.5 NVIDIA's cuDNN deep neural network acceleration library
cymem 2.0.5 Manage calls to calloc/free through Cython
cython 0.29.23 The Cython compiler for writing C extensions for the Python language
cython-blis 0.4.1 Fast matrix-multiplication as a self-contained Python library – no system dependencies!
cytoolz 0.11.0 Cython implementation of Toolz. High performance functional utilities
dask-core 2021.5.0 Parallel Python with task scheduling
deep-learning-essentials 2.8 A collection of the essential packages to work with deep learning packages and ArcGIS Pro.
fastai 1.0.60 fastai makes deep learning with PyTorch faster, more accurate, and easier
fastprogress 0.2.3 A fast and simple progress bar for Jupyter Notebook and console.
fasttext 0.9.2 fastText - Library for efficient text classification and representation learning
filelock 3.0.12 A platform independent file lock.
fsspec 0.9.0 A specification for pythonic filesystems
gast 0.2.2 Python AST that abstracts the underlying Python version
google-auth 1.21.3 Google authentication library for Python
google-auth-oauthlib 0.4.2 Google Authentication Library, oauthlib integration with google-auth
google-pasta 0.2.0 pasta is an AST-based Python refactoring library
googledrivedownloader 0.4 Minimal class to download shared files from Google Drive.
graphviz 2.38 Open Source graph visualization software.
grpcio 1.35.0 HTTP/2-based RPC framework
imageio 2.8.0 A Python library for reading and writing image data
joblib 1.0.1 Lightweight pipelining: using Python functions as pipeline jobs.
keepalive 0.5 An HTTP handler for urllib that supports HTTP 1.1 and keepalive
keras-applications 1.0.8 Applications module of the Keras deep learning library.
keras-base 2.3.1
keras-gpu 2.3.1 Deep Learning Library for Theano and TensorFlow
keras-preprocessing 1.1.2 Data preprocessing and data augmentation module of the Keras deep learning library
laspy 1.7.0 A Python library for reading, modifying and creating LAS files
libboost 1.73.0 Free peer-reviewed portable C++ source libraries
libopencv 4.5.0 Computer vision and machine learning software library.
libprotobuf 3.14.0 Protocol Buffers - Google's data interchange format. C++ Libraries and protoc, the protobuf compiler.
libwebp 1.2.0 WebP image library
llvmlite 0.36.0 A lightweight LLVM python binding for writing JIT compilers.
locket 0.2.1 File-based locks for Python for Linux and Windows
markdown 3.3.4 Python implementation of Markdown.
murmurhash 1.0.5 Cython bindings for MurmurHash2
ninja 1.10.2 A small build system with a focus on speed
numba 0.53.0 NumPy aware dynamic Python compiler using LLVM
nvidia-ml-py3 7.352.0 Python bindings to the NVIDIA Management Library
onnx 1.7.0 Open Neural Network Exchange library
onnx-tf 1.5.0 Experimental Tensorflow Backend for ONNX
opencv 4.5.0 Computer vision and machine learning software library.
opt_einsum 3.3.0 Optimizing einsum functions in NumPy, Tensorflow, Dask, and more with contraction order optimization.
partd 1.2.0 Data structure for on-disk shuffle operations
plac 1.1.0 The smartest command line arguments parser in the world
plotly 4.5.4 An interactive JavaScript-based visualization library for Python
pooch 1.0.0 A friend to fetch your Python library's sample data files
preshed 3.0.2 Cython Hash Table for Pre-Hashed Keys
protobuf 3.14.0 Protocol Buffers - Google's data interchange format.
py-boost 1.73.0 Free peer-reviewed portable C++ source libraries.
py-opencv 4.5.0 Computer vision and machine learning software library.
pyasn1 0.4.8 ASN.1 types and codecs
pyasn1-modules 0.2.8 A collection of ASN.1-based protocols modules.
pytorch 1.4.0 PyTorch is an optimized tensor library for deep learning using GPUs and CPUs.
pywavelets 1.1.1 Discrete Wavelet Transforms in Python
rdflib 5.0.0 Library for working with RDF, a simple yet powerful language for representing information.
retrying 1.3.3 Simplify the task of adding retry behavior to just about anything.
rsa 4.7.2 Pure-Python RSA implementation
sacremoses 0.0.43 Python port of Moses tokenizer, truecaser and normalizer.
scikit-image 0.17.2 Image processing routines for SciPy
scikit-learn 0.23.2 A set of python modules for machine learning and data mining
sentencepiece 0.1.91 SentencePiece python wrapper
soupsieve 2.2.1 A modern CSS selector implementation for BeautifulSoup
spacy 2.2.4 Industrial-strength Natural Language Processing
sparqlwrapper 1.8.5 SPARQL Endpoint interface to Python for use with rdflib
srsly 1.0.2 Modern high-performance serialization utilities for Python
tensorboard 2.4.0 TensorBoard lets you watch Tensors Flow
tensorboard-plugin-wit 1.6.0 What-If Tool TensorBoard plugin
tensorboardx 2.1 Tensorboard for PyTorch.
tensorflow 2.1.0 TensorFlow is a machine learning library.
tensorflow-addons 0.9.1 Useful extra functionality for TensorFlow 2.x
tensorflow-base 2.1.0 Base GPU package, tensorflow only.
tensorflow-estimator 2.1.0 TensorFlow Estimator
tensorflow-gpu 2.1.0 Metapackage for selecting a TensorFlow variant.
termcolor 1.1.0 ANSII Color formatting for output in terminal.
thinc 7.4.0 Learn super-sparse multi-class models
threadpoolctl 2.1.0 Python helpers to control the threadpools of native libraries
tifffile 2020.10.1 Read and write image data from and to TIFF files.
tokenizers 0.8.1 Fast State-of-the-Art Tokenizers optimized for Research and Production
toolz 0.11.1 A functional standard library for Python
torch-cluster 1.5.4 Extension library of highly optimized graph cluster algorithms for use in PyTorch
torch-geometric-1.5.0 Geometric deep learning extension library for PyTorch
torch-scatter 2.0.4 Extension library of highly optimized sparse update (scatter and segment) operations
torch-sparse 0.6.1 Extension library of optimized sparse matrix operations with autograd support
torch-spline-conv 1.2.0 PyTorch implementation of the spline-based convolution operator of SplineCNN
torchvision 0.5.0 image and video datasets and models for torch deep learning
tqdm 4.59.0 A Fast, Extensible Progress Meter
transformers 3.3.0 State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch
typeguard 2.7.0 Runtime type checker for Python
wasabi 0.6.0 A lightweight console printing and formatting toolkit
werkzeug 0.16.1 The comprehensive WSGI web application library.

Additional Notes

  • Though this package distributes the GPU based versions of packages, CPU versions can still be installed and used on any machine Pro supports. To install TensorFlow for the CPU, from the Python backstage you can install the tensorflow-mkl package to get a CPU only version.
  • This installer adds packages to the default arcgispro-py3 environment. Any subsequent clones of that environment will also include this full collection of packages. This collection of packages is validated and tested against the version of Pro is installed alongside, and upgrades of Pro will also require reinstallation of the deep learning libraries. Note that when you upgrade the software to a new release, you'll need to uninstall the Deep Learning Libraries installation as well as Pro or Server, and reinstall the new version of this package for that release.
  • This installer is only available for ArcGIS Pro 2.6+, and ArcGIS Server 10.8.1+ -- for earlier releases, you'll need to follow the documentation for that release on installing the packages through the Python backstage or Python command prompt.
  • If you want these packages for a specific environment only, you can install the deep-learning-essentials package which has the same list of dependencies as a standalone conda metapackage.