Skip to content
Python API designed to work externally with ArcGIS REST Services to query and extract data, and view service properties. Uses arcpy for some functions if available, otherwise uses open source alternatives to interact with the ArcGIS REST API. Also includes a subpackage for administering ArcGIS Server Sites.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.idea
.vscode
docs
restapi
.gitignore
LICENSE
MANIFEST.in
README.md
ReleaseNotes.txt
ReleaseNotes1.txt master updates Nov 29, 2018
setup.py

README.md

restapi

This is a Python API for working with ArcGIS REST API, ArcGIS Online, and Portal/ArcGIS Enterprise. This package has been designed to work with arcpy or open source and does not require arcpy. It will try to use arcpy if available for some data conversions, otherwise will use open source options. Also included is a subpackage for administering ArcGIS Server Sites. This is updated often, so continue checking here for new functionality!

One of the first things you might do is to connect to a services directory (or catalog):

Connect to external services

# connect NOAA ArcGIS Server Instance
rest_url = 'https://gis.ngdc.noaa.gov/arcgis/rest/services'

# no authentication is required, so no username and password are supplied
ags = restapi.ArcServer(rest_url)

# get folder and service properties
print('Number of folders: {}'.format(len(ags.folders)))
print('Number of services: {}'.format(len(ags.services)))

# walk thru directories
for root, folders, services in ags.walk():
    print(root)
    print(folders)
    print(services)
    print('\n')

Connecting to a map service from within the ArcServer object

# access "ahps_gauges" service (stream gauges)
gauges = ags.getService('ahps_gauges')
print(gauges.url) #print(MapService url

# print(layer names
print(gauges.list_layers())

# access "observed river stages" layer
lyr = gauges.layer('observed_river_stages') #not case sensitive, also supports wildcard search (*)

# list fields from col layer
print(lyr.list_fields())

You can also query the layer and get back arcpy.da Cursor like access

# run search cursor for gauges in California
# (maximimum limit may be 1000 records, can use get_all=True to exceed transfer limit)
# can filter fields by putting a field list, can use actual shape field name to get
#  geometry or use the ArcGIS-like token "SHAPE@"
# all fields are gathered by the default ("*") and fields can be filtered by providing a list
query = "state = 'CA'"
for row in lyr.cursor(where=query, fields=['SHAPE@', u'gaugelid', u'status', u'location']):
    print(row)

# Note: can also do this from the MapService level like this:
# cursor = gauges.cursor('observed_river_stages', where=query)

The layer can also be exported to a shapefile or KMZ

# export Nebraska "College/University" layer to feature class
# make scratch folder first
folder = os.path.join(os.environ['USERPROFILE'], r'Desktop\restapi_test_data')
if not os.path.exists(folder):
    os.makedirs(folder)

# export layer to shapefile (can also call from Map Service)
output = os.path.join(folder, 'California_Stream_Gauges.shp')
lyr.layer_to_fc(output, where=query, sr=102100) #override spatial reference with web mercator

# export to KMZ
kmz = output.replace('.shp', '.kmz')
lyr.layer_to_kmz(kmz, where=query)

Clipping a layer is also easy

# clip lyr by polygon (Sacramento area)
esri_json = {"rings":[[[-121.5,38.6],[-121.4,38.6],
                      [-121.3,38.6],[-121.2,38.6],
                      [-121.2,38.3],[-121.5,38.3],
                      [-121.5,38.6]]],
            "spatialReference":
                {"wkid":4326,"latestWkid":4326}}

# clip by polygon and filter fields (can use polygon shapefile or feature class as well)
sac = os.path.join(folder, 'Sacramento_gauges.shp')
lyr.clip(esri_json, sac, fields=['gaugelid', 'location'])

You can also connect to a MapService directly

url = 'http://gis.srh.noaa.gov/arcgis/rest/services/ahps_gauges/MapServer'
gauges = restapi.MapService(url)

Feature Editing

# feature service testing
fs_url = 'http://sampleserver3.arcgisonline.com/ArcGIS/rest/services/SanFrancisco/311Incidents/FeatureServer'
fs = restapi.FeatureService(fs_url)

# reference a feature layer within featuer service
#   note - layer method returns a restapi.FeatureLayer, different from restapi.MapServiceLayer
incidents = fs.layer('incidents')

# add new feature
adds = [{"attributes":
   	        {
        	"req_type":"Damaged Property",
        	"req_date":"07/19/2009",
        	"req_time":"11:08",
        	"address":"173 RESTAPI DR",
        	"x_coord":"-122.4167",
        	"y_coord":"37.7833",
        	"district":"7",
        	"status":1},
        "geometry": {"x":-122.4167,
                      "y":37.7833,
                      "spatialReference":
                          {"wkid":4326}}}]

result = incidents.addFeatures(adds)

We can also add attachments

# add attachment, get new OID from add results
oid = result.addResults[0]  # must get an OID to add attachment to

# download python image online and add it to the featuer we just added above
url = 'http://www.cis.upenn.edu/~lhuang3/cse399-python/images/pslytherin.png'
im = urllib.urlopen(url).read()
tmp = os.path.join(os.path.dirname(sys.argv[0]), 'python.png')
with open(tmp, 'wb') as f:
    f.write(im)

# add attachment
incidents.addAttachment(oid, tmp)
os.remove(tmp)

# get attachment info from service and download it
attachments = incidents.attachments(oid)

for attachment in attachments:
    print(attachment)
    print attachment.contentType, attachment.size)
    attachment.download(folder) # folder is a user specified output directory

Update feature and delete features

# update the feature we just added
adds[0]['attributes']['address'] = 'Address Not Available'
adds[0]['attributes']['objectid'] = oid
incidents.updateFeatures(adds)

# now delete feature
incidents.deleteFeatures(oid)

Offline capabilities (Sync)

# if sync were enabled, we could create a replica like this:
# can pass in layer ID (0) or name ('incidents', not case sensative)
replica = fs.createReplica(0, 'test_replica', geometry=adds[0]['geometry'], geometryType='esriGeometryPoint', inSR=4326)

# now export the replica object to file geodatabase (if arcpy access) or shapefile with hyperlinks (if open source)
restapi.exportReplica(replica, folder)

Working with Image Services

url = 'http://pca-gis02.pca.state.mn.us/arcgis/rest/services/Elevation/DEM_1m/ImageServer'
im = restapi.ImageService(url)

# clip DEM
geometry = {"rings":[[
                [240006.00808044084, 4954874.19629429],
                [240157.31010183255, 4954868.8053006204],
                [240154.85966611796, 4954800.0316874133],
                [240003.55764305394, 4954805.4226145679],
                [240006.00808044084, 4954874.19629429]]],
            "spatialReference":{"wkid":26915,"latestWkid":26915}}

tif = os.path.join(folder, 'dem.tif')
im.clip(geometry, tif)

# test point identify
x, y = 400994.780878, 157878.398217
elevation = im.pointIdentify(x=x, y=y, sr=103793)
print(elevation)

Geocoding

# hennepin county, MN geocoder
henn = 'http://gis.hennepin.us/arcgis/rest/services/Locators/HC_COMPOSITE/GeocodeServer'
geocoder = restapi.Geocoder(henn)
# find target field, use the SingleLine address field by default
geoResult = geocoder.findAddressCandidates('353 N 5th St, Minneapolis, MN 55403')

# export results to shapefile
print('found {} candidates'.format(len(geoResult))
geocoder.exportResults(geoResult, os.path.join(folder, 'target_field.shp'))

# Esri geocoder
esri_url = 'http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Locators/ESRI_Geocode_USA/GeocodeServer'
esri_geocoder = restapi.Geocoder(esri_url)

# find candidates using key word arguments (**kwargs) to fill in locator fields, no single line option
candidates = esri_geocoder.findAddressCandidates(Address='380 New York Street', City='Redlands', State='CA', Zip='92373')
print('Number of address candidates: {}'.format(len(candidates)))
for candidate in candidates:
    print(candidate.location)

# export results to shapefile
out_shp = os.path.join(folder, 'Esri_headquarters.shp')
geocoder.exportResults(candidates, out_shp)

Geoprocessing Services

# test esri's drive time analysis GP Task
gp_url = 'http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Network/ESRI_DriveTime_US/GPServer/CreateDriveTimePolygons'
gp = restapi.GPTask(gp_url)

# get a list of gp parameters (so we know what to pass in as kwargs)
print('\nGP Task "{}" parameters:\n'.format(gp.name)
for p in gp.parameters:
    print('\t', p.name, p.dataType)

point = {"geometryType":"esriGeometryPoint",
         "features":[
             {"geometry":{"x":-10603050.16225853,"y":4715351.1473399615,
                          "spatialReference":{"wkid":102100,"latestWkid":3857}}}],
         "sr":{"wkid":102100,"latestWkid":3857}}

# run task, passing in gp parameters as keyword arguments (**kwargs)
gp_res = gp.run(Input_Location=str(point), Drive_Times = '1 2 3', inSR = 102100)

# returns a GPResult() object, can get at the first result by indexing (usually only one result)
# can test if there are results by __nonzero__()
if gp_res:
    result = gp_res.results[0]
    
    # this returned a GPFeatureRecordSetLayer as an outputParameter, so we can export this to polygons
    print('\nOutput Result: "{}", data type: {}\n'.format(result.paramName, result.dataType))

    # now export the result value to fc (use the value property of the GPResult object from run())
    drive_times = os.path.join(folder, 'drive_times.shp')
    restapi.exportFeatureSet(drive_times, gp_res.value)

A note about input Geometries

restapi will try to use arcpy first if you have it, otherwise will defer to open source. Both support the reading of shapefiles to return the first feature back as a restapi.Geometry object

It also supports arcpy Geometries and shapefile._Shape() objects

>>> shp = r'C:\TEMP\Polygons.shp' # a shapefile on disk somewhere
>>> geom = restapi.Geometry(shp)
>>> print(geom.envelope())
-121.5,38.3000000007,-121.199999999,38.6000000015

Token Based Security

restapi also supports secured services. This is also session based, so if you sign in once to an ArcGIS Server Resource (on the same ArcGIS Site), the token will automatically persist via the IdentityManager().

There are 3 ways to authticate:

# **kwargs for all accessing all ArcGIS resources are
# usr   -- username
# pw    -- password
# token -- token (as string or restapi.Token object)
# proxy -- url to proxy

# secured url
secured_url = 'http://some-domain.com/arcgis/rest/services'

# 1. username and password
ags = restapi.ArcServer(url, 'username', 'password')  # token is generated and persists

# 2. a token that has already been requested
ags = restapi.ArcServer(url, token=token)  # uses a token that is already active

# 3. via a proxy (assuming using the standard esri proxy)
#   this will forward all subsequent requests through the proxy
ags = restapi.ArcServer(url, proxy='http://some-domain.com/proxy.ashx')

You can even just generate a token and let the IdentityManager handle the rest. It is even smart enough to handle multiple tokens for different sites:

# login to instance 1
usr = 'username'
pw = 'password'

# urls to two different ArcGIS Server sites
url_1 = 'http://some-domain.com/arcserver1/rest/services'
url_2 = 'http://domain2.com/arcgis/rest/services'

# generate tokens
tok1 = restapi.generate_token(url_1, usr, pw)
tok2 = restapi.generate_token(url_2, usr, pw)

# now we should be able to access both ArcGIS Server sites via the IdentityManager
arcserver1 = restapi.ArcServer(url_1) # tok1 is automatically passed in and handled
arcserver2 = restapi.ArcServer(url_2) # tok2 is used here

The admin Subpackage

restapi also contains an administrative subpackage (warning: most functionality has not been tested!). You can import this module like this:

from restapi import admin

Connecting to a Portal

url = 'https://domain.gis.com/portal/home'
portal = restapi.admin.Portal(url, 'username', 'password')

# get servers
servers = portal.servers

# stop sample cities service
server = servers[0]

service = server.service('SampleWorldCities.MapServer')
service.stop()

To connect to an ArcGIS Server instance that you would like to administer you can do the following:

# test with your own servers
url = 'localhost:6080/arcgis/admin/services' #server url
usr = 'username'
pw = 'password'

# connect to ArcGIS Server instance
arcserver = admin.ArcServerAdmin(url, usr, pw)

To list services within a folder, you can do this:

folder = arcserver.folder('SomeFolder')  # supply name of folder as argument
for service in folder.iter_services():
    print(service.serviceName, service.configuredState

    # can stop a service like this
    # service.stop()

    # or start like this
    # service.start()

print('\n' * 3)

# show all services and configured state (use iter_services to return restapi.admin.Service() object!)
for service in arcserver.iter_services():
    print(service.serviceName, service.configuredState)

Security

You can set security at the folder or service level. By default, the addPermssion() method used by Folder and Service objects will make the service unavailable to the general public and only those in the administrator role can view the services. This is done by setting the 'esriEveryone' principal "isAllowed" value to false. You can also assign permissions based on roles.

arcserver.addPermission('SomeFolder')  # by default it will make private True 

# now make it publically avaiable (unsecure)
arcserver.addPermission('SomeFolder', private=False)

# secure based on role, in this case will not allow assessor group to see utility data
#   assessor is name of assessor group role, Watermain is folder to secure
arcserver.addPermission('Watermain', 'assessor', False)  

# note, this can also be done at the folder level:
folder = arcserver.folder('Watermain')
folder.addPermission('assessor', False)

Stopping and Starting Services

Services can easily be started and stopped with this module. This can be done from the ArcServerAdmin() or Folder() object:

# stop all services in a folder
arcserver.stopServices(folderName='SomeFolder') # this can take a few minutes

# look thru the folder to check the configured states, should be stopped
for service in arcserver.folder('SomeFolder').iter_services():
    print(service.serviceName, service.configuredState)

# now restart
arcserver.startServices(folderName='SomeFolder') # this can take a few minutes

# look thru folder, services should be started
for service in arcserver.folder('SomeFolder').iter_services():
    print(service.serviceName, service.configuredState)
    
# to do this from a folder, simply get a folder object back
folder = arcserver.folder('SomeFolder')
folder.stopServices()
for service in folder.iter_services():
    print(service.serviceName, service.configuredState)

Updating Service Properties

The admin package can be used to update the service definitions via JSON. By default, the Service.edit() method will pass in the original service definition as JSON so no changes are made if no arguments are supplied. The first argument is the service config as JSON, but this method also supports keyword arguments to update single properties (**kwargs). These represent keys of a the dictionary in Python.

# connect to an individual service (by wildcard) - do not need to include full name, just
# enough of the name to make it a unique name query
service = arcserver.service('SampleWorldCities') #provide name of service here

# get original service description
description = service.description

# now edit the description only by using description kwarg (must match key exactly to update)
service.edit(description='This is an updated service description')

# edit description again to set it back to the original description
service.edit(description=description)

There are also some helper methods that aren't available out of the box from the ArcGIS REST API such as enabling or disabling extensions:

# disable Feature Access and kml downloads
service.disableExtensions(['FeatureServer', 'KmlServer'])

# you can also list enabled/disabled services
print(service.enabledExtensions)
# [u'KmlServer', u'WFSServer', u'FeatureServer']

service.disabledExtensions
# [u'NAServer', u'MobileServer', u'SchematicsServer', u'WCSServer', u'WMSServer']

# Edit service extension properites
# get an extension and view its properties
fs_extension = service.getExtension('FeatureServer')

print(fs_extension) # will print as pretty json

For Service objects, all properties are represented as pretty json. Below is what the FeatureService Extension looks like:

{
  "allowedUploadFileTypes": "", 
  "capabilities": "Query,Create,Update,Delete,Uploads,Editing", 
  "enabled": "true", 
  "maxUploadFileSize": 0, 
  "properties": {
    "allowGeometryUpdates": "true", 
    "allowOthersToDelete": "false", 
    "allowOthersToQuery": "true", 
    "allowOthersToUpdate": "false", 
    "allowTrueCurvesUpdates": "false", 
    "creatorPresent": "false", 
    "dataInGdb": "true", 
    "datasetInspected": "true", 
    "editorTrackingRespectsDayLightSavingTime": "false", 
    "editorTrackingTimeInUTC": "true", 
    "editorTrackingTimeZoneID": "UTC", 
    "enableOwnershipBasedAccessControl": "false", 
    "enableZDefaults": "false", 
    "maxRecordCount": "1000", 
    "realm": "", 
    "syncEnabled": "false", 
    "syncVersionCreationRule": "versionPerDownloadedMap", 
    "versionedData": "false", 
    "xssPreventionEnabled": "true", 
    "zDefaultValue": "0"
  }, 
  "typeName": "FeatureServer"
}

Setting properties for extensions is also easy:

# set properties for an extension using helper method, use **kwargs for setting capabilities
service.setExtensionProperties('FeatureServer', capabilities='Query,Update,Delete,Editing')

# verify changes were made
print(fs_extension.capabilities
# 'Query,Update,Delete,Editing'

# alternatively, you can edit the service json directly and call the edit method
# change it back to original settings
fs_extension.capabilities = 'Query,Create,Update,Delete,Uploads,Editing'
service.edit()

# verify one more time...
print(fs_extension.capabilities)
# 'Query,Create,Update,Delete,Uploads,Editing'

Access the Data Store

You can iterate through the data store items easily to read/update/add items:

# connect to the server's data store
ds = arcserver.dataStore

# iterate through all items of data store
for item in ds:
    print(item.type, item.path
    # if it is an enterprise database connection, you can get the connection string like this
    if item.type == 'egdb':
        print(item.info.connectionString)
    # else if a folder, print(server path
    elif item.type == 'folder':
        print(item.info.path)
    print('\n')

User and Role Stores

When viewing usernames/roles you can limit the number of names returned using the "maxCount" keyword argument. To view and make changes to Role Store:

# connect to role store
rs = arcserver.roleStore

# print roles
for role in rs:
    print(role)

# find users within roles
for role in rs:
    print(role, 'Users: ', rs.getUsersWithinRole(role))

# add a user to role
rs.addUsersToRole('Administrators', 'your-domain\\someuser')

# remove user from role
rs.removeUsersFromRole('Administrators', 'your-domain\\someuser')

# remove an entire role
rs.removeRole('transportation')

To view and make changes to the User Store:

# connect to user store
us = arcserver.userStore

# get number of users
print(len(us)

# iterate through first 10 users
for user in us.searchUsers(maxCount=10):
    print(user)
    
# add new user
us.addUser('your-domain\\someuser', 'password')

# assign roles by using comma separated list of role names
us.assignRoles('your-domain\\someuser', 'Administrators,Publishers')

# get privileges from user
us.getPrivilegeForUser('your-domain\\someuser')

# remove roles from user 
us.removeRoles('your-domain\\someuser', 'Administrators,Publishers')

Log Files

You can easily query server log files like this:

import restapi
import datetime

# query log files (within last 3 days), need to convert to milliseconds
threeDaysAgo = restapi.date_to_mil(datetime.datetime.now() - relativedelta(days=3))
for log in arcserver.queryLogs(endTime=threeDaysAgo, pageSize=25):
    print(log.time
    for message in log:
        print(message)
    print('\n')

A note about verbosity

When using the admin subpackage you will likely be making changes to services/permissions etc. On operations that change a configuration, the @passthrough decorator will report back if the operation is successful and return results like this:

{u'status': u'SUCCESS'}

The printing of these messages can be shut off by changing the global "VERBOSE" variable so these messages are not reported. This can be disabled like this:

admin.VERBOSE = False 
You can’t perform that action at this time.