Add script to download Survey123 responses

[rudokemper]

It seems quite feasible that we can add a data flow for consuming submissions data from an ArcGIS Survey123 form. This is a nice feature to add to connectors; with a Survey123 data flow added, we can say that GuardianConnector supports data integration from most of the major data collection apps being used by communities today, including the popular enterprise one.

There are two ways of doing this.

ArcGIS REST API

To do this, we can leverage the ArcGIS API query operation on the feature service layer storing all of the Survey123 results, like this: https://{subdomain}.arcgis.com/{service_id}/arcgis/rest/services/{feature_id}/FeatureServer/query

A simple way to get all of the data is using the following params:

    params = {
        'where': '1=1',  # This will query all records
        'outFields': '*',  # This will fetch all fields
        'f': 'geojson',  # Request the response in GeoJSON format
        'returnGeometry': 'true'  # Include geometry in the response
    }

Then, to get the attachments, we can iterate through all of the returned features' object_id and make the following request https://{subdomain}.arcgis.com/{service_id}/arcgis/rest/services/{feature_id}/FeatureServer/{object_id}/attachments. The response will be a GeoJSON file with URLs to download for all attachments for that object_id.

This will work for any Survey123 feature service layers that are public; for anything that is private, we will need to generate a token in exchange for user credentials, like this:

curl -X POST "https://www.arcgis.com/sharing/rest/generateToken" \
     -d "username=your_username" \
     -d "password=your_password" \
     -d "client=requestip" \
     -d "f=json"

This will return a token. You would then append &token= to the request params.

There is a script available in our (private) gc-programs repo that leverages the ArcGIS REST API and could be retooled for Windmill.

ArcGIS API for Python

We could also look at using the ArcGIS API for Python. This requires you to have an ArcGIS account. But the process could be something like the following:

from arcgis.gis import GIS
from arcgis.features import FeatureLayer

# Authenticate (replace with your credentials or use `gis = GIS()` for interactive login)
gis = GIS("https://your-org.maps.arcgis.com", "username", "password")

# Get the feature layer
survey_layer_url = "https://services.arcgis.com/.../FeatureServer/0"  # Replace with your layer URL
survey_layer = FeatureLayer(survey_layer_url)

# Query all features (you can filter if needed)
features = survey_layer.query()

# Download attachments (if available)
for feature in features:
    object_id = feature.attributes['OBJECTID']
    
    attachments = survey_layer.attachments.get_list(object_id)
    
    for attachment in attachments:
        attachment_id = attachment['id']
        file_name = attachment['name']
        
        survey_layer.attachments.download(object_id, attachment_id, save_path="./downloads")

        print(f"Downloaded: {file_name}")

There are also download_survey123.py and download_s123_attachments.py scripts in CMI's (private) cmitools repo which use the ArcGIS API for Python.