`to_featureclass` fails when exporting to shapefile under certain conditions

[stdavis]

Describe the bug
When exporting a dataset with a numeric field that has all null values, GeoAccessor.to_featureclass() fails with a ValueError.

This is in an environment without arcpy.

To Reproduce
Steps to reproduce the behavior:

import arcgis
from pathlib import Path


current_folder = Path(__file__).parent
url = "https://services1.arcgis.com/99lidPhWCzftIe9K/ArcGIS/rest/services/DWQAssessedWaters/FeatureServer/0"

feature_layer = arcgis.features.FeatureLayer(url)

print("getting features...")
feature_set = feature_layer.query()

sdf = feature_set.sdf

print("exporting to shapefile...")
sdf.spatial.to_featureclass(current_folder / "test.shp", sanitize_columns=False)

error:

Traceback (most recent call last):
  File "/workspaces/deq-enviro/cloudrun/src/repro.py", line 20, in <module>
    sdf.spatial.to_featureclass(current_folder / "test.shp", sanitize_columns=False)
  File "/usr/local/lib/python3.10/dist-packages/arcgis/features/geo/_accessor.py", line 2668, in to_featureclass
    result = to_featureclass(
  File "/usr/local/lib/python3.10/dist-packages/arcgis/features/geo/_io/fileops.py", line 1275, in to_featureclass
    res = _pyshp2(df=df, out_path=out_location, out_name=fc_name)
  File "/usr/local/lib/python3.10/dist-packages/arcgis/features/geo/_io/fileops.py", line 1507, in _pyshp2
    shpfile.record(*row)
  File "/usr/local/lib/python3.10/dist-packages/shapefile.py", line 2300, in record
    self.__dbfRecord(record)
  File "/usr/local/lib/python3.10/dist-packages/shapefile.py", line 2335, in __dbfRecord
    value = float(value)
ValueError: could not convert string to float: '385-501-9582'

Screenshots
n/a

Expected behavior
I expect the shapefile to be created successfully.

Platform (please complete the following information):

• OS: Docker image: ghcr.io/osgeo/gdal:ubuntu-small-latest
• Python API Version: 2.2.0.2
• pyshp package version: 2.3.1

Additional context
This is my workaround:

import arcgis
from pathlib import Path


current_folder = Path(__file__).parent
url = "https://services1.arcgis.com/99lidPhWCzftIe9K/ArcGIS/rest/services/DWQAssessedWaters/FeatureServer/0"

feature_layer = arcgis.features.FeatureLayer(url)

print("getting features...")
feature_set = feature_layer.query()

sdf = feature_set.sdf
for field in sdf.columns:
    #: reset field values for non-string fields who's values are all null
    #: this prevents problems when converting to shapefile
    if sdf[field].isnull().all() and sdf[field].dtype != "string":
        print(f"resetting field type as string for field: {field}")
        sdf[field] = sdf[field].astype("string")

print("exporting to shapefile...")
sdf.spatial.to_featureclass(current_folder / "test.shp", sanitize_columns=False)

I believe that the problem may be in this snippet of code from arcgis/features/geo/_io/fileops.py beginning on line 1462 :

else:
  cfields.append(c)
  if isinstance(df[c].loc[idx], (str)):
      shpfile.field(name=c, size=255)
  elif isinstance(df[c].loc[idx], (int)):
      shpfile.field(name=c, fieldType="N", size=5)
  elif isinstance(df[c].loc[idx], np.int32):
      shpfile.field(name=c, fieldType="N", size=10)
  elif isinstance(df[c].loc[idx], (float, np.float64, np.int64)):
      shpfile.field(name=c, fieldType="F", size=19, decimal=11)
  elif (
      isinstance(
          df[c].loc[idx],
          (datetime.datetime, np.datetime64),
      )
      or df[c].dtype.name.find("datetime") > -1
  ):
      shpfile.field(name=c, fieldType="D", size=8)
      dfields.append(c)
  elif isinstance(df[c].loc[idx], (bool)):
      shpfile.field(name=c, fieldType="L", size=1)

Because the "Perimeter" field type in the source data does not satisfy any of these conditionals, it is never passed to shpfile.field(...) and thus never added to the shapefile. This creates an off-by-one issue with the rows of data are added.

[nanaeaubry]

@stdavis Thanks for reporting this and providing all the detail! we will take a look

[nanaeaubry]

@stdavis What is the field type of "Perimeter" if it is not being addressed in the conditions?

[stdavis]

I don't remember. You would have to step through the code to see it.

[achapkowski]

Your issue is because you are using a shapefile. If you have NULL values, shapefiles are not the right choice for your data.

https://support.esri.com/en-us/knowledge-base/faq-how-do-you-represent-null-values-when-converting-fr-000004419

[stdavis]

Thanks for the tip! I had no idea that shapefiles do not support null values.

I do still think that you have a significant bug that needs to be resolved. Currently, the code just skips over the null values field and tries to jam data into the wrong field. If someone were to input a dataset with all numeric fields, the tool might finish successfully but the values would all be in the wrong fields. In my example above, data from Phone field was attempting to be jammed into the Perimeter field.

I would expect this library to do automatic conversion something like this doc: https://pro.arcgis.com/en/pro-app/latest/tool-reference/appendices/geoprocessing-considerations-for-shapefile-output.htm#:~:text=Null%20value%20representation

Or I would expect it to return a more helpful error message saying that Shapefiles do not support null values.

I've implemented a workaround in my own code to convert the problem fields to strings and empty values so it's not a huge deal to me. But I do think that you have a chance to significantly improve the experience for future devs that run into this issue. I had to dig into arcgis package code to figure out what was going on.