Publishing a feature service hangs

[avni-ef]

Describe the bug
Publishing a feature service can hang forever on _check_publish_status.
At first I thought this is related to this issue since I run it with future=False, but I upgraded to 2.4.1, and the problem still occurs.

To Reproduce
Steps to reproduce the behavior:

folders = gis.content.folders
root_folder = folders.get()
item = root_folder.add(item_properties=item_properties, file=geojson_local_path).result()
publish_parameters = {
            "name": feature_title,
            "maxRecordCount": 2000,
            "hasStaticData": True,
            "layerInfo": {
                "capabilities": "Query"
            }
}

feature_service = item.publish(publish_parameters=publish_parameters,
                                                   overwrite=False,future=False)

error:

<no error, process just hangs forever>

Screenshots
If applicable, add screenshots to help explain your problem.

Expected behavior
Feature service should be published successfully

Platform (please complete the following information):

• OS: [e.g. iOS] Debian GNU/Linux (containerized)
• Browser -
• Python API Version arcgis==2.4.1; arcgis_mapping==4.30.0

Additional context
I ran py-spy on the stuck process, seems the process hangs in _check_publish_status

Python v3.10.8 (/usr/local/bin/python3.10)

Thread 1 (idle): "MainThread"
    _check_publish_status (arcgis/gis/__init__.py:17547)
    _publish (arcgis/gis/__init__.py:17195)
    publish (arcgis/gis/__init__.py:16818)
    ...

[nanaeaubry]

@avni-ef We are unable to reproduce, please contact support with this issue

[avni-ef]

Hi,
Problem reproduced several times. I've ran some diagnostics on the docker container running this code, it seems there are threads stuck in Sleeping state and never terminate, I assume its the requests connection pool not removing these threads when request times out.

I've ran the following script to extract the process state:

import os
import glob
import threading
import sys
import traceback

def diagnose_hanging_threads():    
    print(f"Python threads: {threading.active_count()}")
    for thread in threading.enumerate():
        print(f"  {thread.name}: alive={thread.is_alive()}, daemon={thread.daemon}")
    
    # the python process PID is  PID=1 in container
    main_pid = 1
    proc_task_dir = f"/proc/{main_pid}/task"
    
    if os.path.exists(proc_task_dir):
        thread_dirs = glob.glob(f"{proc_task_dir}/*")
        print(f"\nSystem threads for PID {main_pid}: {len(thread_dirs)}")
        
        for thread_dir in thread_dirs:
            thread_id = os.path.basename(thread_dir)
            try:
                with open(f"{thread_dir}/stat", 'r') as f:
                    stat_line = f.read().strip()
                    fields = stat_line.split()
                    thread_state = fields[2]
                    thread_name = fields[1].strip('()')
                
                print(f"  Thread {thread_id}: {thread_name} (state: {thread_state})")
                
                try:
                    with open(f"{thread_dir}/comm", 'r') as f:
                        comm = f.read().strip()
                        if comm != thread_name:
                            print(f"    -> Command: {comm}")
                except:
                    pass
                    
            except Exception as e:
                print(f"  Thread {thread_id}: Error - {e}")
   
    print("\n=== STACK TRACES ===")
    for thread_id, frame in sys._current_frames().items():
        print(f"\nThread {thread_id}:")
        traceback.print_stack(frame, limit=5)

Results:

  Thread 1: python3 (state: S)
  Thread 7: python3 (state: S)
  Thread 8: python3 (state: S)
  Thread 9: python3 (state: S)
  Thread 10: jemalloc_bg_thd (state: S)
  Thread 12: python3 (state: S)
  Thread 13: python3 (state: S)
  Thread 14: python3 (state: S)
  Thread 15: python3 (state: S)
  Thread 16: python3 (state: S)
  Thread 17: jemalloc_bg_thd (state: S)
  Thread 18: jemalloc_bg_thd (state: S)
  Thread 19: jemalloc_bg_thd (state: S)
  Thread 80: python3 (state: S)

Please advise.

[avni-ef]

@nanaeaubry Could you please reopen this issue considering the latest findings?

[nanaeaubry]

@avni-ef Looking at the threads is not much help for us. We have been unable to reproduce so the best bet is to share your data with us so we can see if there are errors in the data itself.

You can share it here or reach out to support for help and they can better assist you.

[avni-ef]

No problem, I'll collect the data and share it when it reproduces.
Although, even if its in on ArcGIS side, I would expect this SDK to properly handle errors, and clean up any stuck threads that might cause it to hang.