ee_to_pandas doesn't handle '<band>': None in feature collection

[msteckle]

I have a code snippet here that calculates a median composite of images (image collection). I then sample the pixels near a point using reduceRegions (points_ee in the code snippet below was created by finding the x-meter buffer around a point geometry). Finally, I want to export the feature collection generated from reduceRegions. I thought ee_to_pandas will work for me:

# Loop through date ranges and export sampled composites
for RANGE in date_ranges:
    
    # select cloud-filtered sentinel 2 imagery for time step
    s2_by_date = s2_sr.filterDate(RANGE[0], RANGE[1])
    sentinel2 = s2_by_date.filterBounds(points_ee)

    # if image collection has images:
    if sentinel2.size().getInfo() != 0:

        # create composite for time step
        composite = (sentinel2.select('ndvi')).median()
        composite_date = (sentinel2.select('date')).reduce(ee.Reducer.first())
        composite = composite.addBands(composite_date)

        # sample composite with points (returns feature collection)
        sampled = composite.reduceRegions(collection = points_ee,
                                          reducer = ee.Reducer.median(),
                                          scale = SCALE,
                                          crs = 'EPSG:4326')
        
        pp = pprint.PrettyPrinter(indent=1)
        pp.pprint(sampled.getInfo())
        
        # export feature collection as csv
        FILE = f'{PATH}/{RANGE[0]}_to_{RANGE[1]}.csv'
        if os.path.isfile(FILE):
            print('FILE:', FILE, ' ALREADY EXISTS', flush=True)
        else:
            df = geemap.ee_to_pandas(sampled, col_names=['date_first', 'ndvi'])
            df[IDCOL] = unique_ids
            df.to_csv(FILE)
            print(f'Saved to {FILE}')

However, in my case, when reduceRegions returns an NDVI value of None, ee_to_pandas fails. Below, I have printed the feature collection for reference, along with the error message. You can see that 'ndvi' is None for this buffered point (likely because of my cloud masking).

{'columns': {'date_first': 'Float',
             'ndvi': 'Float<-1.0, 1.0>',
             'system:index': 'String'},
 'features': [{'geometry': {'coordinates': [[[-149.082388, 69.94605896597855],
                                             [-149.08260790097276,
                                              69.94604822119668],
                                             [-149.0828102971945,
                                              69.9460168421673],
                                             [-149.08297907749014,
                                              69.94596732674606],
                                             [-149.08310080684603,
                                              69.9459036164723],
                                             [-149.08316579586892,
                                              69.9458307827875],
                                             [-149.08316887197944,
                                              69.94575462331628],
                                             [-149.0831097909788,
                                              69.94568120035568],
                                             [-149.08299325628104,
                                              69.94561635831536],
                                             [-149.0828285443447,
                                              69.94556525852067],
                                             [-149.082628766174,
                                              69.9455319684027],
                                             [-149.0824098237037,
                                              69.94551913776591],
                                             [-149.082189144132,
                                              69.94552778789142],
                                             [-149.0819842929101,
                                              69.94555723025475],
                                             [-149.08181157573264,
                                              69.94560512132445],
                                             [-149.0816847407488,
                                              69.945667649082],
                                             [-149.08161388425285,
                                              69.94573983642394],
                                             [-149.08160464694737,
                                              69.94581593730628],
                                             [-149.08165776477938,
                                              69.94588989410818],
                                             [-149.08176901015383,
                                              69.94595581981565],
                                             [-149.08192952826775,
                                              69.94600846664432],
                                             [-149.0821265418499,
                                              69.9460436437936],
                                             [-149.0823443682408,
                                              69.94605855106973],
                                             [-149.082388, 69.94605896597855]]],
                            'type': 'Polygon'},
               'id': '0',
               'properties': {'date_first': 1565906698240, 'ndvi': None},
               'type': 'Feature'}],
 'type': 'FeatureCollection'}
Traceback (most recent call last):
  File "/home/6ru/anaconda3/envs/gee/lib/python3.11/site-packages/geemap/common.py", line 8931, in ee_to_df
    df = df[col_names]
         ~~^^^^^^^^^^^
  File "/home/6ru/anaconda3/envs/gee/lib/python3.11/site-packages/pandas/core/frame.py", line 3767, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/6ru/anaconda3/envs/gee/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 5877, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/home/6ru/anaconda3/envs/gee/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 5941, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "['ndvi'] not in index"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/poseidon/remotesensing/arctic/scripts/gee/S2_timeseries_extraction.py", line 331, in <module>
    df = geemap.ee_to_df(sampled, col_names=['date_first', 'ndvi'])
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/6ru/anaconda3/envs/gee/lib/python3.11/site-packages/geemap/common.py", line 8938, in ee_to_df
    raise Exception(e)
Exception: "['ndvi'] not in index"

When I remove the line data = ee_object.map(lambda f: ee.Feature(None, f.toDictionary(property_names))) from ee_to_pandas and instead set data = ee_object I no longer run into the Exception: "['ndvi'] not in index" error. So, I've solved my problem, but I wonder if you might want to incorporate something into your ee_to_pandas function to handle this issue.

Thanks Dr. Wu!

[giswqs]

Thank you for reporting. The line data = ee_object.map(lambda f: ee.Feature(None, f.toDictionary(property_names))) is used to avoid exporting geometry coordinates, which can be overwhelming.

I noticed that when converting an ee.Feature to ee.Dictionary, GEE automatically removes the column with None values. See below. I am not sure if this is intentional behavior or a GEE bug.

Anyway, I have improved the function to add the missing column back to the dataframe. See #1776. Update the package using geemap.update_package() and restart the kernel to take effect.

BTW, you should avoid using for loop with GEE. It is not computationally efficient. In your case, zonal statistics is a much better solution. Something like this

image = colleciton.toBands()
geemap.zonal_stats(image, vector, out_csv)

• https://geemap.org/notebooks/12_zonal_statistics/

[msteckle]

Thank you Dr. Wu for the update and suggestion! I struggle with GEE inefficiencies a lot, so I appreciate it.