EEException: Invalid value at 'file_format'

[ljstrnadiii]

I am pretty stoked on the idea of this repo! Thanks for putting this out there 🙏 . This could be a huge game changer in the way people use earth engine!

Anyways, here is my setup and issue:

import ee
import xarray as xr

ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')

bbox = ee.Geometry.Rectangle(113.33, -43.63, 153.56, -10.66)

ds = xr.open_dataset(
    "ECMWF/ERA5_LAND/HOURLY", 
    engine='ee',
    geometry=bbox, 
    crs="EPSG:4326", 
)

# get some reasonable subset
subset = ds.isel(lon=slice(0,100), lat=slice(0,100))

and then I select a subset to run the mean to test actual data streaming:

subset[['surface_solar_radiation_downwards_hourly']].mean()

and I get EEException: Invalid value at 'file_format' (type.googleapis.com/google.earthengine.v1.ImageFileFormat), "NUMPY_NDARRAY"

Any ideas?

[ljstrnadiii]

I get the same issue with .to_zarr(..., compute=False), which brings up another point. I see the time dimension is very large. Is that because each "asset" or pixel group concats along that dimension? I am guessing yes, just like stackstac.

If so, in your beam example, are all those assets written to all those time dimensions, or does xbeam automatically apply a .resample(time=...) or is there some rechunking going on to unique chunks? If it doesn't, there would be a ton of nan chunks, which will equate to many many chunks and/or tasks for downstream dask users, etc. What is your plan there or is this another thing that xbeam handles?

[ljstrnadiii]

I ask about these details above because I would be stoked to add a Flyte counterpart to your beam example if you are receptive to it 🙏

[alxmrs]

I have some idea about what could be going on. For the computePixels() API, there was a recent backend addition for ImageFileFormat (NUMPY_NDARRAY). It's somewhat documented here:
https://developers.google.com/earth-engine/apidocs/ee-data-computepixels

Looking around the EE client code, it seems to be here in the latest version of the earthengine-api:
https://github.com/google/earthengine-api/blob/8ed12bd0a5864f18d639b82ea4dd912c52a8f39c/python/ee/image_converter.py#L33

Can you try upgrading your EE client library to see if that could resolve the issue? If not, we'll have to pick @naschmitz's brain about it.

Last, I would also be stoked to see more Xee examples & integrations! Happy to see Flyte in our documentation when possible.

[ljstrnadiii]

Thanks for the hunch! I'll give it a shot tomorrow morning and get back then.

[ljstrnadiii]

Looks like going from 0.1.369 to 0.1.374 did the trick! Thanks for the help!

I might open another issue to help materialize my question a bit better around duplicates in the time dimension 🙏

[alxmrs]

Thanks for raising an issue! With #56, I think we can prevent this error from happening again.