Perform Intersection and download data

[maby200]

Hi

I am using the tutorial from Caravan_part1_Earth_Engine.ipynb and there is a part that says: " The shapefile needs to have one field that indicates the basin/gauge ID, which will be used to link the derived data to the individual basin. Make sure to adapt the corresponding variable in the "General configuration" section below. "

ASSET_NAME = ''
OUTPUT_FOLDER_NAME = ''
BASIN_ID_FIELD = ''
BASIN_PREFIX = ''
AREA_MAX_THRESHOLD = 2000
AREA_MIN_THRESHOLD = 50

The basin_ID field in my shp file is:

| MYHYBAS_ID | col1 | col2 | ... |
| 6120315190 | ... | ........ | ... |
| 612..............00 | ... | ........ | ... |
| .......................... | ... | ........ | ... |

(This table has 1070 12-level basins)

In the tutorial, everything runs good but there is a part: Perform Intersection and download data where an error rises and says:

0%
0/16 [00:03<?, ?it/s]

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

[<ipython-input-7-34e2c1968ea5>](https://localhost:8080/#) in <module>()
    57         # the area of all intersecting HydroATLAS polygons, even the small bits that we exclude when
    58         # we aggregate the attributes.
    ---> 59         if polygon["properties"]["Intersect"] not in results[basin_id]["area_fragments"]:
    60             results[basin_id]["area_fragments"].append(polygon["properties"]["Intersect"])

KeyError: 6120320260

I thought this error was due to my mistake of using level-3 basins, so I changed it to the 12-level but still rising error. Could you give me any clue, please?

edit 3: I have fixed geometries from hydroBASINS but same error (with different basin_ID) yields.

[kratzert]

Hey @maby200

Edit: See answer in my first comment below this post.

To be honest, I am not quite sure I understand what happens here, as I have never seen such an error. It seems like there is a mismatch over basin-id's that are returned from

# list of all available basins
basin_ids = basins.aggregate_array(BASIN_ID_FIELD).getInfo()

which is used to populate the results dictionary just below

# dictionary to store the results
results = {b: defaultdict(list) for b in basin_ids}

So any basin id that is in your shapefile in the column name like the value of BASIN_ID_FIELD, should exist as a key in the results dictionary. If you want, you can send my your basin shape file or share the asset with me (f.kratzert (at) gmail.com) and I could have a look at it.

On another note: You probably don't want to process 1070 basins all at once. I should maybe add this to the wiki guide. There are two reasons for this:

The raw generated text files (spatial average per basin, per feature, and per hour) of ERA5L is still huge. As a reference, I process basins in batches of ~600 polygons and the raw text files that are saved in Drive (and then processed in the second notebook) are still in the range of 80-90GB. You can extrapolate from that how large it would be for 1070 basins and if that fits into your Drive. It will also take quite a while (~2-3 days) before all EE tasks will be completed. So what I usually do is to split my shapefile into smaller groups, then process them sequentially, one after the other.

[kratzert]

Okay, I found the problem and it is quite easy to fix: The problem is that your basin id field has the exact same name than the HydroATLAS_lvl12 data that is being intersected. So when the intersecting polygon is created, the features of both shape files (your catchments and HydroATLAS) are combined. This is why this basin (6120320260) and some others appear again, even though they have been filtered out previously because of their area being smaller than the MIN_AREA_THRESHOLD. So all you need to do is to give your basin id field in your shape file a new name (I renamed the field in QGIS, then ingested the updated shapefile to Earth Engine and it worked).

But again, consider splitting your shapefile into smaller batches because your still have 758 polygons in the are boundaries or otherwise make sure your have around 100GB of free space in your Drive, which is needed if you want to get the ERA5L data for the full time period.

P.S.: Next time you encounter a bug/code problem, feel free to open an Issue, which I think would be the better place for that. I will add a check to the notebook that raises an error if the BASIN_ID_FIELD is the same as the default fields of HydroATLAS

[kratzert]

#8 adds a check to the notebook that will raise an error of the BASIN_ID_FIELD is identical to any of the HydroATLAS fields.

[maby200]

Nice! I'll change it.
And one more question: Does the tutorial accepts level-5 basins from HydroALTAS?

[kratzert]

The code in this repository is not a "tutorial" but the code to derive the data for the Caravan dataset and/or to extend this dataset. I can not give you any promise on other use cases and I fear that it is up to you to test that and eventually adapt parts if it doesn't work.

[maby200]

Fine :)
Thank you @kratzert