Skip downloading when not in pre-sampled coords #1821
Conversation
There was a problem hiding this comment.
So the issue occurs when the range specified for sample_ssl4eo.py is different from the range specified for download_ssl4eo.py and some indices have no lat/lon specified in sampled_locations.csv? Makes sense, it's probably good to skip those. I wonder if we should add a warning that we are skipping those. For example, if someone requests to try downloading 1M images but only gets 1K, they might be surprised.
|
@wangyi111 may be interested in reviewing this |
|
yes I think if we skip then a warning is good. Normally, the indices range for download_ssl4eo.py is supposed to be covered by sampled_locations.csv. So the warning can help the user check again if this is really needed. One rare use case could be you want to filter out some coords in sampled_locations.csv (e.g. you want to download more images within an area), move those entries to another sampled_locations_special.csv file for download. In this case, you still specify the full indices range in sampled_locations.csv to download_ssl4eo.py, and the script can skip those ids that are not in the new csv. |
|
I'd recommend returning False if skipped and returning True if completed, then at the end of the script printing how many were skipped. |
|
I added a warning message. Adding return values and summing them is more work, but @calebrob6 can add another commit/PR if he wants. |
* Skip downloading an entry if it is not in the given pre-sampled list. * Add warning message when skipping --------- Co-authored-by: Michiaki Tatsubori <mich@mich-mbp5.local> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
SSL4EO's download_ssl4eo.py can fail with NoKey error if given pre-sampled coords like sampled_locations.csv do not contain any index specified by the indices range.