Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: Update coastsat_package Dependency to Handle Request Size Limits Gracefully #202

Closed
2 tasks done
2320sharon opened this issue Nov 7, 2023 · 2 comments
Closed
2 tasks done
Assignees
Labels
bug Something isn't working

Comments

@2320sharon
Copy link
Collaborator

2320sharon commented Nov 7, 2023

Issue: Update coastsat_package Dependency to Handle Request Size Limits Gracefully

Current Behavior

When using the coastsat_package package, if 10 or more requests exceed the download request size limit set by Google Earth Engine, all ongoing downloads halt abruptly.

Expected Behavior

The package should be updated to ensure that:

  • Each request that exceeds the request size limit is printed/logged for review.
  • Any remaining downloads continue

Goal

The primary goal is to maximize data acquisition. The system should be robust enough to handle request size limits without interrupting the entire download process, ensuring that the maximum amount of data is downloaded even if some data points are missed due to size constraints.

Steps to Reproduce

  1. Initiate multiple downloads using the coastsat_package package where the expected request size exceeds the limit set by Google Earth Engine.
  2. Observe the behavior when multiple requests go over the size limit.

It's essential to address this issue to improve the reliability and efficiency of data collection processes using the coastsat_package package.

Related Issues

SatelliteShorelines/coastsat_package#5

Tasks

  • update the coastsat_package dependency in the pyproject.toml

  • Test the updated version with the config information in sample_data.zip
    sample_data.zip

@2320sharon 2320sharon added the bug Something isn't working label Nov 7, 2023
@2320sharon 2320sharon self-assigned this Nov 7, 2023
@2320sharon
Copy link
Collaborator Author

2320sharon commented Nov 7, 2023

I've tested coastsat, coastsat_package, and coastseg with the inputs included in this issue. Both coastseg and coastsat_package encountered the issue with the request limit being exceeded. When I examined the downloaded imagery I noticed some of the images were just noise as you can see in the screenshot. However when I examined the coastsat downloads I noticed that those dates were not included in the downloaded imagery. This leads me to believe there was an update to coastsat that handles prevented these bad images from being downloaded.

image

  • I will need to investigate coastsat's code further and update coastsat_package's code accordingly.

@2320sharon
Copy link
Collaborator Author

Closed in coastseg 1.1.15

2320sharon added a commit that referenced this issue Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant