Issue downloading LandSat 8 Collection 2

[jacquesmoati]

Hello,
I encountered an issue downloading some LandSat 8 Collection 2 images with landsatxplore API.
The output / error written is :

    raise EarthExplorerError(error_msg)
landsatxplore.errors.EarthExplorerError: Download is not available

I checked on usgs website (earth explorer) that the tile I am looking for is available.
Thank you for your help

[abhilashsinghimd]

I am also facing the same issue.

[fkroeber]

I believe this error is due to the wrongly composed url used to query the data from the USGS API. Specifically, I found the data_product_ids specified in landsat/earthexplorer.py to cause the error. To me it seems as they have changed recently and being updated by @yannforget in a corresponding commit de530c1. But to query older products, the previously used data_product_ids are still necessary. By reimplementing them into earthexplorer.py I was able to download collection 2 data:

At the beginning of earthexplorer.py...

DATA_PRODUCTS = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "5e83d0a0f94d7d8d",
    "landsat_etm_c2_l1": "5e83d0d0d2aaa488",
    "landsat_ot_c2_l1": "5e81f14ff4f9941c",
    "landsat_tm_c2_l2": "5e83d11933473426",
    "landsat_etm_c2_l2": "5e83d12aada2e3c5",
    "landsat_ot_c2_l2": "5e83d14f30ea90a9",
    "sentinel_2a": "5e83a42c6eba8084",
}

DATA_PRODUCTS_II = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "5e83d0a0f94d7d8d",
    "landsat_etm_c2_l1": "5e83d0d0d2aaa488",
    "landsat_ot_c2_l1": "5e81f14ff4f9941c",
    "landsat_tm_c2_l2": "5e83d11933473426",
    "landsat_etm_c2_l2": "5e83d12aed0efa58",
    "landsat_ot_c2_l2": "5e83d14fec7cae84",
    "sentinel_2a": "5e83a42c6eba8084",
}

... and at the end of the script...

  if not dataset:
      dataset = guess_dataset(identifier)
  if is_display_id(identifier):
      entity_id = self.api.get_entity_id(identifier, dataset)
  else:
      entity_id = identifier
  try:
      url = EE_DOWNLOAD_URL.format(
          data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id
      )
      filename = self._download(url, output_dir, timeout=timeout, skip=skip)
  except:
      url = EE_DOWNLOAD_URL.format(
          data_product_id=DATA_PRODUCTS_II[dataset], entity_id=entity_id
      )
      filename = self._download(url, output_dir, timeout=timeout, skip=skip)

This may also solve #42

Of course, once again the data_product_ids are hardcoded this way and the construction with a try-except-clause is pretty ugly and just a fast workaround. In the medium term it seems preferable to avoid hard-coding and query the data_product_id for each scene via the USGS API (also in case they change it a again some time). I guess the "dataset-metadata" or "dataset-get-customization" methods can be used for this purpose https://m2m.cr.usgs.gov/api/docs/reference/. I haven't found time to try and implement it yet, but feel free to pick it up from here.

[hiewliwen]

@fkroeber I made a couple of changes to the data_product_ids as they were not the original codes.

DATA_PRODUCTS_II = {
    "landsat_tm_c1": "5e83d08fd9932768",
    "landsat_etm_c1": "5e83a507d6aaa3db",
    "landsat_8_c1": "5e83d0b84df8d8c2",
    "landsat_tm_c2_l1": "5e83d0a0f94d7d8d",
    "landsat_etm_c2_l1": "5e83d0d08fec8a66",
    "landsat_ot_c2_l1": "5e81f14f92acf9ef",
    "landsat_tm_c2_l2": "5e83d11933473426",
    "landsat_etm_c2_l2": "5e83d12aed0efa58",
    "landsat_ot_c2_l2": "5e83d14fec7cae84",
    "sentinel_2a": "5e83a42c6eba8084",
}

[TomekMatuszek]

Hi, can you explain how should I make above changes in earthexplorer.py? I edited file in Notepad++ and saved it, but after importing module again, I still have issues with downloading Landsat 8 data. Code below.

from landsatxplore.api import API
from landsatxplore.earthexplorer import EarthExplorer

scenes = api.search(
dataset = 'landsat_ot_c2_l1',
latitude = 16.85,
longitude = 52.35,
start_date = '2020-06-01',
end_date = '2020-09-01',
max_cloud_cover = 10
)

ee.download('LC81610482020178LGN00', output_dir='F:/Studia')

Thanks for help,
Tomek

[rrsc1234]

Hi. I want to download Landsat 9 C2 L1 data. So what changes I need to make in earthexplorer.py? With the above mentioned changes I am successfully able to download Landsat 8 data. But here I am stuck with downloading of Landsat 9 data.

[jorcy-yu]

Hi,I encountered an issue downloading LandSat 8 Collection 2 images with landsatxplore API.
The output / error written is :
Traceback (most recent call last):
File "E:/code/arc/scripts/download.py", line 102, in
Download_from_Landsatexplore(dataset, search_list)
File "E:/code/arc/scripts/download.py", line 66, in Download_from_Landsatexplore
ee.download(scene['entityId'], output_dir_demon)
File "D:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgis-py3\lib\site-packages\landsatxplore\earthexplorer.py", line 101, in download
filename = self._download(url, output_dir, timeout=timeout)
File "D:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgis-py3\lib\site-packages\landsatxplore\earthexplorer.py", line 79, in _download
local_filename = r.headers['Content-Disposition'].split('=')[-1]
File "D:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgis-py3\lib\site-packages\requests\structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'content-disposition'

[ChangpeiHe]

if still cannot solve this problem, it's advised to apply a proxy

[Hanjx01]

Hi，with this change i can download data before 2023,but sitll connot download data after 2023,how can i solve it?thank u

[polair]

@Hanjx01 and others. For anyone not understanding what's happening here and who arrived from a search engine, there are some hardcoded url paths as others have mentioned and these need to be updated in time. However, there is also the fact that new data (2023, for example) is available for query but not available for download. This can be verified in the web app.