Deprecation note of download capabilities #418
Description
Some history
Since v. 1.0, download of Sentinel-2 was possible from ESA’s Open Access Hub via force-level1-sentinel2. In September 2018, ESA activated the Long Term Archive (LTA) to roll out old data products from online to offline storage. This has been technically addressed in FORCE, but the retrieval of LTA-images made the use of long time series virtually impossible due to very strict quota limits for regular users.
In 2020, @ernstste developed force-level1-csd to download Landsat and Sentinel-2 imagery from Google cloud storage. This enabled mass downloads of Sentinel-2 and provided a means to download Landsat data, which was not included in FORCE before. force-level1-sentinel2 was deprecated with the provision of force-level1-csd. Since then, the capabilities have deteriorated stepwise. When USGS reprocessed the entire Landsat archive as collection 2, Google decided not to provide collection 2 on their cloud storage, hence Landsat provision was effectively discontinued in early 2022 (#167). As of August 28, 2024, downloading newer Sentinel-2 data became impossible since Google was retiring their mechanic of providing metadata (#334). Updating the tool to their new mechanic would potentially incur fees when users exceed their limits. I strictly refuse to include functionality that will trigger hidden fees. Most recently, gsutil used in force-level1-csd is not maintained anymore by Google, hence creating vulnerabilities (#416).
force-level1-landsat was developed in response to #167 by @ernstste in 2022. The tool is a standalone tool (https://github.com/ernstste/landsatlinks), which is being copied and rebranded during installation to make it feel like a native FORCE tool. Adaptations had to be made due to changes in authentication, as well as the introduction of download quotas. As of today, this tool is still working as expected.
force-lut-modis is a tool to download MODIS water vapor products and to generate water vapor look-up-tables for usage in the atmospheric correction of Landsat. Adaptations had to be made due to changes in authentication. This tool is principally still functional, but MODIS operations will cease in the near future (#260).
Executive decision
Providing download functionality is a constant battle to keep up with providers updating or substituting their API, authentication mechanisms, deprecation of data / collections, or even decommissioning and launching new platforms with new mechanisms and requirements to receive data. I am deeply thankful to @ernstste for supporting FORCE on those matters!
Regarding FORCE, this has gradually resulted in an undesirable situation, where I know that some tools are not working as users would expect and this leaves the overall impression that the software is not maintained properly. If I think about my software’s scope, FORCE’s strength and ultimate goals are the efficient and high-quality mass-processing and analysis of satellite image archives. I always considered the download tools as nice-to-have add-ons, but never as one of the core routines.
That said, I have eventually come to the decision to discontinue the download capabilities of FORCE. This step will enable me to focus on the core functionalities of FORCE without constant worrying about deteriorating download tools or changing provision points.
Steps forward
As part of this, the code will be trimmed for the next release. Specifically, force-level1-csd, force-level1-landsat, and force-lut-modis will be removed along with all its dependencies.
Users who require those functionalities can still use the tools in old FORCE versions in the future. If you work with local FORCE installations, you can always retrieve old releases from the release archive (https://github.com/davidfrantz/force/releases). In addition, Docker images for old FORCE versions will remain accessible (https://hub.docker.com/r/davidfrantz/force/tags).
As another step, I will provide a curated section to the documentation (https://force-eo.readthedocs.io/en/latest/) for collecting tools that are known to be interoperable with FORCE and its workflows. This will include download tools - but will also feature other tools that can nicely be integrated or are even specifically designed to work with FORCE or its generated data. Actually, this is a long overdue step as there are quite some powerful and useful external tools that somewhat comprise the FORCE ecosystem. Contributions to this list are highly(!!!) welcome, please see https://github.com/davidfrantz/force/blob/main/CONTRIBUTING.md for a quick guideline on how to contribute to the documentation (as well as code, of course).
Thanks for your understanding,
David