diff --git a/CHANGELOG.md b/CHANGELOG.md index 87806ba6..21151ced 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. - Spatial search support for collections via `bbox` parameter on `/collections` endpoint. Collections are now indexed with a `bbox_shape` field (GeoJSON polygon) derived from their spatial extent for efficient geospatial queries when created or updated. [#481](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/481) - Introduced SFEOS Tools (`sfeos_tools/`) - An installable Click-based CLI package for managing SFEOS deployments. Initial command `add-bbox-shape` adds the `bbox_shape` field to existing collections for spatial search compatibility. Install with `pip install sfeos-tools[elasticsearch]` or `pip install sfeos-tools[opensearch]`. [#481](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/481) +- Moved SFEOS Tools to its own repository at [Healy-Hyperspatial/sfeos-tools](https://github.com/Healy-Hyperspatial/sfeos-tools). The CLI package is now maintained separately. [#PR_NUMBER] - CloudFerro logo to sponsors and supporters list [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485) - Latest news section to README [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485) diff --git a/README.md b/README.md index 737fb812..f23300ae 100644 --- a/README.md +++ b/README.md @@ -30,12 +30,20 @@ The following organizations have contributed time and/or funding to support the ## Latest News -
- -- **10/12/2025:** Collections search **bbox** functionality added! The collections search extension now supports bbox queries. Collections will need to be updated via the API or with the new **[SFEOS-tools](#sfeos-tools-cli)** CLI package to support geospatial discoverability. Thanks again to **CloudFerro** for their sponsorship of this work! +- **10/15/2025:** šŸš€ SFEOS Tools v0.1.0 Released! - The new `sfeos-tools` CLI is now available on [PyPI](https://pypi.org/project/sfeos-tools/) +- **10/15/2025:** Added `reindex` command to **[SFEOS-tools](https://github.com/Healy-Hyperspatial/sfeos-tools)** for zero-downtime index updates when changing mappings or settings. The new `reindex` command makes it easy to apply mapping changes, update index settings, or migrate to new index structures without any service interruption, ensuring high availability of your STAC API during maintenance operations. +- **10/12/2025:** Collections search **bbox** functionality added! The collections search extension now supports bbox queries. Collections will need to be updated via the API or with the new **[SFEOS-tools](https://github.com/Healy-Hyperspatial/sfeos-tools)** CLI package to support geospatial discoverability. šŸ™ Thanks again to **CloudFerro** for their sponsorship of this work! - **10/04/2025:** The **[CloudFerro](https://cloudferro.com/)** logo has been added to the sponsors and supporters list above. Their sponsorship of the ongoing collections search extension work has been invaluable. This is in addition to the many other important changes and updates their developers have added to the project. -
+
+View Older News (Click to Expand) + +------------- +- **09/25/2025:** v6.5.0 adds a new GET/POST /collections-search endpoint (disabled by default via ENABLE_COLLECTIONS_SEARCH_ROUTE) to avoid conflicts with the Transactions Extension, and enhances collections search with structured filtering (CQL2 JSON/text), query, and datetime filtering. These changes make collection discovery more powerful and configurable while preserving compatibility with transaction-enabled deployments. + + +
+ ## Project Introduction - What is SFEOS? @@ -170,7 +178,7 @@ These endpoints support advanced collection discovery features including: - **Implementation Note**: When collections are created or updated, a `bbox_shape` field is automatically generated from the collection's spatial extent and indexed as a GeoJSON polygon for efficient geospatial queries - **Migrating Legacy Collections**: Collections created before this feature was added will not be discoverable via bbox search until they have the `bbox_shape` field added. You can either: - Update each collection via the API (PUT `/collections/{collection_id}` with the existing collection data) - - Run the migration tool (see [SFEOS Tools CLI](#sfeos-tools-cli) for installation and connection options): + - Use the [SFEOS Tools CLI](https://github.com/Healy-Hyperspatial/sfeos-tools) (install with `pip install sfeos-tools[elasticsearch]` or `pip install sfeos-tools[opensearch]`): - `sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl` - `sfeos-tools add-bbox-shape --backend opensearch --host db.example.com --no-ssl` @@ -488,7 +496,7 @@ The system uses a precise naming convention: ## SFEOS Tools CLI -- **Overview**: SFEOS Tools is an installable CLI package for managing and maintaining SFEOS deployments. +- **Overview**: [SFEOS Tools](https://github.com/Healy-Hyperspatial/sfeos-tools) is an installable CLI package for managing and maintaining SFEOS deployments. This CLI package provides utilities for managing and maintaining SFEOS deployments. - **Installation**: ```shell @@ -498,14 +506,11 @@ The system uses a precise naming convention: # For OpenSearch (from PyPI) pip install sfeos-tools[opensearch] - # For local development - pip install -e sfeos_tools[elasticsearch] - # or - pip install -e sfeos_tools[opensearch] ``` - **Available Commands**: - `add-bbox-shape`: Add bbox_shape field to existing collections for spatial search support + - `reindex`: Reindex all STAC indices (collections and per-collection items) to new versioned indices and update aliases; supports both Elasticsearch and OpenSearch backends. Use this when you need to apply mapping changes, update index settings, or migrate to a new index structure. The command handles the entire process including creating new indices, reindexing data, and atomically updating aliases with zero downtime. - **Basic Usage**: ```shell @@ -540,9 +545,15 @@ The system uses a precise naming convention: # Using --help for more information sfeos-tools --help sfeos-tools add-bbox-shape --help + sfeos-tools reindex --help + ``` -For more details, see the [SFEOS Tools README](./sfeos_tools/README.md). +- **Documentation**: + For complete documentation, examples, and advanced usage, please visit the [SFEOS Tools GitHub repository](https://github.com/Healy-Hyperspatial/sfeos-tools). + +- **Contributing**: + Contributions, bug reports, and feature requests are welcome! Please file them on the [SFEOS Tools issue tracker](https://github.com/Healy-Hyperspatial/sfeos-tools/issues). ## Ingesting Sample Data CLI Tool diff --git a/sfeos_tools/LICENSE b/sfeos_tools/LICENSE deleted file mode 100644 index 1c6074b8..00000000 --- a/sfeos_tools/LICENSE +++ /dev/null @@ -1,21 +0,0 @@ -MIT License - -Copyright (c) 2025 Jonathan Healy and CloudFerro S.A. - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/sfeos_tools/README.md b/sfeos_tools/README.md deleted file mode 100644 index 5347c28b..00000000 --- a/sfeos_tools/README.md +++ /dev/null @@ -1,113 +0,0 @@ -# SFEOS Tools - -CLI tools for managing [stac-fastapi-elasticsearch-opensearch](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch) deployments. - -## Installation - -### For Elasticsearch - -```bash -pip install sfeos-tools[elasticsearch] -``` - -Or for local development: -```bash -pip install -e sfeos_tools[elasticsearch] -``` - -### For OpenSearch - -```bash -pip install sfeos-tools[opensearch] -``` - -Or for local development: -```bash -pip install -e sfeos_tools[opensearch] -``` - -### For Development (both backends) - -```bash -pip install sfeos-tools[dev] -``` - -Or for local development: -```bash -pip install -e sfeos_tools[dev] -``` - -## Usage - -After installation, the `sfeos-tools` command will be available: - -```bash -# View available commands -sfeos-tools --help - -# View version -sfeos-tools --version - -# Get help for a specific command -sfeos-tools add-bbox-shape --help -``` - -## Commands - -### add-bbox-shape - -Add `bbox_shape` field to existing collections for spatial search support. - -**Basic usage:** - -```bash -# Elasticsearch -sfeos-tools add-bbox-shape --backend elasticsearch - -# OpenSearch -sfeos-tools add-bbox-shape --backend opensearch -``` - -**Connection options:** - -```bash -# Local Docker Compose (no SSL) -sfeos-tools add-bbox-shape --backend elasticsearch --no-ssl - -# Remote server with SSL -sfeos-tools add-bbox-shape \ - --backend elasticsearch \ - --host db.example.com \ - --port 9200 \ - --user admin \ - --password secret - -# Using environment variables -ES_HOST=my-cluster.cloud.com ES_PORT=9243 ES_USER=elastic ES_PASS=changeme \ - sfeos-tools add-bbox-shape --backend elasticsearch -``` - -**Available options:** - -- `--backend`: Database backend (elasticsearch or opensearch) - **required** -- `--host`: Database host (default: localhost or ES_HOST env var) -- `--port`: Database port (default: 9200 or ES_PORT env var) -- `--use-ssl / --no-ssl`: Use SSL connection (default: true or ES_USE_SSL env var) -- `--user`: Database username (default: ES_USER env var) -- `--password`: Database password (default: ES_PASS env var) - -## Development - -To develop sfeos-tools locally: - -```bash -# Install in editable mode with dev dependencies -pip install -e ./sfeos_tools[dev] - -# Run the CLI -sfeos-tools --help -``` - -## License - -MIT License - see the main repository for details. diff --git a/sfeos_tools/setup.py b/sfeos_tools/setup.py deleted file mode 100644 index d5cf4416..00000000 --- a/sfeos_tools/setup.py +++ /dev/null @@ -1,56 +0,0 @@ -"""Setup for SFEOS Tools.""" - -from setuptools import find_packages, setup - -with open("README.md", "r", encoding="utf-8") as f: - long_description = f.read() - -setup( - name="sfeos-tools", - version="0.1.0", - description="CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments", - long_description=long_description, - long_description_content_type="text/markdown", - author="Jonathan Healy, CloudFerro S.A.", - license="MIT", - url="https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch", - packages=find_packages(), - python_requires=">=3.8", - install_requires=[ - "click>=8.0.0", - ], - extras_require={ - "elasticsearch": [ - "stac_fastapi_core", - "sfeos_helpers", - "stac_fastapi_elasticsearch", - ], - "opensearch": [ - "stac_fastapi_core", - "sfeos_helpers", - "stac_fastapi_opensearch", - ], - "dev": [ - "stac_fastapi_core", - "sfeos_helpers", - "stac_fastapi_elasticsearch", - "stac_fastapi_opensearch", - ], - }, - entry_points={ - "console_scripts": [ - "sfeos-tools=sfeos_tools.cli:cli", - ], - }, - classifiers=[ - "Development Status :: 4 - Beta", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.8", - "Programming Language :: Python :: 3.9", - "Programming Language :: Python :: 3.10", - "Programming Language :: Python :: 3.11", - "Programming Language :: Python :: 3.12", - ], -) diff --git a/sfeos_tools/sfeos_tools/__init__.py b/sfeos_tools/sfeos_tools/__init__.py deleted file mode 100644 index 9106f13e..00000000 --- a/sfeos_tools/sfeos_tools/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""SFEOS Tools - Utilities for managing stac-fastapi-elasticsearch-opensearch deployments.""" - -__version__ = "0.1.0" diff --git a/sfeos_tools/sfeos_tools/cli.py b/sfeos_tools/sfeos_tools/cli.py deleted file mode 100644 index 827e8116..00000000 --- a/sfeos_tools/sfeos_tools/cli.py +++ /dev/null @@ -1,229 +0,0 @@ -"""SFEOS CLI Tools - Utilities for managing stac-fastapi-elasticsearch-opensearch deployments. - -This tool provides various utilities for managing and maintaining SFEOS deployments, -including database migrations, maintenance tasks, and more. - -Usage: - sfeos-tools add-bbox-shape --backend elasticsearch - sfeos-tools add-bbox-shape --backend opensearch -""" - -import asyncio -import logging -import sys - -import click - -from stac_fastapi.sfeos_helpers.database import add_bbox_shape_to_collection -from stac_fastapi.sfeos_helpers.mappings import COLLECTIONS_INDEX - -logging.basicConfig(level=logging.INFO) -logger = logging.getLogger(__name__) - - -async def process_collection_bbox_shape(client, collection_doc, backend): - """Process a single collection document to add bbox_shape field. - - Args: - client: Elasticsearch/OpenSearch client - collection_doc: Collection document from database - backend: Backend type ('elasticsearch' or 'opensearch') - - Returns: - bool: True if collection was updated, False if no update was needed - """ - collection = collection_doc["_source"] - collection_id = collection.get("id", collection_doc["_id"]) - - # Use the shared function to add bbox_shape - was_added = add_bbox_shape_to_collection(collection) - - if not was_added: - return False - - # Update the collection in the database - if backend == "elasticsearch": - await client.index( - index=COLLECTIONS_INDEX, - id=collection_id, - document=collection, - refresh=True, - ) - else: # opensearch - await client.index( - index=COLLECTIONS_INDEX, - id=collection_id, - body=collection, - refresh=True, - ) - - logger.info(f"Collection '{collection_id}': Added bbox_shape field") - return True - - -async def run_add_bbox_shape(backend): - """Add bbox_shape field to all existing collections. - - Args: - backend: Backend type ('elasticsearch' or 'opensearch') - """ - import os - - logger.info( - f"Starting migration: Adding bbox_shape to existing collections ({backend})" - ) - - # Log connection info (showing what will be used by the client) - es_host = os.getenv("ES_HOST", "localhost") - es_port = os.getenv( - "ES_PORT", "9200" - ) # Both backends default to 9200 in their config - es_use_ssl = os.getenv("ES_USE_SSL", "true") - logger.info(f"Connecting to {backend} at {es_host}:{es_port} (SSL: {es_use_ssl})") - - # Create client based on backend - if backend == "elasticsearch": - from stac_fastapi.elasticsearch.config import AsyncElasticsearchSettings - - settings = AsyncElasticsearchSettings() - else: # opensearch - from stac_fastapi.opensearch.config import AsyncOpensearchSettings - - settings = AsyncOpensearchSettings() - - client = settings.create_client - - try: - # Get all collections - response = await client.search( - index=COLLECTIONS_INDEX, - body={ - "query": {"match_all": {}}, - "size": 10000, - }, # Adjust size if you have more collections - ) - - total_collections = response["hits"]["total"]["value"] - logger.info(f"Found {total_collections} collections to process") - - updated_count = 0 - skipped_count = 0 - - for hit in response["hits"]["hits"]: - was_updated = await process_collection_bbox_shape(client, hit, backend) - if was_updated: - updated_count += 1 - else: - skipped_count += 1 - - logger.info( - f"Migration complete: {updated_count} collections updated, {skipped_count} skipped" - ) - - except Exception as e: - logger.error(f"Migration failed with error: {e}") - raise - finally: - await client.close() - - -@click.group() -@click.version_option(version="0.1.0", prog_name="sfeos-tools") -def cli(): - """SFEOS Tools - Utilities for managing stac-fastapi-elasticsearch-opensearch deployments.""" - pass - - -@cli.command("add-bbox-shape") -@click.option( - "--backend", - type=click.Choice(["elasticsearch", "opensearch"], case_sensitive=False), - required=True, - help="Database backend to use", -) -@click.option( - "--host", - type=str, - default=None, - help="Database host (default: localhost or ES_HOST env var)", -) -@click.option( - "--port", - type=int, - default=None, - help="Database port (default: 9200 for ES, 9202 for OS, or ES_PORT env var)", -) -@click.option( - "--use-ssl/--no-ssl", - default=None, - help="Use SSL connection (default: true or ES_USE_SSL env var)", -) -@click.option( - "--user", - type=str, - default=None, - help="Database username (default: ES_USER env var)", -) -@click.option( - "--password", - type=str, - default=None, - help="Database password (default: ES_PASS env var)", -) -def add_bbox_shape(backend, host, port, use_ssl, user, password): - """Add bbox_shape field to existing collections for spatial search support. - - This migration is required for collections created before spatial search - was added. Collections created or updated after this feature will - automatically have the bbox_shape field. - - Examples: - sfeos_tools.py add-bbox-shape --backend elasticsearch - sfeos_tools.py add-bbox-shape --backend opensearch --host db.example.com --port 9200 - sfeos_tools.py add-bbox-shape --backend elasticsearch --no-ssl --host localhost - """ - import os - - # Set environment variables from CLI options if provided - if host: - os.environ["ES_HOST"] = host - if port: - os.environ["ES_PORT"] = str(port) - if use_ssl is not None: - os.environ["ES_USE_SSL"] = "true" if use_ssl else "false" - if user: - os.environ["ES_USER"] = user - if password: - os.environ["ES_PASS"] = password - - try: - asyncio.run(run_add_bbox_shape(backend.lower())) - click.echo(click.style("āœ“ Migration completed successfully", fg="green")) - except KeyboardInterrupt: - click.echo(click.style("\nāœ— Migration interrupted by user", fg="yellow")) - sys.exit(1) - except Exception as e: - error_msg = str(e) - click.echo(click.style(f"āœ— Migration failed: {error_msg}", fg="red")) - - # Provide helpful hints for common errors - if "TLS" in error_msg or "SSL" in error_msg: - click.echo( - click.style( - "\nšŸ’” Hint: If you're connecting to a local Docker Compose instance, " - "try adding --no-ssl flag", - fg="yellow", - ) - ) - elif "Connection refused" in error_msg: - click.echo( - click.style( - "\nšŸ’” Hint: Make sure your database is running and accessible at the specified host:port", - fg="yellow", - ) - ) - sys.exit(1) - - -if __name__ == "__main__": - cli()