Skip to content

zoo-template-common-0.1.1

Choose a tag to compare

View all tags
@gfenoy gfenoy released this 18 Jan 18:19
· 8 commits to main since this release
v0.1.1
c5cb42d

zoo-template-common 0.1.1

zoo-template-common is a Python package providing reusable execution handlers and utilities for CWL (Common Workflow Language) workflow templates in ZOO-Project. It simplifies the development of OGC Application Packages by offering a standardized, extensible foundation for geospatial processing services.

🎯 Why use zoo-template-common?

✅ Reduce boilerplate code – No need to rewrite execution logic for each workflow
✅ Standardized STAC output handling – Automatic processing and cataloging of workflow results
✅ Cloud-native ready – Built-in S3 integration and Kubernetes pod configuration
✅ Extensible architecture – Override hooks to add custom behavior without modifying core logic
✅ Production-tested – Used in EOEPCA and EOAP platforms for Earth Observation processing
✅ Simplifies Calrissian integration – Seamless workflow execution on Kubernetes

🚀 What's New in v0.1.1

Major Improvements

📦 PyPI Publication & Distribution

  • Official PyPI Package: The package is now officially available on PyPI, making installation as simple as pip install zoo-template-common
  • Automated Publishing: Set up CI/CD pipeline for automatic publication to PyPI on new releases
  • Complete Package Metadata: Added comprehensive metadata including classifiers, keywords, and project URLs for better discoverability

📄 License Update

  • Switched to Apache-2.0 License: Moved from previous licensing to Apache-2.0, providing clearer terms for commercial and open-source use
  • Better Ecosystem Alignment: Apache-2.0 aligns with many geospatial and cloud-native projects

🐍 Python Version Requirements

  • Minimum Python 3.10: Updated to require Python 3.10 or newer
  • Support for Python 3.10, 3.11, and 3.12: Tested and compatible with the latest Python versions
  • Improved Type Hinting: Takes advantage of modern Python features for better code quality

📚 Enhanced Documentation

  • Updated Service Templates Documentation: Comprehensive docs reflecting the latest patterns and best practices
  • Real-world Examples: Added examples from production deployments (EOEPCA, EOAP)
  • API Reference: Complete documentation of all classes and methods
  • Integration Guides: Step-by-step guides for integrating with ZOO-Project and Calrissian

🔗 Dependency Management

  • Official zoo-runner-common Integration: Now uses the official zoo-runner-common package (>=0.1.0) as a dependency
  • Simplified Dependency Tree: Cleaner, more maintainable dependency structure
  • Better Version Pinning: Minimum version requirements ensure compatibility

🌟 Key Features

CommonExecutionHandler

The core of this package is the CommonExecutionHandler class, an extensible base class for CWL workflow execution:

  • Pre/Post-Execution Hooks: Inject custom logic before and after workflow execution
  • Automatic STAC Catalog Processing: Seamlessly handle SpatioTemporal Asset Catalog (STAC) outputs
  • Kubernetes Pod Configuration:
    • Environment variable management
    • Node selector configuration
    • Resource allocation
  • Secrets Handling: Secure management of sensitive configuration
  • Tool Log Integration: Automatic collection and registration of workflow execution logs
  • Output Management: Intelligent processing and validation of workflow outputs

CustomStacIO

A specialized STAC I/O class with cloud-native capabilities:

  • S3 Integration: Read and write STAC catalogs directly from/to S3-compatible storage using boto3
  • Multi-Backend Support: Works with both S3 and local file systems transparently
  • Efficient I/O Operations: Optimized for large-scale geospatial data processing

💡 Use Cases

This package is designed to streamline the creation of geospatial processing services in ZOO-Project by providing a reusable foundation for:

  1. CWL Workflow Execution: Execute Complex Workflow Language workflows with Calrissian on Kubernetes
  2. STAC Catalog Management: Process and publish geospatial results via STAC catalogs
  3. Cloud Storage Integration: Seamless integration with S3-compatible storage platforms (AWS S3, MinIO, Ceph, etc.)
  4. OGC Application Packages: Standardized input/output management for OGC API - Processes
  5. Earth Observation Processing: Production-ready foundation for EO data processing pipelines

📖 Quick Start

Installation

pip install zoo-template-common

Basic Example

from zoo_template_common import CommonExecutionHandler
from zoo_calrissian_runner import ZooCalrissianRunner

def my_workflow(conf, inputs, outputs):
    # Initialize the execution handler
    execution_handler = CommonExecutionHandler(conf=conf, outputs=outputs)
    
    # Create and configure the Calrissian runner
    runner = ZooCalrissianRunner(
        cwl=cwl,
        conf=conf,
        inputs=inputs,
        outputs=outputs,
        execution_handler=execution_handler,
    )
    
    # Execute the workflow
    exit_status = runner. execute()
    return exit_status

Extending for Custom Needs

One of the most powerful features is the ability to extend CommonExecutionHandler for specific platforms:

from zoo_template_common import CommonExecutionHandler
import jwt
import requests

class MyCustomHandler(CommonExecutionHandler):
    def __init__(self, conf, outputs):
        super().__init__(conf, outputs)
        # Add your custom initialization
        self.api_token = conf.get("auth_env", {}).get("token", "")
        self.workspace_url = conf.get("platform", {}).get("workspace_url", "")
    
    def pre_execution_hook(self):
        """Called before workflow execution"""
        super().pre_execution_hook()
        
        # Add your custom pre-execution logic
        self.setup_workspace()
        self.validate_permissions()
        self.prepare_inputs()
    
    def post_execution_hook(self, log, output, usage_report, tool_logs):
        """Called after workflow execution"""
        super().post_execution_hook(log, output, usage_report, tool_logs)
        
        # Add your custom post-execution logic
        self.register_results()
        self.update_catalog()
        self.send_notifications()
    
    def setup_workspace(self):
        """Example:  Set up a workspace for the execution"""
        response = requests.post(
            f"{self.workspace_url}/workspaces",
            headers={"Authorization": f"Bearer {self.api_token}"},
            json={"name": "my-workspace"}
        )
        self.workspace_id = response.json()["id"]

🏗️ Architecture

Design Principles

  • Separation of Concerns: Clear separation between workflow execution, I/O handling, and platform-specific logic
  • Open/Closed Principle: Open for extension (via hooks) but closed for modification
  • Dependency Inversion: Depends on abstractions (base classes) rather than concrete implementations
  • Single Responsibility: Each class has a well-defined, single purpose

Integration Points

┌─────────────────────────────────────────┐
│         ZOO-Project Service             │
└────────────────┬────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────┐
│     CommonExecutionHandler              │
│  ┌───────────────────────────────────┐  │
│  │  Pre-Execution Hook               │  │
│  └───────────────────────────────────┘  │
│                 │                        │
│                 ▼                        │
│  ┌───────────────────────────────────┐  │
│  │  ZooCalrissianRunner              │  │
│  │  (executes CWL on Kubernetes)     │  │
│  └───────────────────────────────────┘  │
│                 │                        │
│                 ▼                        │
│  ┌───────────────────────────────────┐  │
│  │  Post-Execution Hook              │  │
│  │  - STAC Processing                │  │
│  │  - Output Registration            │  │
│  └───────────────────────────────────┘  │
└─────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────┐
│      CustomStacIO (S3/Local)            │
└─────────────────────────────────────────┘

🌍 Who is Using It?

Production Deployments

  • eoepca-proc-service-template: Extends CommonExecutionHandler with EOEPCA Workspace API integration for the EOEPCA+ platform
  • zoo-service-template: Uses CommonExecutionHandler directly or extends it for specific ZOO-Project deployments
  • EOAP (Earth Observation Application Platform): Production Earth Observation processing platform

Example Projects

Browse the examples directory for:

  • Basic workflow integration
  • S3 storage configuration
  • STAC catalog processing
  • Custom handler implementations

📋 Requirements

  • Python: >= 3.10
  • Core Dependencies:
    • zoo-runner-common >= 0.1.0
    • loguru >= 0.7.0 (structured logging)
    • pystac >= 1.8.0 (STAC catalog handling)
    • pyyaml >= 6.0 (configuration parsing)
    • boto3 >= 1.28.0 (AWS S3 integration)
    • botocore >= 1.31.0 (AWS core functionality)

🔧 API Reference

CommonExecutionHandler

Constructor

CommonExecutionHandler(conf:  dict, outputs: dict = None)

Key Methods

  • pre_execution_hook(): Override to add logic before workflow execution
  • post_execution_hook(log, output, usage_report, tool_logs): Override to add logic after execution
  • setOutput(outputName, values): Process and set STAC catalog outputs
  • get_pod_env_vars(): Get environment variables for the Calrissian pod
  • get_pod_node_selector(): Get Kubernetes node selector configuration
  • get_additional_parameters(): Get additional execution parameters
  • get_secrets(): Get secrets for secure configuration
  • handle_outputs(log, output, usage_report, tool_logs): Register tool logs in service logs

CustomStacIO

Custom STAC IO class for S3 and local file operations.

Methods

  • read_text(source): Read text content from S3 or local file
  • write_text(dest, txt): Write text content to S3 or local file
  • read_json(source): Read and parse JSON from S3 or local file
  • write_json(dest, json_dict): Write JSON to S3 or local file

📚 Documentation & Support

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details on:

  • Code style and formatting
  • Testing requirements
  • Pull request process
  • Community guidelines

📜 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🙏 Acknowledgments

  • ZOO-Project Community: For building the foundation of open-source geospatial processing
  • EOEPCA Project: For supporting production deployment and testing
  • Contributors: Thanks to all contributors who have helped improve this package

📝 Changelog

v0.1.1 (2026-01-18)

Added

  • PyPI metadata for official package publication
  • Automatic CI/CD pipeline for PyPI releases
  • Comprehensive package classifiers and keywords
  • Enhanced documentation with real-world examples

Changed

  • Minimum Python version updated to 3.10
  • Switched to Apache-2.0 license
  • Added zoo-runner-common as official dependency
  • Updated documentation to reflect latest service template patterns

Fixed

  • Multiple typos in codebase and documentation
  • Package structure improvements for better imports

v0.1.0 (Previous Release)

  • Initial stable release
  • Core CommonExecutionHandler implementation
  • CustomStacIO for S3 integration
  • Basic STAC catalog processing

Ready to streamline your ZOO-Project workflows? Install now:

pip install zoo-template-common
Assets 2
Loading