Skip to content

Add deployment helpers and core functionalities for SRGAN HPC#111

Merged
simon-donike merged 3 commits intoESAOpenSR:mainfrom
mak8427:main
Apr 28, 2026
Merged

Add deployment helpers and core functionalities for SRGAN HPC#111
simon-donike merged 3 commits intoESAOpenSR:mainfrom
mak8427:main

Conversation

@mak8427
Copy link
Copy Markdown
Contributor

@mak8427 mak8427 commented Apr 24, 2026

Summary

This PR adds a Slurm-based HPC deployment launcher for running SRGAN inference over Sentinel-2 cutouts.
Solves #110
The new srgan-hpc CLI supports:

  • config validation
  • single patch submission
  • bounding-box grid submission
  • AOI shapefile submission
  • worker task execution
  • output collection
  • run status inspection

It supports three SRGAN runtime modes:

  • rgbnir: runs the RGB-NIR preset on B04/B03/B02/B08
  • swir: runs the SWIR preset on B05/B06/B07/B8A/B11/B12
  • fused: runs both and writes a 10-band fused GeoTIFF

Main Changes

HPC Launcher and CLI

  • Added the installable deployment.srgan_hpc package.
  • Added srgan-hpc console entrypoint.
  • Added Slurm submission helpers and worker entrypoint script.
  • Added commands for validate-config, submit patch, submit grid, submit aoi, run task, collect, and status.

AOI and Patch Management

  • Added AOI shapefile support.
  • AOI input accepts either a .shp file or a directory containing exactly one .shp.
  • Added CRS validation/reprojection for AOIs.
  • Added grid/patch selection utilities with overlap support.

SRGAN Runtime Configuration

  • Added default, test, and A100 example runtime configs.
  • Added separate RGB-NIR and SWIR product configuration blocks.
  • Added support for Hugging Face presets or local config/checkpoint paths.
  • Added runtime mode selection via mode: rgbnir | swir | fused.

Inference and Output Handling

  • Added SRGAN inference wrapper using opensr_srgan and opensr-utils.
  • Added fused GeoTIFF creation by aligning SWIR output to the RGB-NIR SR grid.
  • Added deterministic fused band order:
    B04, B03, B02, B08, B05, B06, B07, B8A, B11, B12.
  • Added metadata writing, output collection, and run status helpers.

Packaging

  • Included deployment packages in setuptools discovery.
  • Added hpc optional dependencies.
  • Added package data for deployment configs and Slurm scripts.

How Has This Been Tested?

  • Deployment config tests

python -m pytest tests/test_deployment_config.py -q

  • Dry-run submission tests

python -m pytest tests/test_deployment_submit.py -q

  • AOI shapefile path handling tests

python -m pytest tests/test_deployment_aoi.py -q

  • Fused raster stacking tests

python -m pytest tests/test_deployment_raster.py -q

  • Full deployment test subset

python -m pytest
tests/test_deployment_config.py
tests/test_deployment_submit.py
tests/test_deployment_raster.py
tests/test_deployment_aoi.py
-q

Result:

8 passed

  • CLI config validation smoke test

python -m deployment.srgan_hpc.cli validate-config
--config deployment/configs/runtime.default.yaml

Checklist

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@mak8427 mak8427 mentioned this pull request Apr 27, 2026
Copy link
Copy Markdown
Member

@simon-donike simon-donike left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All tests pass (113 existing + 13 new deployment tests). Code reviewed — clean structure, good error handling, dry-run support, no regressions. Approved.

@simon-donike simon-donike merged commit 59b995c into ESAOpenSR:main Apr 28, 2026
@simon-donike
Copy link
Copy Markdown
Member

While it was working locally, it threw some errors due to imports on the GH Actions test run. Fixed that, your changes are now live! Thanks again for your enhancements!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants