Skip to content

Latest commit

 

History

History
936 lines (658 loc) · 33.8 KB

changelog.rst

File metadata and controls

936 lines (658 loc) · 33.8 KB
Raw

Changelog

0.6 | 2018-11-20

SAR metadata

  • new standardized metadata fields orbitNumber_abs, orbitNumber_rel, cycleNumber and frameNumber for all SAR formats
  • customization of output file names with additional metadata fields (e.g. orbit numbers)

software configuration

  • pyroSAR configuration file handling: the paths to the SNAP and Gamma installation as well as relevant metadata directories are now registered in a configuration file config.ini, which is stored in a directory .pyrosar in the user home directory
  • improved SNAP installation verification: pyroSAR now performs a deeper check of the SNAP installation to make sure it is not mistaken with e.g. the Ubuntu package manager snap; relevant installation executables and directories are stored in the configuration file

general functionality

  • deeper integration of package spatialist: all the spatial file handling functionality that was part of pyroSAR is now part of package spatialist; now all the functionality is imported from spatialist and removed from pyroSAR
  • improved search for datasets processed by pyroSAR: new helper functions exist, which make it easier to search for datasets by metadata fields, which are internally searched for in the respective file names
  • introduced gamma function parser: these new tools search for a GAMMA_HOME environment variable and, if found, parse Python functions from the docstring of respective command line tools; for this, new Python scripts are created, which are stored alongside the configuration file in the user home directory; this way users can easily use Python functions with named parameters instead of the positional arguments of the Gamma command line tools
  • improved documentation

Open Data Cube Export

functionality to export processed datasets directly to an Open Data Cube: it is now possible to create Open Data Cube product YML files as well as YML files for data indexing and ingestion into this product; pyroSAR also internally checks for compatibility of a particular dataset with the target product; this way, the resulting files can easily be passed to the Open Data Cube command line tools several bug fixes

SNAP API

improved SNAP processing workflow node linking: it is now possible to add a node also before an existing one, instead of just after it

Python package integrity

  • add trove classifiers for supported operating systems and MIT license for easier online search
  • exchange http with https for all URLs that support it

0.7 | 2019-01-03

several changes to the functioning of the Gamma command API

GAMMA API

processing

  • :func:`pyroSAR.gamma.geocode`:
    • optionally write all Gamma commands to shellscript
    • newly introduced choice of normalization method
    • changed normalization default approach
  • :func:`pyroSAR.gamma.process`:
    • new parameter logfile to specify a logfile instead of just a directory with automated file naming
    • new parameter shellscript to write the executed command to a shell script protocol

command parser

  • add parameters outdir and shellscript to parsed functions
  • extensive improvement to accurately parse more commands
  • add parameter inlist to some commands, which require interactive input via stdin

general

  • several bug fixes
  • extended documentation
  • make use of parsed command functions internally
  • enable passing logpath, outdir and shellscript to all parsed functions via additional parameters for other convenience functions

0.8 | 2019-02-11

Auxiliary Data Handling

GAMMA API

SNAP API

0.9 | 2019-06-15

Drivers

SNAP API

  • function :func:`pyroSAR.snap.util.geocode`:
    • option to export DEM via parameter export_extra
    • added Sentinel-1 ThermalNoiseRemoval node via new parameter removeS1ThermalNoise
    • added Multilook node which is executed to approximate the target resolution if necessary (currently only for Sentinel-1 since metadata entries incidence and image_geometry are required)
    • new parameter groupsize to split workflows into several groups, which are executed separately with intermediate products written to disk. This increases processing speed
    • simplified internal node parametrization for easier use in future functions
    • fail if no POE orbit state vector file is found
    • Terrain-Flattening:
      • added additional parameters additionalOverlap and oversamplingMultiple
      • use bilinear instead of bicubic interpolation
    • Remove-GRD-Border-Noise: decrease borderLimit from 1000 to 500 (SNAP default)
    • new parameter gpt_exceptions to execute workflows containing specific nodes with different GPT versions than the default one
    • automatically remove node parameters on GPT fail and re-run the modified workflow; this is relevant if a node is executed in an older GPT version (e.g. via parameter gpt_exceptions), which does not accept parameters which were introduced in later GPT versions (e.g. those described above for node Terrain-Flattening)
    • disable/enable terrain flattening via new parameter terrainFlattening
    • optionally return workflow filename with new parameter returnWF
    • execute custom pyroSAR S1 GRD border noise removal (see :func:`pyroSAR.S1.removeGRDBorderNoise`)
    • new parameters demResamplingMethod and imgResamplingMethod

GAMMA API

  • SRTM Tools renamed to DEM Tools
    • function :func:`pyroSAR.gamma.dem.dem_autocreate`:
      • define arbitrary output CRS and resolution via new parameters t_srs and tr
      • optionally perform geoid to ellipsoid conversion in either GDAL or GAMMA via new parameter geoid_mode
  • function :func:`pyroSAR.gamma.geocode`:
    • removed multiplication of backscatter with cosine of incident angle via command lin_comb
    • fixed bug in writing correct nodata values to ancillary products defined via parameter export_extra
    • changed default of parameter func_geoback from 2 to 1 (GAMMA default)
  • function :func:`pyroSAR.gamma.correctOSV`:
    • fixed bug in using the first OSV file in a directory for correcting an image, which resulted in S1B files being corrected with S1A OSV files. This occasionally resulted in errors of no DEM overlap while processing S1B scenes
  • fixed bug in treating GAMMA image pixel coordinates as top left instead of pixel center. This is relevant for writing ENVI HDR files for GAMMA images via function :func:`pyroSAR.gamma.par2hdr` resulting in the image to be shifted by 1/2 pixel to Southeast

Command Parser

  • compatibility with GAMMA version released in November 2018
  • delete parsed modules if environment variable GAMMA_HOME was reset causing them to be re-parsed with the new version on module import

general functionality

Auxiliary Data Handling

0.9.1 | 2019-07-05

Auxiliary Data Handling

GAMMA API

SNAP API

0.10 | 2019-12-06

Drivers

GAMMA API

SNAP API

  • workflow parsing
    • improved output XML for better display in SNAP GUI
    • support for nodes with multiple input scenes, e.g. SliceAssembly
  • SAR processor (function :func:`~pyroSAR.snap.auxil.gpt`)
    • write Sentinel-1 manifest.safe with processing results
    • two methods for border noise removal: ESA and pyroSAR via new parameter removeS1BorderNoiseMethod
  • function :func:`pyroSAR.snap.util.geocode`
    • optional speckle filtering with new parameter speckleFilter
    • choose the output backscatter reference area (beta0/gamma0/sigma0) with new parameter refarea
    • default of parameter groupsize changed to 1
    • internally download S1 OSV files
    • internally download SNAP's EGM96 geoid to WGS84 ellipsoid DEM conversion lookup table via new function :func:`pyroSAR.snap.auxil.get_egm96_lookup`
    • support for multi-scene SliceAssembly; can be invoke by passing a list of scenes to parameter infile
    • new parameter removeS1BorderNoiseMethod
    • new parameter gpt_args to pass additional arguments to the GPT call

Datacube Tools

Auxiliary Data Handling

  • OSV download functionality (class :class:`pyroSAR.S1.OSV`)
    • made definition of OSV download directory optional; default is SNAP's auxdata directory
    • organization of downloaded files into SNAP's auxdata structure:
      • compression to zip
      • sort files into subdirs for sensor, year, month
    • removed method :meth:`~pyroSAR.S1.OSV.update`

Ancillary Tools

general

  • bug fixes and documentation improvements

0.10.1 | 2019-12-12

GAMMA API

0.11 | 2020-05-29

Drivers

SNAP API

  • arbitrary nodes can now be parsed. Before, only a small selection of nodes (those used by function :func:`~pyroSAR.snap.util.geocode`) were available. Now, any node and its default parametrization can be parsed to XML from the GPT documentation by internally calling e.g.:

    gpt Terrain-Flattening -h

    The parsed XML representation is saved for faster future reuse. See function :func:`~pyroSAR.snap.auxil.parse_node` for details. In all cases the standard SNAP file suffix is used for output products, e.g. _TF for Terrain-Flattening.

  • multi-source nodes like SliceAssembly now take any number of sources, not just two. See class :class:`~pyroSAR.snap.auxil.Node`.

  • function :func:`pyroSAR.snap.util.geocode`:

    • new argument nodataValueAtSea to decide whether sea areas are masked out. Depends on the quality of the sea mask in the input DEM.
    • automatically download required Sentinel-1 Orbit State Vector (OSV) files.
    • new argument allow_RES_OSV to decide whether to allow usage of the less accurate Sentinel-1 RES OSV files in case the POE file is not available yet.
    • new argument demName to choose the type of the auto-downloaded DEM.

Auxiliary Data Handling

general

  • bug fixes, new automated tests, documentation improvements

0.11.1 | 2020-07-17

  • bug fixes

GAMMA API

0.12 | 2021-02-19

Drivers

GAMMA API

SNAP API

  • function :func:`pyroSAR.snap.util.geocode`:
    • enable grid alignment with new arguments alignToStandardGrid, standardGridOriginX and standardGridOriginY
    • new argument tmpdir to choose the location of temporarily created files
    • bug fixes
  • function :func:`pyroSAR.snap.auxil.gpt`:
    • perform custom pyroSAR S1 GRD border noise removal only if IPF<2.9

Auxiliary Data Handling

0.12.1 | 2021-03-09

SNAP API

  • function :func:`pyroSAR.snap.util.geocode`:
    • output both sigma0 and gamma0 via argument refarea
    • new export_extra option 'layoverShadowMask'
  • numerous bug fixes and API improvements

Auxiliary Data Handling

0.13 | 2021-09-10

Drivers

GAMMA API

SNAP API

Auxiliary Data Handling

general

0.14.0 | 2021-10-12

Drivers

GAMMA API

SNAP API

Auxiliary Data Handling

general

  • replaced http URLs with https where applicable
  • improved documentation

0.15.0 | 2022-01-04

Drivers

GAMMA API

SNAP API

  • function :func:`pyroSAR.snap.util.geocode`:
    • enabled SLC processing
    • enable processing of sigma nought RTC
    • new export_extra argument gammaSigmaRatio
    • simplified workflow by writing layover-shadow mask directly from Terrain-Correction
    • changed processing node sequence:
      • was: Read->ThermalNoiseRemoval->SliceAssembly->Remove-GRD-Border-Noise->Calibration
      • is: Read->Remove-GRD-Border-Noise->Calibration->ThermalNoiseRemoval->SliceAssembly
    • new output image naming scheme, e.g.
      • S1A__IW___A_20210914T191350_VV_gamma0-rtc.tif
      • S1A__IW___A_20210914T191350_VH_sigma0-elp.tif
  • function :func:`pyroSAR.snap.auxil.gpt`:
    • removed argument multisource
    • added argument tmpdir

Auxiliary Data Handling

  • function :func:`pyroSAR.auxdata.dem_autoload`:
    • updated version of 'Copernicus 10m EEA DEM' from '2020_1' to '2021_1'
    • new DEM options:
      • 'Copernicus 30m Global DEM II'
      • 'Copernicus 90m Global DEM II'

general

  • compatibility with sqlalchemy>=1.4

0.15.1 | 2022-01-07

general

  • bug fixes

0.16.0 | 2022-03-03

Drivers

Auxiliary Data Handling

SNAP API

GAMMA API

general

  • correction of multi-look factor computation. Before: approximate target pixel spacing but never exceed it. Now: first best approximate the azimuth spacing as close as possible (even if this means exceeding the target spacing) and then choose the range looks to approximate a square pixel as close as possible. API changes:

0.16.1 | 2022-03-07

Auxiliary Data Handling

0.16.2 | 2022-03-14

SNAP API

0.16.3 | 2022-03-23

SNAP API

0.17.0 | 2022-05-30

SNAP API

GAMMA API

general

  • full support for Sentinel-1 stripmap mode; renamed SM naming pattern to S1..S6 to differentiate different beams
  • bug fixes

0.17.2 | 2022-06-23

Auxiliary Data Handling

  • function :func:`pyroSAR.auxdata.dem_create`:
    • use maximum possible value of dtype (e.g. 255 for unit8) instead of -32767.0 if the nodata value cannot be read from the source file
    • always use the same value for source and destination nodata

0.17.3 | 2022-07-03

Auxiliary Data Handling

  • function :func:`pyroSAR.auxdata.dem_create`:
    • In case the nodata value could not be read from the source file, the function used to define a value itself, which is prone to errors. This value now needs to be set by a user via new argument nodata if it cannot be read from the source file.
    • bug fix: no longer try to download 'Copernicus 30m Global DEM' or 'Copernicus 90m Global DEM' tiles that don't exist.
  • function :func:`pyroSAR.auxdata.dem_autoload`:
    • new argument dst_nodata. This can be used to temporarily override the native nodata value for extrapolation of ocean areas (in combination with hide_nodata=True).

0.18.0 | 2022-08-24

Drivers

GAMMA API

SNAP API

0.19.0 | 2022-09-28

Drivers

SNAP API

Auxiliary Data Handling

  • function :func:`pyroSAR.auxdata.dem_autoload`:
    • new argument crop to optionally return the full extent of all overlapping DEM tiles
    • added download status print messages
    • download and modify a Copernicus DEM index file for future reuse; this removes the need to search the FTP server for files and thus greatly accelerates the process of collecting all files overlapping with the AOI

0.20.0 | 2022-12-27

Drivers

SNAP API

Auxiliary Data Handling

  • function :func:`pyroSAR.auxdata.dem_autoload`:
    • no longer require DEM tiles for creating a mosaic to address ocean cases
    • simplified handling and removed arguments nodata, dst_nodata and hide_nodata
    • the DEM option 'Copernicus 30m Global DEM' now also includes several auxiliary layers that can be downloaded automatically
    • the URLs for DEM options 'SRTM 3Sec' and 'TDX90m' have been updated
  • function :func:`pyroSAR.auxdata.dem_create`:

0.21.0 | 2023-05-11

Drivers

GAMMA API

  • the LAT module is no longer needed: new pyroSAR-internal implementations can be used if the module is missing (concerns commands product, ratio and linear_to_dB)
  • improved backwards compatibility:
    • use multi_look_ScanSAR if present and multi_S1_TOPS otherwise
    • use gc_map2 if possible (present and with all needed arguments) and gc_map otherwise
    • addressed the case where gc_map does not have an argument OFF_par
  • function gamma.pixel_area_wrap: new argument exist_ok (this function will be made more visible in the documentation once matured)
  • bug fixes:

SNAP API

  • function :func:`pyroSAR.snap.util.geocode`: new argument dem_oversampling_multiple with default 2 to increase the DEM oversampling factor for terrain flattening
  • function :func:`pyroSAR.snap.auxil.erode_edges`:
    • do not attempt to perform erosion if the image only contains nodata (this might happen if only parts of the image were geocoded)
    • make sure that a backscatter image is used for erosion (auxiliary data like the local incidence angle often has a larger valid data extent and using such image for erosion would thus not properly erode edges of the backscatter images; additionally this has the effect that all images will have the same valid data extent after erosion)
    • the written mask files (delineating valid data and nodata after erosion of the backscatter image and used for masking all other images) are now compressed (deflate) so that data volume is decreased significantly

Auxiliary Data Handling