- new standardized metadata fields orbitNumber_abs, orbitNumber_rel, cycleNumber and frameNumber for all SAR formats
- customization of output file names with additional metadata fields (e.g. orbit numbers)
- pyroSAR configuration file handling: the paths to the SNAP and Gamma installation as well as relevant metadata directories are now registered in a configuration file config.ini, which is stored in a directory .pyrosar in the user home directory
- improved SNAP installation verification: pyroSAR now performs a deeper check of the SNAP installation to make sure it is not mistaken with e.g. the Ubuntu package manager snap; relevant installation executables and directories are stored in the configuration file
- deeper integration of package spatialist: all the spatial file handling functionality that was part of pyroSAR is now part of package spatialist; now all the functionality is imported from spatialist and removed from pyroSAR
- improved search for datasets processed by pyroSAR: new helper functions exist, which make it easier to search for datasets by metadata fields, which are internally searched for in the respective file names
- introduced gamma function parser: these new tools search for a GAMMA_HOME environment variable and, if found, parse Python functions from the docstring of respective command line tools; for this, new Python scripts are created, which are stored alongside the configuration file in the user home directory; this way users can easily use Python functions with named parameters instead of the positional arguments of the Gamma command line tools
- improved documentation
functionality to export processed datasets directly to an Open Data Cube: it is now possible to create Open Data Cube product YML files as well as YML files for data indexing and ingestion into this product; pyroSAR also internally checks for compatibility of a particular dataset with the target product; this way, the resulting files can easily be passed to the Open Data Cube command line tools several bug fixes
improved SNAP processing workflow node linking: it is now possible to add a node also before an existing one, instead of just after it
- add trove classifiers for supported operating systems and MIT license for easier online search
- exchange http with https for all URLs that support it
several changes to the functioning of the Gamma command API
pyroSAR.gamma.geocode
:- optionally write all Gamma commands to shellscript
- newly introduced choice of normalization method
- changed normalization default approach
pyroSAR.gamma.process
:- new parameter logfile to specify a logfile instead of just a directory with automated file naming
- new parameter shellscript to write the executed command to a shell script protocol
- add parameters outdir and shellscript to parsed functions
- extensive improvement to accurately parse more commands
- add parameter inlist to some commands, which require interactive input via stdin
- several bug fixes
- extended documentation
- make use of parsed command functions internally
- enable passing logpath, outdir and shellscript to all parsed functions via additional parameters for other convenience functions
- new module auxdata with function
pyroSAR.auxdata.dem_autoload
to automatically download tiles of different DEM types overlapping with given geometries - class
pyroSAR.S1.OSV
: reduced search time for new RES orbit state vector files; included more meaningful status messages
- new function
pyroSAR.gamma.srtm.dem_autocreate
to automatically create DEMs in Gamma format from the output of functionpyroSAR.auxdata.dem_autoload
- improved writing of ENVI HDR files from class
pyroSAR.gamma.ISPPar
- class
pyroSAR.gamma.UTM
: improved to work with newer Gamma versions - function
pyroSAR.gamma.geocode
:- improved documentation
- clarified code for better readability
- more consistent naming scheme for all temporarily written files
- export temporarily written files (e.g. local incidence angle) via new parameter export_extra
- additional parametrization tests to ensure best processing result
- changed default of parameter func_interp to 2 to work best with default of parameter normalization_method (see documentation of Gamma command pixel_area)
- function
pyroSAR.snap.util.geocode
:- export temporarily written files (e.g. local incidence angle) via new parameter export_extra
pyroSAR.drivers.SAFE
: read heading angle, incident angle and image geometry (e.g. Ground Range) from metadatapyroSAR.drivers.Archive
: improved cross-compatibility with Python2 and Python3
- function
pyroSAR.snap.util.geocode
:- option to export DEM via parameter export_extra
- added Sentinel-1 ThermalNoiseRemoval node via new parameter removeS1ThermalNoise
- added Multilook node which is executed to approximate the target resolution if necessary (currently only for Sentinel-1 since metadata entries incidence and image_geometry are required)
- new parameter groupsize to split workflows into several groups, which are executed separately with intermediate products written to disk. This increases processing speed
- simplified internal node parametrization for easier use in future functions
- fail if no POE orbit state vector file is found
- `Terrain-Flattening`:
- added additional parameters additionalOverlap and oversamplingMultiple
- use bilinear instead of bicubic interpolation
- `Remove-GRD-Border-Noise`: decrease borderLimit from 1000 to 500 (SNAP default)
- new parameter gpt_exceptions to execute workflows containing specific nodes with different GPT versions than the default one
- automatically remove node parameters on GPT fail and re-run the modified workflow; this is relevant if a node is executed in an older GPT version (e.g. via parameter gpt_exceptions), which does not accept parameters which were introduced in later GPT versions (e.g. those described above for node Terrain-Flattening)
- disable/enable terrain flattening via new parameter terrainFlattening
- optionally return workflow filename with new parameter returnWF
- execute custom pyroSAR S1 GRD border noise removal (see
pyroSAR.S1.removeGRDBorderNoise
) - new parameters demResamplingMethod and imgResamplingMethod
- SRTM Tools renamed to DEM Tools
- function
pyroSAR.gamma.dem.dem_autocreate
:- define arbitrary output CRS and resolution via new parameters t_srs and tr
- optionally perform geoid to ellipsoid conversion in either GDAL or GAMMA via new parameter geoid_mode
- function
- function
pyroSAR.gamma.geocode
:- removed multiplication of backscatter with cosine of incident angle via command lin_comb
- fixed bug in writing correct nodata values to ancillary products defined via parameter export_extra
- changed default of parameter func_geoback from 2 to 1 (GAMMA default)
- function
pyroSAR.gamma.correctOSV
:- fixed bug in using the first OSV file in a directory for correcting an image, which resulted in S1B files being corrected with S1A OSV files. This occasionally resulted in errors of no DEM overlap while processing S1B scenes
- fixed bug in treating GAMMA image pixel coordinates as top left instead of pixel center. This is relevant for writing ENVI HDR files for GAMMA images via function
pyroSAR.gamma.par2hdr
resulting in the image to be shifted by 1/2 pixel to Southeast
- compatibility with GAMMA version released in November 2018
- delete parsed modules if environment variable GAMMA_HOME was reset causing them to be re-parsed with the new version on module import
- new function
pyroSAR.ancillary.multilook_factors
to compute factors depending on image geometry and target resolution pyroSAR.S1.removeGRDBorderNoise
: reached Python3 compatibility
- new function
pyroSAR.auxdata.dem_create
for convenient creation of DEM mosaics as downloaded bypyroSAR.auxdata.dem_autoload
- function
pyroSAR.auxdata.dem_autoload
: download 1 degree tiles instead of 5 degree tiles - class
pyroSAR.S1.OSV
:- download files specific to the Sentinel-1 sensor (S1A/S1B) instead of all matching the acquisition time
- improved time span search, which occasionally resulted in missing OSV files
- function
pyroSAR.auxdata.dem_create
: new parameter resampling_method
- function
pyroSAR.gamma.dem.dem_autocreate
: new parameter resampling_method
- function
pyroSAR.snap.util.geocode
: fixed typo of parameter removeS1BorderNoise
- method
~pyroSAR.drivers.ID.bbox
: choose the output vector file format via new parameter driver or by using one of spatialist's supported file name extensions (seespatialist.vector.Vector.write
) pyroSAR.drivers.SAFE
- new method
~pyroSAR.drivers.SAFE.quicklook
for writing KMZ quicklooks - method
~pyroSAR.drivers.SAFE.getOSV
: renamed parameter outdir to osvdir
- new method
pyroSAR.drivers.Archive
: remove scenes from the database if they cannot be found at their file location. This is performed at each initialization of an Archive object.
- new parameter basename_extensions for adding extra metadata fields to output image names; affects:
pyroSAR.gamma.convert2gamma
pyroSAR.gamma.geocode
pyroSAR.gamma.correctOSV
: make use of OSV files in SNAP's auxdata structurepyroSAR.gamma.geocode
: made border nose removal optional with new parameter removeS1BorderNoise
- workflow parsing
- improved output XML for better display in SNAP GUI
- support for nodes with multiple input scenes, e.g. SliceAssembly
- SAR processor (function
~pyroSAR.snap.auxil.gpt
)- write Sentinel-1 manifest.safe with processing results
- two methods for border noise removal: ESA and pyroSAR via new parameter removeS1BorderNoiseMethod
- function
pyroSAR.snap.util.geocode
- optional speckle filtering with new parameter speckleFilter
- choose the output backscatter reference area (beta0/gamma0/sigma0) with new parameter refarea
- default of parameter groupsize changed to 1
- internally download S1 OSV files
- internally download SNAP's EGM96 geoid to WGS84 ellipsoid DEM conversion lookup table via new function
pyroSAR.snap.auxil.get_egm96_lookup
- support for multi-scene SliceAssembly; can be invoke by passing a list of scenes to parameter infile
- new parameter removeS1BorderNoiseMethod
- new parameter gpt_args to pass additional arguments to the GPT call
pyroSAR.datacube_util.Product.export_ingestion_yml
: new parameter chunking
- OSV download functionality (class
pyroSAR.S1.OSV
)- made definition of OSV download directory optional; default is SNAP's auxdata directory
- organization of downloaded files into SNAP's auxdata structure:
- compression to zip
- sort files into subdirs for sensor, year, month
- removed method
~pyroSAR.S1.OSV.update
pyroSAR.ancillary.parse_datasetname
- support for datasets in NetCDF format
- enable parsing of ancillary products like local incidence angle (*inc_geo.tif)
pyroSAR.ancillary.find_datasets
: new parameters start and stop for time filtering
- bug fixes and documentation improvements
Command API <gamma-command-api>
compatibility with GAMMA version 20191203
pyroSAR.drivers.Archive
: completely restructured to use the SQLAlchemy Object Relational Mapper (ORM). This makes it possible to switch between SQLite+Spatialite and PostgreSQL+PostGIS database backends.pyroSAR.drivers.SAFE.getOSV
: new argument returnMatch to also return the name of an OSV file instead of just downloading it.
arbitrary nodes can now be parsed. Before, only a small selection of nodes (those used by function
~pyroSAR.snap.util.geocode
) were available. Now, any node and its default parametrization can be parsed to XML from the GPT documentation by internally calling e.g.:gpt Terrain-Flattening -h
The parsed XML representation is saved for faster future reuse. See function
~pyroSAR.snap.auxil.parse_node
for details. In all cases the standard SNAP file suffix is used for output products, e.g. _TF for Terrain-Flattening.- multi-source nodes like SliceAssembly now take any number of sources, not just two. See class
~pyroSAR.snap.auxil.Node
. - function
pyroSAR.snap.util.geocode
:- new argument nodataValueAtSea to decide whether sea areas are masked out. Depends on the quality of the sea mask in the input DEM.
- automatically download required Sentinel-1 Orbit State Vector (OSV) files.
- new argument allow_RES_OSV to decide whether to allow usage of the less accurate Sentinel-1 RES OSV files in case the POE file is not available yet.
- new argument demName to choose the type of the auto-downloaded DEM.
- class
pyroSAR.S1.OSV
:- removed progressbar from method
~pyroSAR.S1.OSV.catch
and made it optional in method~pyroSAR.S1.OSV.retrieve
with new argument pbar
- removed progressbar from method
- bug fixes, new automated tests, documentation improvements
- bug fixes
Command API <gamma-command-api>
compatibility with GAMMA version 20200713
pyroSAR.drivers.Archive
:- new argument cleanup to automatically remove missing scenes from database on initialization
- method
~pyroSAR.drivers.Archive.insert
: improved insertion speed - method
~pyroSAR.drivers.Archive.select_duplicates
: new argument value - method
~pyroSAR.drivers.Archive.get_colnames
: new argument table to get column names from arbitrary tables, not just the main data table - method
~pyroSAR.drivers.Archive.drop_element
: option to remove scene from data and duplicates tables simultaneously by removing argument table and adding argument with_duplicates - method
~pyroSAR.drivers.Archive.drop_table
:- new argument verbose
- remove arbitrary tables, not just data and duplicates
- method
~pyroSAR.drivers.Archive.drop_database
: replaced by new functionpyroSAR.drivers.drop_archive
- new method
~pyroSAR.drivers.Archive.add_tables
to add custom tables to a database - bug fixes
pyroSAR.drivers.CEOS_PSR
:- added support for ALOS-1 PALSAR
- added basic support for Level 1.0 data
pyroSAR.drivers.SAFE
:- method
~pyroSAR.drivers.SAFE.getOSV
: new argument useLocal to not search online if local matching files are found
- method
Command API <gamma-command-api>
compatibility with GAMMA version 20201216- function
pyroSAR.gamma.convert2gamma
:- renamed argument S1_noiseremoval to S1_tnr (thermal noise removal)
- new argument S1_bnr (border noise removal)
- function
pyroSAR.gamma.geocode
:- new default
removeS1BorderNoiseMethod='gamma'
- renamed argument tempdir to tmpdir
- new default
- function
pyroSAR.snap.util.geocode
:- enable grid alignment with new arguments alignToStandardGrid, standardGridOriginX and standardGridOriginY
- new argument tmpdir to choose the location of temporarily created files
- bug fixes
- function
pyroSAR.snap.auxil.gpt
:- perform custom pyroSAR S1 GRD border noise removal only if IPF<2.9
- function
pyroSAR.auxdata.dem_autoload
: return None if a VRT was defined
- function
pyroSAR.snap.util.geocode
:- output both sigma0 and gamma0 via argument refarea
- new export_extra option 'layoverShadowMask'
- numerous bug fixes and API improvements
- class
pyroSAR.S1.OSV
:- download files from https://scihub.copernicus.eu/gnss
- new class
pyroSAR.drivers.EORC_PSR
- new argument exist_ok for ID object unpack methods to enable reuse of already unpacked scenes
pyroSAR.drivers.SAFE.getOSV
: new argument url_option to choose between different download URLspyroSAR.drivers.SAFE
align coordinate sorting of attribute meta['coordinates'] with CRS descriptionpyroSAR.drivers.identify_many
: disable progressbar by default
- adaptations to enable processing of
~pyroSAR.drivers.EORC_PSR
data:pyroSAR.gamma.calibrate
pyroSAR.gamma.convert2gamma
pyroSAR.gamma.geocode
pyroSAR.gamma.geocode
:- experimental optional refinement of the geocoding lookup table with new argument refine_lut
- removed arguments normalization_method, func_interp, removeS1BorderNoise, sarSimCC
- limit radiometric normalization to RTC correction method
- simplify and improve computation of RTC contribution area
- file suffices pan and norm have been replaced with gamma0-rtc
- argument export_extra options:
- removed pix_geo
- renamed pix_fine to pix_ratio
- added pix_area_sigma0, pix_area_sigma0_geo, pix_area_gamma0_geo, gs_ratio , gs_ratio_geo, pix_ratio_geo
- use a dedicated temporary directory to unpack the scene and write GAMMA files so that they are separated (the GAMMA files used to be written to the unpacked scene's directory)
- enable multiple scenes as input so that they can be mosaiced in SAR geometry before geocoding
pyroSAR.gamma.correctOSV
: new argument directorypyroSAR.gamma.multilook
: new argument exist_okpyroSAR.gamma.convert2gamma
: new argument exist_ok- function
pyroSAR.gamma.dem.dem_autocreate
:- do not apply an extent buffer by default
- allow geometry in arbitrary CRS
- function
pyroSAR.snap.util.geocode
:- new export_extra option scatteringArea
- extended support for BandMaths operator
- method
pyroSAR.S1.OSV.catch
: new argument url_option with two download URLs to choose from - function
pyroSAR.auxdata.dem_autoload
:- added new DEM option GETASSE30
- align pixels of subsetted VRT with original tiles
- function
pyroSAR.auxdata.dem_create
:- new argument outputBounds
replaced print messages with logging. This made the verbose argument that was used by several functions and methods obsolete; affects the following:
pyroSAR.drivers.identify_many
: replaced by argument pbarpyroSAR.drivers.Archive.add_tables
: removedpyroSAR.drivers.Archive.drop_table
: removedpyroSAR.drivers.Archive.insert
: replaced by argument pbarpyroSAR.drivers.Archive.import_outdated
: removedpyroSAR.drivers.Archive.move
: replaced by argument pbarpyroSAR.drivers.Archive.select
: removedpyroSAR.snap.auxil.execute
: removed
See section
Logging </general/logging>
for details.
- raise more appropriate errors (c430c59)
pyroSAR.drivers.findfiles
: removed (functionality contained inpyroSAR.drivers.ID.findfiles
, now making use ofspatialist.ancillary.finder
)pyroSAR.drivers.Archive.select
:- show progressbar for scene identification if
pbar=True
- enabled input of
~datetime.datetime
objects for argumentsmindate
andmaxdate
- show progressbar for scene identification if
pyroSAR.drivers.identify_many
: issue a warning when a file cannot be accessed (instead of raising aPermissionError
)
pyroSAR.gamma.dem.dem_autocreate
: support for new DEM options provided bypyroSAR.auxdata.dem_autoload
pyroSAR.snap.auxil.get_egm96_lookup
removed in favor of new functionpyroSAR.auxdata.get_egm_lookup
- method
pyroSAR.S1.OSV.retrieve
: thread-safe writing of orbit files - new function
pyroSAR.auxdata.get_egm_lookup
- function
pyroSAR.auxdata.dem_create
- new geoid option 'EGM2008'
- make use of
~pyroSAR.auxdata.get_egm_lookup
for auto-download of EGM lookup files - several bug fixes related to vertical CRS transformation
- bug fix for target pixel alignment
- function
pyroSAR.auxdata.dem_autoload
: new DEM options:- 'Copernicus 10m EEA DEM'
- 'Copernicus 30m Global DEM'
- 'Copernicus 90m Global DEM'
- replaced http URLs with https where applicable
- improved documentation
pyroSAR.drivers.ID.geometry
: new method
Command API <gamma-command-api>
compatibility with GAMMA version 20211208- renamed argument resolution to spacing; affects:
pyroSAR.gamma.geocode
pyroSAR.gamma.ovs
pyroSAR.gamma.multilook
- function
pyroSAR.gamma.calibrate
- removed argument replace
- added argument return_fnames
- function
pyroSAR.gamma.convert2gamma
- added argument return_fnames
- function
pyroSAR.gamma.multilook
- pass multiple Sentinel-1 sub-swaths to argument infile which are then combined into a single MLI using GAMMA command isp.multi_look_ScanSAR
- class
pyroSAR.gamma.ISPPar
:- new object attribute filetype with possible values 'isp' and 'dem'
- function
pyroSAR.snap.util.geocode
:- enabled SLC processing
- enable processing of sigma nought RTC
- new export_extra argument gammaSigmaRatio
- simplified workflow by writing layover-shadow mask directly from Terrain-Correction
- changed processing node sequence:
- was: Read->ThermalNoiseRemoval->SliceAssembly->Remove-GRD-Border-Noise->Calibration
- is: Read->Remove-GRD-Border-Noise->Calibration->ThermalNoiseRemoval->SliceAssembly
- new output image naming scheme, e.g.
- S1A__IW___A_20210914T191350_VV_gamma0-rtc.tif
- S1A__IW___A_20210914T191350_VH_sigma0-elp.tif
- function
pyroSAR.snap.auxil.gpt
:- removed argument multisource
- added argument tmpdir
- function
pyroSAR.auxdata.dem_autoload
:- updated version of 'Copernicus 10m EEA DEM' from '2020_1' to '2021_1'
- new DEM options:
- 'Copernicus 30m Global DEM II'
- 'Copernicus 90m Global DEM II'
- compatibility with sqlalchemy>=1.4