Merge pull request #155 from GeoscienceAustralia/issue_153

Issue 153 branch

Author: Matt Garthwaite

AUTHORS.rst

@@ -5,18 +5,18 @@ Credits
 Development Lead
 ----------------
 
-* Geoscience Australia InSAR team
+* Geoscience Australia InSAR team <insar@ga.gov.au>
 
 PyRate has been developed by `Geoscience Australia <http://www.ga.gov.au>`__
 and the `National Computational Infrastructure <http://nci.org.au/>`__.
 
 Contributors
 ------------
 
-* Sudipta Basak <basaks@gmail.com>
-* Ben Davies
-* Matt Garthwaite <Matt.Garthwaite@ga.gov.au>
-* Simon Knapp
-* Negin Moghaddam
-* Vanessa Newey
-* Garrick Paskos
+* Sudipta Basak (GA)
+* Ben Davies (NCI)
+* Matt Garthwaite (GA)
+* Simon Knapp (GA)
+* Negin Moghaddam (GA)
+* Vanessa Newey (GA)
+* Garrick Paskos (GA)

conda.md

@@ -1,47 +1,43 @@
-Instructions for setting up up an anaconda virtualenv on the linux server 'rhe-compute1' at GA.
+Instructions for setting up an Anaconda virtual environment on a typical Red Hat linux server.
 
-## Download the conda installer
+## Install Anaconda:
 
-For `rhe-compute` [download this version] (https://repo.continuum.io/archive/Anaconda2-4.1.1-Linux-x86.sh). 
+For 64bit linux download this version of the conda installer: https://repo.continuum.io/archive/Anaconda2-4.1.1-Linux-x86.sh
 
-## Install anaconda:
-
-On 64bit linux:
+Run the installer using the linux bash interpreter:
 
     bash Anaconda2-4.1.1-Linux-x86.sh
 
 Accept all default options. This will install anaconda in the `~/anaconda2` directory.
 
 ## Set up proxys for pip (only applicable if working behind proxy):
 
-    export http_proxy=http://proxy.ga.gov.au:8080
-    export https_proxy=http://proxy.ga.gov.au:8080
+    export http_proxy=<proxy_server>:<port>
+    export https_proxy=<proxy_server>:<port>
 
 ## Set up `~/.condarc` with proxy pass information (only applicable if working behind proxy):
 
 For conda installer to work you need to create a `~/.condarc` file with the following proxy-pass information:
 
     proxy_servers:
-        http: http://proxy.ga.gov.au:8080
-        https: http://proxy.ga.gov.au:8080
+        http: <proxy_server>:<port>
+        https: <proxy_server>:<port>
         
-## Clone the repo:
-
-The rest of the instructions assume that you cloned the `PyRate` Git repository in your home directory. 
+## Clone the PyRate repo:
 
-Clone the repo:    
+The rest of the instructions assume that you cloned the `PyRate` Git repository in your home directory: 
 
+    cd ~
     git clone git@github.com:GeoscienceAustralia/PyRate.git
 
-This will prompt for your rsa key passphrase. If you have access to `PyRate` repo and once you have entered the passphrase, `PyRate` will clone in your current directory. 
+This will prompt for your rsa key passphrase. If you have access to `PyRate` repo and once you have entered the passphrase, `PyRate` will clone in your current (home) directory. 
 
 
-## Set up PYRATEPATH
-You need to add the directory containing the `pyrate` folder to the `PYTHONPATH` environment variable.
+## Set up envrionment variables
 
-The environment variable `PYRATEPATH` needs to point to the folder where you put the test data. This is how I set up my environment variable:
+You need to add the directory containing the `pyrate/` directory to the `PYTHONPATH` environment variable, and the environment variable `PYRATEPATH` needs to point to the directory containing the `tests/` directory:
 
-	export PYRATEPATH="/nas/users/u92456/unix/PyRate"
+	export PYRATEPATH="~/PyRate"
 	export PYTHONPATH=$PYRATEPATH/:$PYTHONPATH
 
 
@@ -50,21 +46,23 @@ The environment variable `PYRATEPATH` needs to point to the folder where you put
     source ~/anaconda2/bin/activate ~/anaconda2/
     conda install -c conda conda-env        
 
-The `conda-env` package enables `yml` based installation.
+The `conda-env` package enables YAML based installation.
 
 Note that using the `source activate` command only works cleanly when using the bash shell
 
 ## Create the `pyrate` environment
+
+The required packages for PyRate are listed in the environment.yml (YAML format) file:
 
-    conda env create -f /path/to/PyRate/environment.yml
+    conda env create -f $PYRATEPATH/environment.yml
 
 ## Activate the `pyrate` environment
 
     source ~/anaconda2/bin/activate pyrate
 
 ## Support for `pygrib`, a package used by `PyAPS` (skip this step if not using PyAPS)
 
-`pygrib` does not install properly in `redhat` systems, `rhe-compute1` being one of them. The way around is the following:
+`pygrib` does not install properly in `redhat` systems. The work around is the following:
 
     cp ~/anaconda2/envs/pyrate/lib/libpng16.so.16 ~/anaconda2/envs/pyrate/lib/libpng16.so.16.bk
     conda install libpng=1.2.50
@@ -80,16 +78,21 @@ Explanation of the previous three steps:
 
 ## Run `PyRate` tests
 
-To run the tests, use the following command inside the `PyRate` directory:
+To run the full suite of tests, use the following command inside the `PyRate` directory:
 
-	cd PyRate
+	cd ~/PyRate
 	py.test tests/
-	
+
+To run the tests for a particular module:
+
+	py.test tests/test_orbital.py
+
 ## Deactivate
-Once you are done using `PyRate` you could deactivate from the conda env using: 
+
+Once you are done using `PyRate` you could deactivate the conda environment using: 
 
     source deactivate
 
-## Back to main anaconda if you need to:
+## Back to main Anaconda environment if you need to:
 
-    source activate /home/user/anaconda2
+    source activate /home/user/anaconda2

environment.yml

@@ -11,7 +11,7 @@ dependencies:
 - hdf4
 - hdf5
 - ipython
-- matplotlib
+- matplotlib=1.5.1
 - networkx
 - numpy
 # - pygrib

environment.yml1

@@ -171,7 +171,7 @@ def get_epochs(ifgs):
     """
     Returns an EpochList derived from all given interferograms.
     """
-    log.info('Finding epochs')
+    log.info('Finding unique epochs in given interferogram network')
     if isinstance(ifgs, dict):
         ifgs = [v for v in ifgs.values() if isinstance(v, PrereadIfg)]
     combined = get_all_epochs(ifgs)

pyrate/algorithm.py

@@ -36,6 +36,8 @@
 PYRATE_INCIDENCE_DEGREES = 'INCIDENCE_DEGREES'
 PYRATE_INSAR_PROCESSOR = 'INSAR_PROCESSOR'
 PYRATE_APS_ERROR = 'APS_ERROR'
+PYRATE_MAXVAR = 'CVD_MAXVAR'
+PYRATE_ALPHA = 'CVD_ALPHA'
 MULTILOOKED = 'MULTILOOKED_IFG'
 ORIG = 'ORIGINAL_IFG'
 DEM = 'ORIGINAL_DEM'
@@ -49,7 +51,7 @@
 LINSAMP = 'LINEAR_RATE_SAMPLES'
 PYRATE_ORBITAL_ERROR = 'ORBITAL_ERROR'
 ORB_REMOVED = 'REMOVED'
-REF_PHASE = 'REFERENCE_PHASE'
+PYRATE_REF_PHASE = 'REFERENCE_PHASE'
 REF_PHASE_REMOVED = 'REMOVED'
 NAN_STATUS = 'NAN_STATUS'
 NAN_CONVERTED = 'CONVERTED'

pyrate/ifgconstants.py

@@ -400,8 +400,6 @@ def remove_orbital_error(ifgs, params, preread_ifgs=None):
     preread_ifgs: dict, optional
         dict containing information regarding MPI jobs
     """
-    log.info('Calculating orbital error correction')
-
     if not params[cf.ORBITAL_FIT]:
         log.info('Orbital correction not required')
         return
@@ -411,7 +409,8 @@ def remove_orbital_error(ifgs, params, preread_ifgs=None):
         _ = [preread_ifgs.pop(k) for k in ['gt', 'epochlist', 'md', 'wkt']]
         # perform some general error/sanity checks
         if mpiops.rank == 0:
-            _check_orbital_ifgs(preread_ifgs)
+            if _check_orbital_ifgs(preread_ifgs):
+                return # return if True condition returned
 
     ifg_paths = [i.data_path for i in ifgs] \
         if isinstance(ifgs[0], Ifg) else ifgs
@@ -441,15 +440,18 @@ def remove_orbital_error(ifgs, params, preread_ifgs=None):
 
 
 def _check_orbital_ifgs(preread_ifgs):  # pragma: no cover
-
+    """
+    Check if the correction has already been performed in a previous run
+    """
+    log.info('Checking Orbital error status')
     ifg_paths = sorted(preread_ifgs.keys())
     # preread_ifgs[i].metadata contains ifg metadata
     flags = [ifc.PYRATE_ORBITAL_ERROR in preread_ifgs[i].metadata
              for i in ifg_paths]
     if all(flags):
-        log.info('Skipped orbital correction, ifgs already corrected')
-        return
-    if (sum(flags) < len(flags)) and (sum(flags) > 0):
+        log.info('Skipped orbital corrections, ifgs already corrected')
+        return True
+    elif (sum(flags) < len(flags)) and (sum(flags) > 0):
         log.debug('Detected mix of corrected and uncorrected '
                   'orbital error in ifgs')
 
@@ -460,4 +462,6 @@ def _check_orbital_ifgs(preread_ifgs):  # pragma: no cover
                 msg = '{}: no orbital correction detected'.format(i)
             log.debug(msg)
             raise OrbitalError(msg)
-    log.info('Orbital error status checked')
+    else:
+        log.info('Calculating orbital corrections')
+        return False

pyrate/orbital.py

@@ -38,7 +38,7 @@ def estimate_ref_phase(ifgs, params, refpx, refpy):
         :ref_phs: reference phase correction
         :ifgs: reference phase data removed list of ifgs
     """
-    _validate_ifgs(ifgs)
+    check_ref_phs_ifgs(ifgs)
 
     # set reference phase as the average of the whole image (recommended)
     if params[cf.REF_EST_METHOD] == 1:
@@ -50,7 +50,7 @@ def estimate_ref_phase(ifgs, params, refpx, refpy):
         raise ReferencePhaseError('No such option. Use refest=1 or 2')
 
     for i in ifgs:
-        i.meta_data[ifc.REF_PHASE] = ifc.REF_PHASE_REMOVED
+        i.meta_data[ifc.PYRATE_REF_PHASE] = ifc.REF_PHASE_REMOVED
         i.write_modified_phase()
     return ref_phs, ifgs
 
@@ -135,43 +135,43 @@ def est_ref_phs_method1(phase_data, comp):
     """convenience function for ref phs estimate method 1 parallelisation"""
     ifgv = np.ravel(phase_data, order='F')
     ifgv[comp == 1] = np.nan
-    # reference phase
-    ref_ph = nanmedian(ifgv)
-    return ref_ph
+    return nanmedian(ifgv)
 
 
-def _validate_ifgs(ifgs):
+def check_ref_phs_ifgs(ifgs, preread_ifgs=None):
     """
-    make sure all ifgs have the same ref phase status
+    Function to check that the ref phase status of all ifgs are the same
     """
+    log.info('Checking status of reference phase estimation')
     if len(ifgs) < 2:
         raise ReferencePhaseError('Need to provide at least 2 ifgs')
-    flags = [i.dataset.GetMetadataItem(ifc.REF_PHASE) for i in ifgs]
-    if all(flags):
-        log.info('Ifgs already reference phase corrected')
-        return
-    else:
-        check_ref_phase_ifgs(ifgs, flags)
 
-
-def check_ref_phase_ifgs(ifgs, flags):
-    """
-    Function to check that the ref phase status of all ifgs are the same
-    """
-    count = sum([f == ifc.REF_PHASE_REMOVED for f in flags])
-    if (count < len(flags)) and (count > 0):
+    if preread_ifgs: # check unless for mpi tests
+        flags = [ifc.PYRATE_REF_PHASE in preread_ifgs[i].metadata
+                 for i in ifgs]
+    else:
+        flags = [True if i.dataset.GetMetadataItem(ifc.PYRATE_REF_PHASE)
+                 else False for i in ifgs]
+
+    if sum(flags) == len(flags):
+        log.info('Skipped reference phase estimation, ' \
+                 'ifgs already corrected')
+        return True
+    elif (sum(flags) < len(flags)) and (sum(flags) > 0):
         log.debug('Detected mix of corrected and uncorrected '
                   'reference phases in ifgs')
-
         for i, flag in zip(ifgs, flags):
             if flag:
                 msg = '{}: prior reference phase ' \
                       'correction detected'.format(i.data_path)
             else:
                 msg = '{}: no reference phase ' \
                       'correction detected'.format(i.data_path)
-            log.debug(msg.format(i.data_path))
-            raise ReferencePhaseError(msg)
+                log.debug(msg.format(i.data_path))
+                raise ReferencePhaseError(msg)
+    else: # count == 0
+        log.info('Estimating and removing reference phase')
+        return False
 
 
 class ReferencePhaseError(Exception):