update 0.9.4

Documentation/EGADS Algorithm Handbook - LATEX/cover/alg_cover.tex

@@ -24,7 +24,7 @@
 \setlength{\voffset}{.0cm}
 
 \logoeufar
-\title{EUFAR, Olivier Henry}{}{EGADS Lineage Algorithm Handbook}{Version 0.9.3}
+\title{EUFAR, Olivier Henry}{}{EGADS Lineage Algorithm Handbook}{Version 0.9.4}
 %\author{}{}
 
 

Documentation/changelog.txt

@@ -4,11 +4,37 @@ CHANGELOG | LINEAGE BRANCH
 
 To do list:
 -----------
-  - display a warning if a unit is not well recognized.
+  - modify NasaAmes class to ressemble the NetCdf and EgadsNetCdf classes. (v0.9.5)
+  - change the way to add new units. (v.0.9.5)
 
 To fix list:
 ------------
 
+June 7 2019, Release version 0.9.4
+==================================
+  MODIFIED:
+    - a doc string has been modified in netcdf_io.py to show that the add_attribute function accepts different type of value.
+	- when saving variables in a nasa ames files, if the user enter a name for each variable, it will be used instead of the standard_name or long_name of the EgadsData class.
+	- as the nasa_ames_io class is working only for FFI 1001, the independant variable is now stored directly in the na_dict and not in a list (as for usual variables).
+	- the save_na_file function has been modified to take into account integers and floats numbers. Now by default, floats numbers are saved will all decimal numbers if float_format is not entered.
+	- if an attribute is added to a variable in a new or an existing netcdf file, and if the attribute is a list, it is automatically converted to a string with the elements of the list separated by a ', '.
+	- values coming from a NASA Ames file and injected into an EgadsData class will have the right format (int32, float64, ...) and not only float.
+	- the documentation has been updated.
+	- the function convert_to_nasa_ames of the EgadsNetcdf and NetCdf classes has been modified to take into account improvements brought to the NasaAmes class.
+	- when reading a variable from a nasa ames files, data were stored in a list then in an EgadsData
+
+  FIXED:
+    - old python2 code has been found and removed.
+	- the na_dict object wasn't well passed between functions in the nasa_ames_io class.
+	- in the nasa_ames_io class, if default value is used, integers (ex: 1) are now real integers in the file and not floats (ex: 1.0).
+	- few typos in the documentation.
+	- in the convert_to_nasa_ames function of the netcd_io classes, few metadata were embedded in lists. It has been fixed.
+
+  ADDED:
+    - because of the limitation of Numpy with integers and NaN, an option has been introduce to let the user choose if data has to be read as float when reading a file.
+	- for NetCDF (EgadsNetCdf class) and NasaAmes (NasaAmes class) files, the option 'replace_fill_value' has been added to read variables. If the option 'replace_fill_value' is True, EGADS will replace automatically values set in '_FillValue' or 'missing_value' attributes by numpy.nan. By default the option is False.
+	- a _FillValue attribute is added to the algorithm output IF one of the inputs has a _FillValue or missing value attribute.
+
 
 May 9 2019, Release version 0.9.3
 =================================

PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 1.1
 Name: egads-lineage
-Version: 0.9.3
+Version: 0.9.4
 Summary: EGADS: EUFAR General Airborne Data-processing Software
 Home-page: https://github.com/EUFAR/egads/tree/Lineage
 Author: Olivier Henry

README.md

@@ -1,7 +1,7 @@
 Version:
 -------
 
-EGADS Lineage 0.9.3
+EGADS Lineage 0.9.4
 
 
 Developments:
@@ -46,7 +46,7 @@ Features:
 Installation:
 -------------
 
-Read INSTALL.txt to lear how to install EGADS Lineage.
+Read INSTALL.txt to learn how to install EGADS Lineage.
 
 
 Documentation:

doc/source/conf.py

@@ -48,9 +48,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.9.3'
+version = '0.9.4'
 # The full version, including alpha/beta/rc tags.
-release = '0.9.3'
+release = '0.9.4'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/source/tutorial.rst

@@ -427,10 +427,10 @@ EGADS provides two classes to work with NetCDF files. The simplest, :class:`egad
 Opening
 --------
 
-To open a NetCDF file, simply create a :class:`.NetCdf()` instance and then use the ``open(pathname, permissions)`` command:
+To open a NetCDF file, simply create a :class:`.EgadsNetCdf()` instance and then use the ``open(pathname, permissions)`` command:
 
     >>> import egads
-    >>> f = egads.input.NetCdf()
+    >>> f = egads.input.EgadsNetCdf()
     >>> f.open('/pathname/filename.nc','r')
 
 Valid values for permissions are:
@@ -456,9 +456,9 @@ Reading data
 
 To read data from a file, use the ``read_variable()`` function:
 
-    >>> data = f.read_variable(var_name, input_range)
+    >>> data = f.read_variable(var_name, input_range, read_as_float, replace_fill_value)
 
-where ``var_name`` is the name of the variable to read in, and ``input_range`` (optional) is a list of min/max values.
+where ``var_name`` is the name of the variable to read in, and ``input_range`` (optional, by default ``None``) is a list of min/max values ; if ``read_as_float`` (optional, by default ``False``) is True, EGADS reads the data and convert them to float numbers, if False, the data type is the type of data in file ; if ``replace_fill_value`` (optional, by default ``False``) is True, EGADS reads the data and replace ``_FillValue`` or ``missing_value`` (if one of the attributes exists) in data by NaN (numpy.nan).
 
 If using the ``egads.input.NetCdf()`` class, an array of values contained in ``var_name`` will be returned. If using the ``egads.input.EgadsNetCdf()`` class, an instance of the :class:`~.EgadsData` class will be returned containing the values and attributes of ``var_name``.
 
@@ -477,22 +477,22 @@ Data can be output to variables using the ``write_variable()`` function as follo
 
 where ``var_name`` is a string for the variable name to output, ``dims`` is a tuple of dimension names (not needed if the variable already exists), and ``type`` is the data type of the variable. The default value is *double*, other valid options are *float*, *int*, *short*, *char* and *byte*. 
 
-If using :class:`~.NetCdf`, values for ``data`` passed into ``write_variable`` must be scalar or array. Otherwise, if using :class:`~.EgadsNetCdf`, an instance of :class:`~.EgadsData` must be passed into ``write_variable``. In this case, any attributes that are contained within the :class:`~.EgadsData` instance are applied to the NetCDF variable as well.
+If using :class:`~.NetCdf`, values for ``data`` passed into ``write_variable`` must be scalar or array. Otherwise, if using :class:`~.EgadsNetCdf`, an instance of :class:`~.EgadsData` must be passed into ``write_variable``. In this case, any attributes that are contained within the :class:`~.EgadsData` instance are applied to the NetCDF variable as well. If an attribute with a name equal to ``_FillValue`` or ``missing_value`` is found, NaN in data will be automatically replaced by the missing value.
 
 Conversion from NetCDF to NASA/Ames file format
 ------------------------------------------------
 
-The conversion is only possible on opened NetCDF files. If modifications have been made and haven't been saved, the conversion won't take into account those modifications. Actually, the only File Format Index supported by the conversion in the NASA/Ames format is 1001. Consequently, if variables depend on multiple independant variables (e.g. ``data`` is function of ``time``, ``longitude`` and ``latitude``), the file won't be converted and the function will raise an exception. On the contrary, if multiple independant variables (or dimensions) exist, and if each variable depend on only one independant variable (e.g. ``data`` is only function of ``time``), the file will be converted and the function will generate one file per independant variable. If the user needs to convert a complex file with variables depending on multiple independant variables, the conversion should be done manually by creating a NASA/Ames instance and a NASA/Ames dictionary, by populating the dictionary and by saving the file.
+The conversion is only possible on opened NetCDF files. If modifications have been made and haven't been saved, the conversion won't take into account those modifications. Actually, the only File Format Index supported by the conversion is 1001. Consequently, if more than one independant variables are present in the NetCDF file, the file won't be converted and the function will raise an exception. If the user needs to convert a complex file with variables depending on multiple independant variables, the conversion should be done manually by creating a NASA/Ames instance and a NASA/Ames dictionary, by populating the dictionary and by saving the file.
 
 To convert a NetCDF file, simply use:
 
 * ``f.convert_to_nasa_ames()`` -- convert the currently opened NetCDF file to NASA/Ames file format
-* ``f.convert_to_nasa_ames(na_file, requested_ffi, float_format, delimiter, no_header)`` -- ``na_file``, ``requested_ffi``, ``float_format``, ``delimiter`` and ``no_header`` are optional parameters ; ``na_file`` is the name of the output file once it has been converted, by default the name of the NetCDF file will be used with the extension .na ; ``requested_ffi`` is not used actually, but will be functional in a next version of EGADS ; ``float_format`` is the formatting string used for formatting floats when writing to output file, by default ``%g`` ; ``delimiter`` is a character or a sequence of character for use between data items in the data file, by default '    ' (four spaces) ; if ``no_header`` is set to ``True``, then only the data blocks are written to file, by default ``False``
+* ``f.convert_to_nasa_ames(na_file, float_format, delimiter, no_header)`` -- ``na_file``, ``float_format``, ``delimiter`` and ``no_header`` are optional parameters ; ``na_file`` is the name of the output file once it has been converted, by default the name of the NetCDF file will be used with the extension .na ; ``float_format`` is the formatting string used for formatting floats when writing to output file, by default ``None`` ; ``delimiter`` is a character or a sequence of character for use between data items in the data file, by default '    ' (four spaces) ; if ``no_header`` is set to ``True``, then only the data blocks are written to file, by default ``False``
 
 To convert a NetCDF file to NASA/Ames CSV format, simply use:
 
 * ``f.convert_to_csv()`` -- convert the currently opened NetCDF file to NASA/Ames CSV format
-* ``f.convert_to_csv(csv_file, float_format, no_header)`` -- ``csv_file``, ``float_format`` and ``no_header`` are optional parameters ; ``csv_file`` is the name of the output file once it has been converted, by default the name of the NetCDF file will be used with the extension .csv ; ``float_format`` is the formatting string used for formatting floats when writing to output file, by default ``%g`` ; if ``no_header`` is set to ``True``, then only the data blocks are written to file, by default ``False``
+* ``f.convert_to_csv(csv_file, float_format, no_header)`` -- ``csv_file``, ``float_format`` and ``no_header`` are optional parameters ; ``csv_file`` is the name of the output file once it has been converted, by default the name of the NetCDF file will be used with the extension .csv ; ``float_format`` is the formatting string used for formatting floats when writing to output file, by default ``None`` ; if ``no_header`` is set to ``True``, then only the data blocks are written to file, by default ``False``
 
 Other operations
 -----------------
@@ -604,9 +604,9 @@ Reading data
 
 To read data from a file, use the ``read_variable()`` function:
 
-    >>> data = f.read_variable(var_name)
+    >>> data = f.read_variable(var_name, na_dict, read_as_float, replace_fill_value)
 
-where ``var_name`` is the name of the variable to read in. The data will be read in to an instance of the :class:`~.EgadsData` class, containing the values and attributes of ``var_name``.
+where ``var_name`` is the name of the variable to read in. The data will be read in to an instance of the :class:`~.EgadsData` class, containing the values and attributes of ``var_name`` . By default equal to ``None`` and optional, ``na_dict`` will tell to EGADS in which Nasa Ames dictionary to read data, if ``None`` data are read in the opened file. If ``read_as_float`` (optional, by default ``None``) is True, EGADS reads the data and convert them to float numbers, if False, the data type is the type of data in file ; if ``replace_fill_value`` (optional, by default ``False``) is True, EGADS reads the data and replace ``_FillValue`` or ``missing_value`` (if one of the attributes exists) in data by NaN (numpy.nan).
 
 Writing data
 -------------
@@ -619,14 +619,16 @@ To write data to the current file or to a new file, the user must save a diction
 * ``f.write_variable(data, var_name)`` -- write or replace a variable ; the function will search if ``data`` is already in the dictionary by comparing ``varname`` with other variable names in the dictionary, if it is found, ``data`` will replace the old variable, if not ``data`` is considered as a new variable ; ``data`` can be an :class:`~.EgadsData` or a vector/matrix.
 * ``f.write_variable(data, var_name, var_type, attr_dict, na_dict)`` -- ``var_type``, ``attr_dict`` and ``na_dict`` are optional ; ``attr_dict`` (a dictionary of standard NASA/ames variable attributes: 'name', 'units', '_FillValue' and 'scale_factor') must be provided if ``data`` is not an :class:`~.EgadsData` (in that case, variable attributes are retrieve from the :class:`~.EgadsData`.metadata dictionary) ; if ``na_dict`` is provided, the function saves the variable in the NASA/Ames dictionary ``na_dict``
 
+If a :class:`~.EgadsData` is passed into the ``write_variable`` function, any attributes that are contained within the :class:`~.EgadsData` instance are automatically populated in the NASA Ames dictionary as well, those which are not mandatory are stored in the SCOM attribute. If an attribute with a name equal to ``_FillValue`` or ``missing_value`` is found, NaN in data will be automatically replaced by the missing value.
+
 Saving a file
 --------------
 
 Once a dictionary is ready, use the ``save_na_file()`` function to save the file:
 
     >>> data = f.save_na_file(file_name, na_dict, float_format):
 
-where ``file_name`` is the name of the new file or the name of the current file, ``na_dict`` the name of the dictionary to be saved (optional, if not provided, the current dictionary will be used), and ``float_format`` the format of the floating numbers in the file (by deffault, two decimal places).
+where ``file_name`` is the name of the new file or the name of the current file, ``na_dict`` the name of the dictionary to be saved (optional, if not provided, the current dictionary will be used), and ``float_format`` the format of the floating numbers in the file (by default, no round up).
 
 Conversion from NASA/Ames file format to NetCDF
 ------------------------------------------------

egads/_version.py

@@ -1,2 +1,2 @@
-__version__ = '0.9.3'
+__version__ = '0.9.4'
 __branch__ = 'Lineage'

egads/algorithms/comparisons/compare_param_lcss.py

@@ -1,15 +1,15 @@
 __author__ = "mfreer"
 __date__ = "2013-02-17 18:01"
-__version__ = "1.7"
+__version__ = "1.8"
 __all__ = ['CompareParamLcss']
 
-import egads.core.egads_core as egads_core
+from egads.core.egads_core import EgadsAlgorithm
 import egads.core.metadata as egads_metadata
 import numpy
 import math
 
 
-class CompareParamLcss(egads_core.EgadsAlgorithm):
+class CompareParamLcss(EgadsAlgorithm):
 
     """
     
@@ -42,7 +42,7 @@ class CompareParamLcss(egads_core.EgadsAlgorithm):
     """
 
     def __init__(self, return_Egads=True):
-        egads_core.EgadsAlgorithm.__init__(self, return_Egads)
+        EgadsAlgorithm.__init__(self, return_Egads)
 
         self.output_metadata = egads_metadata.VariableMetadata({'units':'',
                                                                'long_name':'maximum common subsequence length',
@@ -72,7 +72,7 @@ def __init__(self, return_Egads=True):
 
     def run(self, R, S, epsilon, norm=True):
 
-        return egads_core.EgadsAlgorithm.run(self, R, S, epsilon, norm)
+        return EgadsAlgorithm.run(self, R, S, epsilon, norm)
 
     def _algorithm(self, R, S, epsilon, norm):
 

egads/algorithms/thermodynamics/pressure_dynamic_angle_incidence_vdk.py

@@ -104,7 +104,7 @@ def _algorithm(self, delta_P_t, delta_P_b, delta_P_l, delta_P_r, delta_P_s, C_al
 
         print('Warning: the algorithm is still in beta phase and must be reviewed.')
 
-        P_tot =  (numpy.sqrt((1. / 125.) * ((delta_P_t + delta_P_r + delta_P_b + delta_P_l)**2 
+        P_tot = (numpy.sqrt((1. / 125.) * ((delta_P_t + delta_P_r + delta_P_b + delta_P_l)**2
                                       + (- 4 * delta_P_t + delta_P_r + delta_P_b + delta_P_l)**2
                                       + (delta_P_t - 4 * delta_P_r + delta_P_b + delta_P_l)**2 
                                       + (delta_P_t + delta_P_r - 4 * delta_P_b + delta_P_l)**2
@@ -114,9 +114,9 @@ def _algorithm(self, delta_P_t, delta_P_b, delta_P_l, delta_P_r, delta_P_s, C_al
         k_b = (delta_P_r - delta_P_l) / P_tot
 
         sum_out = numpy.zeros((12, len(k_a)))
-        for i in range(0,12):
+        for i in range(0, 12):
             sum_in = numpy.zeros((12, len(k_b)))
-            for j in range(0,12):
+            for j in range(0, 12):
                 sum_in[j] = C_alpha[i][j] * (k_b ** j)
             sum_out[i] = (k_a**i) * numpy.sum(sum_in, axis = 0)
         alpha_cp = numpy.sum(sum_out, axis = 0)

egads/algorithms/transforms/seconds_to_isotime.py

@@ -70,7 +70,6 @@ def __init__(self, return_Egads=True):
     def run(self, t_secs, t_ref=None, fmt=None):
         return egads_core.EgadsAlgorithm.run(self, t_secs, t_ref, fmt)
 
-
     def _algorithm(self, t_secs, t_ref, fmt):
         if fmt:
             fmt = str(fmt)

egads/algorithms/transforms/time_to_decimal_year.py

@@ -67,7 +67,7 @@ def run(self, t, t_ref=None):
     def _algorithm(self, t, t_ref=None):
         t_ref_s = 0
         if t_ref:
-            t_ref_s = egads.algorithms.transforms.IsotimeToSeconds().run([t_ref], '19500101T000000')  # @UndefinedVariable
+            t_ref_s = egads.algorithms.transforms.IsotimeToSeconds().run([t_ref], '19500101T000000')
         t_s = t + egads.EgadsData(value=t_ref_s, units='s').rescale('year').value
         t_y = t_s + 1950
         return t_y

egads/core/egads_core.py

@@ -1,6 +1,6 @@
 __author__ = "ohenry"
 __date__ = "2018-03-07 16:42"
-__version__ = "1.3"
+__version__ = "1.4"
 __all__ = ["EgadsData", "EgadsAlgorithm"]
 
 import logging
@@ -32,7 +32,7 @@ def __new__(cls, value, units='', variable_metadata={}, dtype=None, **attrs):
             units = variable_metadata.get('units', '')
             if not units:
                 units = variable_metadata.get('Units', '')
-        
+
         # quantities can't handle the CF time unit 'time since epoch'
         # to allow a proper operation of EGADS, a new attribute has been added,
         # transparent to the user, if the epoch is needed.
@@ -327,7 +327,26 @@ def run(self, *args):
                             metadata[key] = out_category
                 except KeyError:
                     pass
+
+        fillvalue_list = []
+        for arg in args:
+            if isinstance(arg, EgadsData):
+                try:
+                    fillvalue_list.append(arg.metadata['_FillValue'])
+                except KeyError:
+                    try:
+                        fillvalue_list.append(arg.metadata['missing_value'])
+                    except KeyError:
+                        pass
+                except AttributeError:
+                    pass
+
+        if fillvalue_list:
+            for i in range(len(self.output_metadata)):
+                self.output_metadata[i]['_FillValue'] = fillvalue_list[0]
+
         output = self._call_algorithm(*args)
+
         if len(self.metadata['Outputs']) > 1:
             result = []
             for i, value in enumerate(output):