forked from OPENDAP/hdf4_handler
-
Notifications
You must be signed in to change notification settings - Fork 0
BES module to read hdf4 files
License
captainkirk99/hdf4_handler
Â
Â
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
 |  | |||
Repository files navigation
What's new for Hyrax 1.16.3 ---------------------------- CF option: 1. Enhance the support of handling HDF-EOS2 swath multiple dimension map pairs. The enhancement includes the support of multiple swaths. This fix solves the MOD09/MYD09 issue docoumented in HFRHANDLER-332. Note for this enhancement: 1) Limitations (1) The Latitude and Longitude must be 2 dimensional arrays. (2) The number of dimension maps must be an even number in a swath. (3) The handling of MODIS level 1B is still kept as the old way. (4) When there is a one pair of dimension maps in a swath and the the geo-dimensions defined in the dimension maps are only used by 2-D Latitude and Longitude fields, we still keep the old way. 2) Variable/dimension name conventions (1) The HDF-EOS2 file contains only one swath The swath name is not included in the variable names. For Latitude and Longitude, the interpolated Latitude and Longitude variable names are named as "Latitude_1","Latitude_2","Longitude_1","Longitude_2". The dimension and other variable names are just modified by following the CF conventions. A DDS example can be found under https://github.com/OPENDAP/hdf4_handler/ Go to the following directory: /bes-testsuite/h4.nasa.with_hdfeos2/MYD09.dds.bescmd.baseline (2) The HDF-EOS2 file contains multiple swaths The swath name are included in the variable and dimension names to avoid name clashings. The swath names are added as suffix for variable and dimension names. Examples are like: "temperature_swath1","Latitude_swath1","Latitude_swath1_1" etc. A DDS example can be found under https://github.com/OPENDAP/hdf4_handler/ Go to the following directory: /bes-testsuite/h4.with_hdfeos2/swath_3_3d_dimmap.dds.bescmd.baseline 3) For applications that don't want to handle dimension maps, one can change the BES key "H4.DisableSwathDimMap=false" at h4.conf.in to "H4.DisableSwathDimMap= true". 2. Add a BES key to turn off the handling of HDF-EOS2 swath dimension map. What's new for Hyrax 1.16.2 ---------------------------- Clean up some compiler warnings. What's new for Hyrax 1.16.1 ---------------------------- Default option: Fix the memory leaking caused by handling vdata. What's new for Hyrax 1.15.4 ---------------------------- CF option: 1. Map the AIRS version 6 HDF-EOS Grid/Swath attributes to DAP2. What's new for Hyrax 1.15.3 ---------------------------- CF option: 1. Enhance the support of handling the scale_factor and add_offset to follow the CF. The scale_factor and add_offset rule for the MOD/MYD21 product is different than other MODIS products. We make an exception for this product only to ensure the scale_factor and add_offset follow the CF conventions. What's new for Hyrax 1.15.0 ---------------------------- CF option: 1. Enhance the support of handling the scale_factor and add_offset to follow the CF. The scale_factor and add_offset rule for the MOD16A3 product is different than other MODIS products. We make an exception for this product only to ensure the scale_factor and add_offset follow the CF conventions. What's new for Hyrax 1.14.0 ---------------------------- CF option: 1. Enhance the support of handling the scale_factor and add_offset to follow the CF. The scale_factor and add_offset rule for the MOD16A2 product is different than other MODIS products. We make an exception for this product only to ensure the scale_factor and add_offset follow the CF conventions. Default option(Not developed by The HDF Group): 1. We fixed several coding issues discovered by the coverity scan. We also fixed quite a few memory leaking issues discovered by valgrind. What's new for Hyrax 1.13.4 ---------------------------- This is a maintenance release. No new features are added and no outstanding bugs are fixed. ***************************************************** ********Special note about the version number******** ***************************************************** Since Hyrax 1.13.2, Hyrax just treats each handler as a module. So We stop assigning an individual version number for the HDF4 handler. Updated for version 3.12.2(November 16 2016, Hyrax 1.13.2) 1) Minor code re-arrangement by removing the DAP4 macro and other misc. minor code updates. 2) The [Known Issues] is updated to reflect the current findings of some issues that either not appropriate or not worth the effort for the handler to handle. [Known Issues] 1) For AMSR E level-3 data, the variable like SI_12km_N_18H_DSC doesn't have any CF scale/offset, _FillValue, unit attributes. These attributes have to be retrieved from the product document. Therefore, the plot generated directly from the OPeNDAP handler may not be correct. If AMSR_E level 3 HDF4 products are still served by NASA data centers, the Hyrax's NcML module can be used with the HDF4 handler to provide the missing CF attributes to generate the correct plot. More information on how to use NcML to provide the missing CF attributes, please check http://hdfeos.org/examples/ncml.php 2) For the HDF-EOS2 swath, the HDF4 handler identifies the latitude and longitude coordinate variables based on the variable names rather than via the CF units attribute since one cannot use the HDF-EOS2 library to add attributes for an HDF-EOS2 field. Therefore, the HDF-EOS2 swath field attributes can only be added and retrieved by using the HDF4 APIs. It is difficult and inefficient for the handler to use HDF4 APIs when handling the HDF-EOS2 swath fields. Given that all NASA HDF-EOS2 swath files we observed simply use name "Latitude" for the latitude field and "Longitude" for the longitude field, we believe using variable names "Latitude" and "Longitude" to identify latitude and longitude under swath geo-location fields is sufficient to serve NASA HDF-EOS2 products. Even if the handler cannot identify the latitude and longitude as coordinate variables, the handler will still generate the correct DDS and DAS and Data responses although they may not follow the CF conventions. 3) DataCache may not work on MacOS for rare cases. Some HDF4 variables can be cached in disk. The cached file name uses the variable name as the key to distinguish each other. On MacOS system that is not configured to be case-sensitive (using diskutil info to check your OS), when the DataCache BES key is turned on, two legal variable names like TIME and Time with the same shape may share the same cached file name. Thus this may cause the data inconsistent. Given that the DataCache feature is turned off by default and the usage of this feature is only on Linux, the handler just documents this as an known issue. Updated for version 3.12.1(April 29 2016, Hyrax 1.13.1)) 1) Improve the calculation of XDim and YDim for the sinusoidal projection. 2) Improve the handling of BES keys by moving the initialization of BES keys to the Handler constructor. 3) Bug Fixes: (1) Remove the character dot(.) when the _FillValue is NaN. In the previous version, the character dot(.) was added at the end of NaN. This prevents the NetCDF Java clients to access the DAS. (2) Correct the (0,360) longitude value when the HDF-EOS2 EnableEOSGeoCacheFile is turned on. Some HDF-EOS2 products make the longitude range from 0 to 360. When the HDF-EOS2 lat/lon cache is on, the longtiude values were not retrieved correctly in the previous versions. This was discovered by testing an AMSR-E product when updating the testsuite for the handler. [Known Issues] 1) DataCache may not work on MacOS for rare cases. Some HDF4 variables can be cached in disk. The cached file name uses the variable name as the key to distinguish each other. On MacOS system that is not configured to be case-sensitive(using diskutil info to check your OS), when the DataCache BES key is turned on, two legal variable names like TIME and Time with the same shape may share the same cached file name. Thus this may cause the data inconsistent. Given that the DataCache feature is turned off by default and the usage of this feature is only on Linux, the handler just documents this as an known issue. 2) Note for version 3.12.0: no new features or bug fixes for version 3.12.0. Only several warnings are removed. The version was accidently bumped from 3.11.7 to 3.12.0. Updated for version 3.11.7 (15 September 2015,Hyrax 1.13.0,1.12.2,1.12.1) New Features: 1) We added 1-D coordinate variables and CF grid_mapping attributes for HDF-EOS2 grid with Sinusoidal projection. This request is from LP DAAC. 2) We added the DDS and DAS cache for AIRS version 6 products and the data cache for HDF4 data read via SDS interfaces. Several other improvement is also made for AIRS. This request is from GES DISC. Bug Fixes: 1) We fixed a bug caused by casting values via pointers between different datatypes. This is first reported by NSIDC and then from OPeNDAP. 2) We fixed the attribute missing issue for the HDF-EOS2 swath geo-location variables.(variables under Geolocation Fields) 3) We also made the correct representation of scale_factor and add_offset CF attributes for HDF-EOS2 swath geo-location variables. ****** You may need to read the information about BES keys in the file h4.conf.in to see if the default values need to be changed for your service. ****** Updated for version 3.11.6 (15 November 2014,Hyrax 1.11.2,1.11.1,1.11.0,1.10.1,1.10.0) In this version, we added the following features: 1) Implement an option to cache HDF-EOS2 grid latitude and longitude values to improve performance. 2) Implement an option not to pass file ids for the compatibility with NcML modules. 3) Improve the access of DDS by using the file structure obtained in the stage of building DAS. 4) Add the CF support of AIRS version 6 level 2(swath). 5) Support the mapping of vgroup attributes to DAP. 6) Support the mapping of HDF-EOS2 swath and grid object(like vgroup) attributes to DAP. Bug fixes: 1) Obtained the CF add_offset values for MOD/MYD08_M3 products. 2) Fixed the wrong dimension size when a dimension is unlimited. 3) Make vdata field attributes mapped to correct DAS container. NASA HDF4 and HDF-EOS2 products that are supported up to this release: HDF-EOS2: AIRS, MODIS, some MISR, some Merra, some MOPITT HDF4: TRMM version 6 and version 7, some CERES, some OBPG Performance enhancement: AIRS version 6 and MODIS O8_M3-like products HDF-EOS2 grid lat/lon are calculated via cache You may need to read the information about BES keys in the file h4.conf.in to see if the default values need to be changed for your service. Updated for version 3.11.5 (25 April 2014,Hyrax 1.9.7) In this version, we fix the following datatype mapping issues: 1) We make HDF4 DFNT_CHAR array map to DAP string for variables. In the previous versions, DFNT_CHAR is mapped to DAP BYTE for variables. This may cause some NASA HDF4 vdata DFNT_CHAR array to DAP BYTE array, which is not right. The fix makes the vdata fields of some NASA MISR (MISR_AEROSOL etc.) products map correctly to DAP. 2) We fix the mapping of DFNT_INT8 attribute value to DAP. In the previous versions, DFNT_INT8(signed 8-bit integer) is mapped to DAP BYTE(unsigned 8-bit integer). This will cause misrepresentation when the value is < 0. This fix will make some attribute values of some MODIS or MISR non-physical fields map correctly to DAP. 3) For attribute _FillValue, we enforce that it is always a number even if the attribute datatype is DFNT_CHAR or DFNT_UCHAR. In this release,we make the string representation of a _FillValue to a number. We also make turn on the H4.DisableStructMetaAttr key to be 'true'the default setting of h4.conf.in. This will improve the performance to generate DAS. Updated for version 3.11.4 (1 April 2014,Hyrax 1.9.3) This version improves I/O by reducing the number of HDF4/HDF-EOS2 file open / close requests. The default BES key settings are changed. See USAGE document. Support to TRMM version 7 (3B43, 3B42, 3B31, 3A26, 3A25, 3A12, 3A11, 2B31, 2A25, 2A23, 2A21, 2A12, 1C21, 1B21, 1B11 and 1B01) and some TRMM level 3 version 6(CSH and 3A46) products are added. The emphasis is on level 3 grid products. Some memory leaks detected by valgrind are fixed. Error handling is greatly improved:resources are released properly when errors occur. The testsuite is updated accordingly. All the updates are solely applied to the CF option. [Known Issues] Note:HFRHANDLER-??? means The HDF Group's JIRA ID. HFRHANDLER-223:(U)CHAR8 type _FillValue attribute value conversion produces string representation, instead of real value when the dataset is in numeric type. HFRHANDLER-224:Some attributes of Lat/Lon geolocation fields in HDF-EOS2 swath are dropped in DAS output. Updated for version 3.11.3 (1 February 2014,Hyrax 1.9.2) This version optimizes handler for MOD08 M3 and AIRS version 6 products with new BES keys. See the USAGE document for the new BES keys for details. Updated for version 3.11.2 (18 October 2013) This version changes BaseType::read(). Updated for version 3.11.1 (10 September 2013) This version addresses some issues in the code base. Updated for version 3.11.0 (30 May 2013) Limitation of handling MOD14 product is documented under USAGE. MOD14 is not an HDF-EOS2 product so having MOD03 geo-location file will not change the output of the HDF4 handler. This version fixes the 7.6 of "Known Issues" documented in README for version 3.10.0. Search for "7.6" in this document for details. Updated for version 3.10.1 (27 Nov 2012) This version fixes a bug for reading Int16 type dataset from NASA MODIS 09 products. Updated for version 3.10.0 (30 Sep 2012) 1. Introduction The HDF4 handler version 3.10.0 is a major update for the CF support. The README for this version is *solely* for the CF support. The README consists of several sections: Usage, New Features, Bug fixes, Code Improvement, Limitations, and Known Issues. 2. Usage The current version uses BES Keys to make the configuration of the CF option more flexible and easier. The current version also doesn't require the HDF-EOS2 library to support the CF conventions for non-HDFEOS2 HDF4 files. Check the USAGE for the detailed information. 3. New Features 3.1 Testsuite Expansion If you configure the handler with '--with-hdfeos2', the 'make check' will test a new set of HDF-EOS2 test files that are added. Please make sure to clean everything with 'make distclean' if you want to test with a different configuration option each time. Source codes are also provided for the new set of HDF-EOS2 test files. 3.2 Naming conventions follow HDF5 handler naming conventions for consistency. Again, this is for the CF support only. 3.2.1 Non-CF-compliant characters Any non-CF-compliant character for objects and their attribute names will be changed to '_'. There is only one exception: if the first character of a name is '/', handler will ignore the first '/'. This is a request from NASA data centers since it means that the a path is prefixed to the name. The first '/' is ignored for better readability. The object names include all HDF-EOS2 and HDF4 objects supported by the CF option. They are HDF-EOS2 swath fields and grid fields, HDF4 SDS, HDF4 Vdata, HDF4 Vdata fields and corresponding attributes. 3.2.2 Multiple HDF-EOS2 swath and grid For multiple HDF-EOS2 swath and grid files, since we only find a bunch of AIRS and MODIS grid products in this category and so far the field names under these grids can be distinguished by themselves, so we simply keep the original field names. Grid names are not prefixed. For example, in AIRS.2002.08.24.L3.RetStd_H008.v4.0.21.0.G06104133343.hdf there are two fields with similar names: 1) Field under vgroup "ascending": TotalCounts_A 2) Field under vgroup "descending": TotalCounts_D As you can see, the field "TotalCounts" in each vgroup can be distinguished by their field names anyway. Prefixing a grid name to field name such as "ascending_TotalCounts_A" and "descending_TotalCounts_D" makes the field name difficult to read. 3.2.3 Non-HDFEOS2 SDS objects For pure HDF4 SDS objects, the handler prefixes the vgroup path for SDS object names. 3.2.4 SDS objects in Hybrid HDF-EOS2 files For SDS objects in hybrid HDF-EOS2 files, to make these SDS objects distinguished from the HDF-EOS2 grid or swath field names, handler adds "_NONEOS" to the object name. This is based on the following fact: These added SDS objects often share the same name as the HDF-EOS2 fields. For example, band_number can be found both under HDF-EOS2 swath and under the root vgroup. Thus, name clashing needs to be done. Adding "_NONEOS" is a better way to handle the name clashing than simply adding a "_1". 3.2.5 Vdata object name conventions The vdata name conventions are subject to change. We will evaluate the current name conventions after hearing feedback from users. Since the main NASA HDF4 object is SDS, hopefully this Vdata name conventions may not be critical. 3.2.5.1 Vdata object mapped to DAP array Since we would like to make sure that users can easily figure out the DAP variables mapped from vdata, we use the following name conventions. For example, a vdata "level1B" has a field "scan" under the group "g1" The mapped DAP variable name is "vdata_g1_level1B_vdf_scan." The "vdata_" prefix tells the user that this is an original vdata. The "_vdf_" tells the user that the string followed by "_vdf_" is the vdata field name. The handler also adds the following attributes to DAS to serve the similar purpose: std::string VDdescname = "hdf4_vd_desc"; std::string VDdescvalue = "This is an HDF4 Vdata."; std::string VDfieldprefix = "Vdata_field_"; These attributes will be generated if the corresponding BES key is turned on. See USAGE for details. 3.2.5.2 Vdata fields mapped to DAP attributes For the current version, if the number of vdata records is less than 10, the vdata field will be mapped to DAP attributes. This is based on the fact that small number of vdata records is mostly metadata (like attributes), so this kind of mapping may be more reasonable. However, the attribute name that are mapped from vdata field simply follows the conventions below: <vdata path>_<vdata name>_<vdata field name> The above name convention is simply following the handling of SDS. The rationale here is that distinguishing between SDS and Vdata fields in DAP attribute(DAS) is not as critical as distinguishing between SDS and Vdata fields in DAP array(DDS). However, this may cause user's confusion. We will evaluate this approach in the future based on user's feedback. 3.3 Name clashings Name clashing is handled similarly like the HDF5 handler. Only the name that has name clashing will be changed to a different name. This is according to NASA user's request. Previously, all corresponding names will be changed if there is a name clashing found in the file. See 3.2 for the details about the naming conventions. 3.4 New BES Keys There are six BES keys are newly added. H4.EnableCF H4.EnableMODISMISRVdata H4.EnableVdata_to_Attr H4.EnableCERESMERRAShortName H4.DisableVdataNameclashingCheck H4.EnableVdataDescAttr H4.EnableCF is the most important key. It must be set to be true if CF option is used. Other five keys are valid only if H4.EnableCF is set to be true. For more information, check the USAGE file. 3.5 Dimension maps and scale/offset handling for MODIS swath 3.5.1 Interpolating more fields according to the dimension maps Previously, we only supported the interpolation of latitude and longitude. In this release, we add the support of other fields such as solar_zenith, etc. if dimension map is used to store the field. Additionally, we also provide an option to directly read these fields (latitude, longitude, solar zenith etc.) from MOD03 or MYD03 files distributed by LAADS. For more information on this option, check the USAGE file. 3.5.2 Enhance the scale/offset handling for MODIS level 1B fields According to the discussions with NASA users, we found that for fields such as EV_1KM_RefSB and EV_1KM_Emissive, for values greater than 65500 are special values that should not be considered as the real data signal. So in this release, we keep values between 65500 and 65535 when applying the scale and offset function. Furthermore, we also calculated valid_min and valid_max for EV_???_RefSB and EV_???_Emissive to assure that users can plot data easily with Panoply and IDV. 3.6 Dimensions in MERRA products In this release, the handler uses HDF-EOS2 latitude and longitude fields as the coordinate variables. The added HDF4 SDS XDim and YDim are used inside the file as coordinate variables. 3.7 Vdata mapping The new handler can handle vdatas robustly either as an attribute in DAS or as an array in DDS. Review the previous 3.2.5.2 section for details. Several BES keys are also provided for users to have options on how to generate vdata DAP output. See USAGE for details. Vdata subsetting is handled robustly except for some HDF-EOS2 Vdata objects. See section 7.5 for the known issue. 3.8 Handling general HDF4 files with the CF option The previous release listed the NASA HDF4 products we handled specially to follow the CF convention. For other HDF4 products (we call them general HDF4 files), we do the following: A. Make the object and attribute names follow the CF conventions B. Follow the default handler's mapping to map SDS. In this way, some SDS objects that have dimension scales can be visualized by Panoply and IDV. C. Map Vdata according to 3.7. 4. Bug Fixes 4.1 Attribute names are cleaned up. If an attribute name contains non alpha numeric characters like '(' or '%', they are replaced with '_' to meet the CF naming conventions. 4.2 Products that use SOM projection are handled correctly and Panoply can display MISR data in block-by-block basis. However, please read 7.11 for the known issue for the interpretation of final visualization output. 4.3 We continued correcting attributes related to scale/offset. For example, the handler corrected "SCALE_FACTOR" and "OFFSET" attributes for AMSR_E L2A product by renaming them to "scale_factor" and "add_offset". It also cleaned any extra spaces in attribute names. We also corrected the rule on how to applying scale/offset equations in MODIS product (e.g., Solar_Zenith dataset). Finally, the handler renames Number_Type attributes to Number_Type_Orig if data field's type is changed by applying scale/offset by the handler (e.g., LPDAAC MOD11_L2 LST). 4.4 Strange ECS metadata sequences are handled. Some MODIS products have metadata name sequence "coremetadata.0, coremetadata.0.1, ..." instead of "coremdatadata.0, coremetadata.1, ..." 4.5 Mismatched valid_range attribute is removed from CERES ES4 product. Panoply fails to visualize the product if the valid_range attribute in lat/lon dataset doesn't match the calculated coordinate variable values returned by the handler. Thus, the handler removes the valid_attribute from coordinate variables. 4.6 There was a bug regarding subsetting the vdata field when the stride is greater than 1. It was fixed in this release. We also found a similar bug inside the HDF-EOS2 library regarding the subsetting of 1-D HDF-EOS2 swath. This should be fixed in the next HDF-EOS2 release. 5. Code Improvement 5.1 Refactored codes There were huge code overlap between hdfdesc.cc and HE2CF.cc. Those code lines are combined and are moved to HDFCFUtil.cc. 5.2 Error handlings and debugging Error handlings are improved to ensure the closing of all opened HDF4 API handles when error occurs. DEBUG macros are replaced with BESDEBUG or BESLog. 6. Limitations Again, all the limitations here are for the CF support (when the CF option is enabled) only. 6.1 Unmapped objects The handler ignores the mapping of image, palette, annotation, vgroup attributes and HDF-EOS2 swath group and grid group attributes. Note HDF4 global attributes(attributes from SD interfaces), HDF4 SDS objects, HDF4 Vdata attributes, HDF-EOS2 global(ECS metadata etc.) and field attributes are mapped to DAP. 6.2 The handler doesn't handle unlimited dimension. The result may be unexpected. (e.g., some fields in CER_ES8_TRMM-PFM_Edition2 product cannot be handled.) 6.3 Non-printable vdata (unsigned) character type data will not appear in DAS. If vdata char type column has a non-printable value like '\\005', it will not appear in DAS when vdata is mapped to attribute because the BES key, H4.EnableVdata_toAttr, is enabled. See the file USAGE for the usage of the key. 6.4 Vdata with string type is handled in character-by-character basis in 2D array. For example, when the vdata is a string of characters like "2006-1--01T16:17:12.693310Z", the handler represents it as Byte vdata_PerBlockMetadataTime_vdf_BlockCenterTime[VDFDim0_vdata_PerBlockMetadataTime_vdf_BlockCenterTime = 2][VDFDim1_vdata_PerBlockMetadataTime_vdf_BlockCenterTime = 28] = {{50, 48, 48, 54, 45, 49, 48, 45, 48, 49, 84, 49, 54, 58, 49, 55, 58, 49, 50, 46, 54, 57, 51, 51, 49, 48, 90, 0},{50, 48, 48, 54, 45, 49, 48, 45, 48, 49, 84, 49, 54, 58, 49, 55, 58, 51, 51, 46, 52, 51, 56, 57, 50, 54, 90, 0}}; 7. Known Issues 7.1 CER_SRBAVG3_Aqua-FM3-MODIS_Edition2A products have many blank spaces in long_name attribute. These products have datasets with really long_name attribute with size 277. However, most of them are blank spaces in the middle. For example, you'll see DAS output like below: String long_name "1.0 Degree Regional Monthly Hourly (200+ blank spaces) CERES Cloud Properties"; String units "unitless"; This is not a bug in the handler. The data product itself has such long attribute. 7.2 Longitude values for products that use LAMAZ projection will differ in i386 platform. For i386 machines, handler will generate different longitude values from x86_64 machines for the products that use Lambert azimuthal projection (LAMAZ) near North Pole or South Pole. For example, the handler will return 0 for 64-bit machine while it'll return -135 for 32-bit machine in the middle point of longitude in the NSIDC AMSR_E_L3_5DaySnow_V09_20050126.hdf product: < Float64 Longitude[YDim = 3][XDim = 3] = {{-135, 180, 135},{-90, 0, 90}, {-45, 0, 45}}; --- > Float64 Longitude[YDim = 3][XDim = 3] = {{-135, -180, 135},{-90, -135, 90}, {-45, -1.4320576915337e-15, 45}}; This is due to the calculations in the current GCTP library that HDF-EOS2 library uses. However, this will not affect the final visualization because, for North Pole or South Pole, the longitude can be anything from -180 to 180. So depending on floating point accuracy, handler may get different results for longitude of this pixel from GCTP. The longitude value is irrelevant at North Pole or South Pole for visualization clients. 7.3 IDV can't visualize SOM projection MISR products that use SOM projection have 3D lat/lon. Although Panoply can visualize them but IDV cannot. Handler doesn't treat the 3rd dimension as a separate coordinate variable and coordinate attribute on dataset includes only latitude and longitude variable names. 7.4 Vdata is mapped to attribute if there are less than or equal to 10 records For example, the DAS output of TRMM data 1B21 will show Vdata as an attribute: DATA_GRANULE_PR_CAL_COEF { String hdf4_vd_desc "This is an HDF4 Vdata."; Float32 Vdata_field_transCoef -0.5199999809; Float32 Vdata_field_receptCoef 0.9900000095; Float32 Vdata_field_fcifIOchar 0.000000000, 0.3790999949, 0.000000000, -102.7460022, 0.000000000, 24.00000000, 0.000000000, 226.0000000, 0.000000000, 0.3790999949, 0.000000000, -102.7460022, 0.000000000, 24.00000000, 0.000000000, 226.0000000; } This is part of the vdata handling convention, not an issue. However, we list here just for user's convenience. See 3.7 for more information. 7.5 Vdata subsetting in HDF-EOS2 data products may not work. Subsetting HDF-EOS2 vdata with large step index (e.g. a_vdata[0:999:999]) may not work due to a bug in HDF-EOS2 library. The bug has been reported to HDF-EOS2 developer and should be fixed in the next HDF-EOS2 release. Reading the entire Vdata is OK. 7.6 DDX generation will fail on PO.DAAC AVHHR product. For example, handler can't generate DDX output for NASA JPL PO.DAAC AVHRR product 2006001-2006005.s0454pfrt-bsst.hdf. Please see the OPeNDAP ticket #1930 for details. 7.7 It is possible to have name clashing between dimension names and variable names. Currently, the handler only checks the name clashing for the variables and the name clashing for the dimensions, not combined. Here's a reason: Many good COARDS files will have the following layout: lat[lat=180] lon[lon=360] If we want to check the name clashings for the combined set, this kind of good files will always have name clashings. And depending on the code flow, the final layout may become something like: lat_1[lat_1=180] lon_1[lon_1=360]. These are absolutely bad names for normal users. Instead, if we don't consider the combined set, the chance of the name clashing due to changing the conflicted coordinate variable names is very rare. So we may not do this at all until we really find a typical product that causes a problem. 7.8 long_name attribute for <variable name>_NONEOS The handler generates long_name attribute to indicate the original variable name for the SDS objects after renaming them with _NONEOS suffix. Those _NONEOS variables appear in hybrid files --- files that have additional HDF4 objects written by the HDF4 APIs on top of the existing HDF-EOS2 files. This addition of long_name attribute cannot be turned off using the H4.EnableVdataDescAttr=false key described in USAGE document. 7.9 Empty dimension name creates a variable with empty name. It is possible to create dataset with no dimension name in HDF-EOS2 library. In such case, the handler generates fake dimension variable without dataset name in DDS like below: Int32 [4]; Since there's no dataset name, data reading will also fail. 7.10 Special CF handlings for some products The handler doesn't correct scale_factor/add_offset/_FillValue for every HDF4 product to make it follow CF conventions. For example, the handler doesn't apply the scale and offset function(log equation) for OBPG CZCS Level 3 products. The handler doesn't insert or correct fill value attribute for some OBPG products so their plots may include fill values if you visualize them. OPeNDAP server administrator can fix these easily by using the NcML handler. PO.DAAC AVHRR product 2006001-2006005.s0454pfrt-bsst.hdf has "add_off" and doesn't specify fill value in attribute. Therefore, the final visualization image will not be correct for such product. 7.11 Bit shifting required in MISR product is not handled. Some datasets in MISR products combine two datasets into one, which requires bit shifting for the correct interpretation of the data. Handler doesn't perform such operation so the final visualization image may not be correct. (e.g., Blue Radiance in MISR_AM1_GRP_ELLIPSOID_GM_P117_O058421_BA_F03_0024.hdf) 7.12 Subsetting through Hyrax HTML interface will not work on LAMAZ products. You cannot subset Latitude_1 and Longitude_1 datasets using the HTML form. Checking the check box will not insert any array subscription text into the text box. Please see the OPeNDAP ticket #2075 for details. Kent Yang (myang6@hdfgroup.org) Updated for version 3.9.4 (19 Jan, 2011) If your system is non-i368 such as 64-bit architecture, please read IMPORTANT NOTE on INSTALL document regarding '--with-pic' configuration option (i.e., '-fPIC' compiler option). You need to install both HDF4 (and HDF-EOS2) library with '--with-pic' option if you encounter a linking problem. 1. Fixed the bug in uint16/uint32 type attribute handling. 2. The following bug fix applies to only --with-hdfeos2 configuration option. 2.1 Corrected the handling the scale/offset for MODIS products because the MODIS scale/offset equation is quite different from the CF standard. There are three different "scale_factor" and "add_offset" equations in MODIS data files: 1) For MODIS L1B, MODIS 03,05,06,07,08,09A1,17 and ATML2 level 2 swath products, MCD43B4, MCD43C1, MOD and MYD 43B4 level 3 grid files, the scale offset equation is correct_data_value = scale * (raw_data_value - offset). 2) For MODIS 13, MODIS 09GA, and MODIS 09GHK, the scale offset equation is correct_data_value=(raw_data_value -offset)/scale_factor. 3) For MODIS 11 level 2 swath products, the equation is correct_data_value = scale * raw_data_value + offset. We decide the type based on the group name. If the group name consists of "L1B", "GEO", "BRDF", "0.05Deg", "Reflectance", "MOD17A2","North", "mod05", "mod06", "mod07", "mod08", or "atm12", it is type 1. If the group name consists of "LST", it is type 2. If group name consists of "VI", "1km_2D", "L2g_2d", it is type 3. For type 1, use (raw_value-offset)*scale. For type 2, use (raw_value*scale+offset). For type 3, use (raw_value-offset)/scale. For recalculation of MODIS, one of the following conditions must meet: "radiance_scales" and "radiance_offsets" attributes are available, or "reflectance_scales" and "reflectance_offsets" attributes are available, or "scale_factor" and "add_offset" attributes are available. If any of the above conditions meet, recalculation will be applied, otherwise nothing will happen. If scale is 1 and offset is 0, we don't perform recalculation to improve performance. Data values are adjusted based on it type. If "scale_factor" and "add_offset" attributes are not available, "radiance_scales" and "radiance_offsets" attributes, or "reflectance_scales" and "reflectance_offsets" attributes, are used instead. After adjustment, the data type is converted uniformly to float not to lose precision even if its original data type is integer. The "valid_range" attribute is removed accordingly as it does not reflect the actual values any more. Since some netCDF visualization tools will apply the linear scale and offset equation to the data value if the CF "scale_factor" and "add_offset" attributes appear, these two attributes are renamed as "orig_scale_factor" and "orig_add_offset" respectively to prevent the second adjustment. 2.2 Latitude and longitude are provided for HDF-EOS2 grid files that use Space-Oblique Mercator(SOM) and Lambert Azimuthal Equal Area(LAMAZ) projections. We added the support for LAMAZ projection data such as MOD29 from NSIDC. For grid files using HDF-EOS2 Lambert Azimuthal Equal Area(LAMAZ) projection, the latitude and longitude values retrieved from the HDF-EOS2 library include infinite numbers. Those infinite numbers are removed and replaced with new values through interpolation. Therefore, an HDF-EOS2 grid file with LAMAZ projection can be served correctly. 2.3 Fixed memory release error that occurs on iMac (OS X Lion) with the STL string:string Map. 2.4 For OBPG L3m products, two additional CF attributes, "scale_factor" and "add_offset", are added if their scaling function is linear. The values of these two attributes are copied directly from file attributes, "Slope" and "Intercept". 2.5 Known Bugs: 1) Attribute names are not sanitized if they contain non-CF compliant characters such as '('. NSIDC MOD29 data product is a good example. 2) If different scale/offset rules should be applied to different datasets like MOD09GA product, the current handler cannot handle them properly. We apply scale/offset rule globally on per file basis, not on per individual dataset basis and CF visualization clients like IDV and Panoply will not display some datasets correctly since they'll apply scale/offset rule according to the CF-convention which doesn't match MODIS's scale/offset rule. Updated for 3.9.3 (21 Aug. 2011) Fixed a lingering issue with the processing of HDF-EOS attributes when the handler is not compiled with the HDF-EOS library. The handler was returning an error because those attributes were not parsing correctly. In the end, it appears there were two problems. The first was that some files contain slightly malformed EOS attributes: the 'END' token is missing the 'D'. The second was that this triggered a bogus error message. Fixed the nagging 'dim_0' bug where some files with variables that use similar names for arrays trigger a bug in the code that merges the das into the dds. The results was that sometimes the code tried to add a dim_0 attribute to a variable that already had one. This I fixed by correcting an error in the way the STL's string::find() method was used. Updated for 3.9.2 (17 Mar. 2011) In this patch, we added the following three features: 1. We add the mapping of the SDS objects added by using HDF4 APIs to an HDF-EOS2 file. These SDS objects are normally NOT physical fields, so they are not supposed to be plotted by Java tools such as IDV and Panoply. The attributes and values of these SDS objects may be useful for end users. 2. We also fix the bug of handling MERRA data. In the previous release, the unit of time is not handled correctly. This release fixes this bug under the condition that the file name of MERRA data must start with MERRA. 3. We also enhance the support of mapping HDF4 files that uses HDF4 SDS dimension scales. Especially we make a patch specially for P.O. DAAC's AVHRR files. Now with enough Heap space, IDV can visualize AVHRR files via OPeNDAP. What we haven't done: 1. We haven't mapped the Vdata objects added by using HDF4 APIs to an HDF-EOS2 file. 2. We haven't handled the plotting of the vertical profile files (Such as MOP Level 2 data). More investigation needs to be done on how IDV can handle things. 3. Other limitations listed in Section III of 3.9.1 that are not addressed above. Kent Yang (myang6@hdfgroup.org) Updated for 3.9.1 (14 Sep. 2010) In this release, we greatly enhance the support of the access of NASA HDF-EOS2 and HDF4 products. The whole note for 3.9.0 includes three sections. Section I. Configuration The handler is enhanced to support the access of many NASA HDF-EOS2 products and some NASA pure HDF4 products by many CF-compliant visualization clients such as IDV and Panoply. To take advantage of this feature, one MUST use HDF-EOS2 library and configure with the following option: ./configure --with-hdf4=<Your HDF4 library path> --with-hdfeos2=<Your HDF-EOS2 library path> --prefix=<Your installation path> Without specifying the option "--with-hdfeos2" will result in configuring the default HDF4 OPeNDAP handler. The HDF4 handler with the default options can NOT make the NASA HDF-EOS2 products and some NASA pure HDF4 products work with CF-compliant visualization clients. Some variable paths are pretty long(>15 characters). COARDS conventions require the number of characters in a field doesn't exceed 15 characters. So the above configuration option may cause some OPeNDAP clients that are still following COARDS conventions. To compensate that, we provide a configuration option to shorten the name so that in doesn't exceed 15 characters. To address the potential name clashing issue, both options may make some variable names to change so that unique variable names are present in the OPeNDAP output. To best preserve the original variable names, we recommend not to use --enable-short-name option if necessary. To configure the handler with the short name option, do the following: ./configure --with-hdf4=<Your HDF4 library path> --with-hdfeos2=<Your HDF-EOS2 library path> --prefix=<Your installation path> --enable-short-name To find the information on how to build the HDF-EOS2 library, please check http://hdfeos.org/software/hdfeos.php#ref_sec:hdf-eos2 To build RPMs by yourself, check the directory 'build_rpms_eosoption'. Section II. NASA products that are supported to be accessed via Java and other OPeNDAP visualization clients The following NASA HDF-EOS2 products are tested with IDV and Panoply, check the Limitation section for the limitations: 1). NASA GES DISC AIRS/MERRA/TOMS 2). NASA LAADS/LP DAAC/NSIDC Many MODIS products 3). NASA NSIDC AMSR_E/NISE products 4). NASA LaRC MISR/MOPITT/CERES-TRMM The following NASA special HDF4 products are tested with IDV Panoply, check the Limitation section for the limitations: 1). NASA GES DISC TRMM Level 1B, Level 2B Swath TRMM Level 3 Grid 42B and 43B 2). OBPG(Ocean Color) SeaWiFS/MODIST/MODISA/CZCS/OCTS level 2 and level 3m(l3m) 3). Some LaRC CERES products CER_AVG_Aqua-FM3-MODIS,CER_AVG_Terra-FM1-MODIS CER_ES4_Aqua-FM3_Edition1-CV or similar one CER_ISCCP-D2like-Day_Aqua-FM3-MODIS or similar one CER_ISCCP-D2like-GEO_ or similar one CER_SRBAVG3_Aqua or similar one CER_SYN_Aqua or similar one CER_ZAVG or similar one Section III. Limitations 1. Visualization clients and http header size 1). Visualization Slowness or even failures with IDV or panoply clients for big size field We found that for big size variable array(>50 MB), the visualization of the variable is very slow. Sometimes, IDV or Panoply may even generate an "out of memory" error. 2). Some NASA HDF files(some CERES files e.g.) include many (a few hundred) fields and the field names are long. This will cause the maximum http header size to exceed the default maximum http header size and a failure will occur. To serve those files, please increase your max http header size by adding the following line at your server.xml under the line containing <Connector port="8080" protocol="HTTP/1.1" maxHttpHeaderSize="819200" 2. HDF-EOS2 files 1) HDF-EOS2 Lambert Azimuthal Equal Area(LAMAZ) projection grid For LAMAZ projection data, the latitude and longitude values retrieved from the HDF-EOS2 library include infinite numbers. So an HDF-EOS2 grid file with LAMAZ projection can not be served correctly. 2) Latitude and longitude values that don't follow CF conventions 2.1) Missing (Fill) values inside latitude and longitude fields Except the HDF-EOS2 geographic(also called equidirectional, equirectangular, equidistant cylindrical) projection, clients may NOT display the data correctly. 2.2) 3-D latitude and longitude fields Except some CERES products listed at section 2, clients may NOT display the data correctly if the latitude and longitude fields are 3-D arrays. 3) HDF-EOS2 files having additional HDF4 objects Some HDF-EOS2 files have additional HDF4 objects. The object may be vdata or SDS. That means, some contents are added to an HDF-EOS2 file by using the HDF4 APIs directly after the HDF-EOS2 file is created. The HDF-EOS2 API may not retrieve the added information. Up to this release, we found those information are mainly related to metadata and those metadata may not be critical to visualize and analyze the real physical data variables. So in this release, those objects are currently ignored. One should NOTE that attributes of existing HDF-EOS2 data variables are NOT ignored. 4) Variables stored as 3-D or 4-D arrays Some variables stored as 3-D or 4-D arrays are either missing or hard to find the third or the fourth dimension's coordinate variables. The handler will use integer number(1,2,3,......) to represent the third or the fourth dimension as levels. Clients can still visualize the data in a horizontal plane level by level. 3. Pure HDF4 files All pure HDF4(e.g. non-HDFEOS2) products we've tested are listed under Section II. Other pure HDF4 products are not tested and may NOT be visualized by Java OPeNDAP clients. 4. Misc. For pure HDF4 products, currently attributes in a vgroup(not SDS or SD) are not mapped to DAP. To speed up the performance, we choose not to generate structMetadata, coreMetadata, archiveMetadata attributes for some CERES products. For applications that need these, please contact myang6@hdfgroup.org or eoshelp@hdfgroup.org. You can also post a message at hdfeos.org/forums . Kent Yang(myang6@hdfgroup.org) Updated for 3.8.1 The OPeNDAP HDF4 Data Handler is enhanced by adding two more configuration options: 1) --enable-cf 2) --use-hdfeos2=/path/to/hdfeos2_library The option 1) uses the StructMetadata parser. A valid HDF-EOS2 file always has the StructMetadata information and the parser can infer geolocation information from the StructMetadata. By retrieving such information, the HDF4 handler can generate DAP Grids that OPeNDAP visualization clients can display on a world map. The option 2) REQUIRES option 1) and it uses the HDF-EOS2 library instead of the StructMetadata parser. The benefit of using HDF-EOS2 library is tremendous. It can support more HDF-EOS2 files by handling different projections like polar and sinusoidal. In addition, it can detect any anomalies that are common in some HDF-EOS2 files and handle them intelligently. Thus, we recommend the server administrator to install HDF-EOS2 library first and configure the handler with BOTH 1) and 2) options. THE LIMITATIONS of the ENHANCED HDF4 HANDLER Please note that the enhanced handler has some limitations. o No support for using the HDF4 handler cache directory. o No support for Grids other than geographic 1-D projection. However, option 2) will make some Grids with other projections (polar, sinusoidal) work. o No vgroup to DAP structure mapping. Thus, the files that have same dataset name under different vgroups will throw a DDS semantics violation error. o No support for files that have the same dimension names with different dimension sizes. For example if a swath variable "A" has dimension lat=360 and lon=1440 (e.g., A[lat=360][lon=1440]) and another swath variable "B" has dimension lat=180 and lon=720 (e.g., B[lat=180][lon=720]), the handler will throw an error for inconsistent dimension. Updated for 3.7.12 (16 March 2009) This is the OPeNDAP HDF4 Data Handler. It is used along with the OPeNDAP DAP Server. For information about building the OPeNDAP netCDF Data Handler, see the INSTALL file. This handler uses a configuration parameter, set in the bes.conf file, to control where copies of some metadata objects built by the server are cached. By default this cache is in /tmp - you are encouraged to change that. Set the location using the 'HDF4.CacheDir' parameter. For example, if you have set the BES.CacheDir parameter to /var/run/bes/cache you might set HDF4.CacheDir to /var/run/bes/hdf4_cache. A configuration edition helper script, `bes-hdf4-data.sh' is provided in this package for easy configuration of the Hyrax BES server, designed to edit bes.conf. The script is called using: <code> bes-hdf4-data.sh [<bes.conf file to modify> [<bes modules dir>]] </code> The `bes-conf' make target runs the script while trying to select paths cleverly, and should be called using: <code> make bes-conf </code> Test data are also installed, so after installing this handler, Hyrax will have data to serve providing an easy way to test your new installation and to see how a working bes.conf should look. To use this, make sure that you first install the bes, and that dap-server gets installed too. Finally, every time you install or reinstall handlers, make sure to restart the BES and OLFS. This data handler is one component of the OPeNDAP DAP Server; the server base software is designed to allow any number of handlers to be configured easily. See the DAP Server README and INSTALL files for information about configuration, including how to use this handler. Copyright information: This software was originally written at JPL as part of the DODS NASA ESIP Federation Cooperative Agreement Notice. The original copyright described free use of the software 'for research purposes' although it was not clear what exactly those were. In Spring of 2003 we (OPeNDAP) sought clarification of that language and JPL/CalTech asked us to change the copyright to the copyright text now included in the code. In Fall of 2005 we decided to release the software under the LGPL, based on a previous discussion with personnel at JPL. James Gallagher Support for HDF data types: <code> HDF Version: This release of the server supports HDF4.2r1. It also supports reading/parsing the HDF-EOS attribute information which is then available to DAP clients. SDS: This is mapped to a DAP2 Grid (if it has a dimension scale) or Array (if it lacks a dim scale). Raster image: This is read via the HDF 4.0 General Raster interface and is mapped to Array. Each component of a raster is mapped to a new dimension labeled accordingly. For example, a 2-dimensional, 3-component raster is mapped to an m x n x 3 Array. Vdata: This is mapped to a Sequence, each element of which is a Structure. Each subfield of the Vdata is mapped to an element of the Structure. Thus a Vdata with one field of order 3 would be mapped to a Sequence of 1 Structure containing 3 base types. Note: Even though these appear as Sequences, the data handler does not support applying relational constraints to them. You can use the array notation to request a range of elements. Attributes: HDF attributes on SDS, rasters are straight-forwardly mapped to DAP attributes (HDF doesn't yet support Vdata attributes). File attributes (both SDS, raster) are mapped as attributes of a DAP variable called "HDF_GLOBAL" (by analogy to the way DAP handles netCDF global attributes, i.e., attaching them to "NC_GLOBAL"). Annotations: HDF file annotations mapped in the DAP to attribute values of type "String" attached to the fake DAP variable named "HDF_ANNOT". HDF annotations on objects are currently not read by the server. Vgroups: Vgroups are straight-forwardly mapped to Structures. </code> Special characters in HDF identifiers: A number of non-alphanumeric characters (e.g., space, #, +, -) used in HDF identifiers are not allowed in the names of DAP objects, object components or in URLs. The HDF4 data handler therefore deals internally with translated versions of these identifiers. To translate the WWW convention of escaping such characters by replacing them with "%" followed by the hexadecimal value of their ASCII code is used. For example, "Raster Image #1" becomes "Raster%20Image%20%231". These translations should be transparent to users of the server (but they will be visible in the DDS, DAS and in any applications which use a client that does not translate the identifiers back to their original form). Known problems: Handling of floating point attributes: Because the DAP software encodes attribute values as ASCII strings there will be a loss of accuracy for floating point attributes. This loss of accuracy is dependent on the version of the C++ I/O library used in compiling/linking the software (i.e., the amount of floating point precision preserved when outputting to ASCII is dependent on the library). Typically it is very small (e.g., at least six decimal places are preserved). Handling of global attributes: - The server will merge the separate global attributes for the SD, GR interfaces with any file annotations into one set of global attributes. These will then be available through any of the global attribute access functions. - If the client opens a constrained dataset (e.g., in SDstart), any global attributes of the unconstrained dataset will not be accessible because the constraint creates a "virtual dataset" which is a subset of the original unconstrained dataset. Todd Karakashian (Todd.K.Karakashian at jpl.nasa.gov) Isaac Henry (ike at seanet.jpl.nasa.gov) Jake Hamby (Jake.Hamby at jpl.nasa.gov) NASA/JPL April 1998
About
BES module to read hdf4 files
Resources
License
Stars
Watchers
Forks
Packages 0
No packages published
Languages
- C++ 86.8%
- C 10.2%
- M4 1.6%
- Makefile 0.6%
- Shell 0.5%
- Lex 0.2%
- Other 0.1%