Fix more memory leaks in netcdf-c library

This is a follow up to PR #1173

Sorry that it is so big, but leak suppression can be complex.

This PR fixes all remaining memory leaks -- as determined by
-fsanitize=address, and with the exceptions noted below.

Unfortunately. there remains a significant leak that I cannot
solve. It involves vlens, and it is unclear if the leak is
occurring in the netcdf-c library or the HDF5 library.

I have added a check_PROGRAM to the ncdump directory to show the
problem.  The program is called tst_vlen_demo.c To exercise it,
build the netcdf library with -fsanitize=address enabled. Then
go into ncdump and do a "make clean check".  This should build
tst_vlen_demo without actually executing it.  Then do the
command "./tst_vlen_demo" to see the output of the memory
checker.  Note the the lost malloc is deep in the HDF5 library
(in H5Tvlen.c).

I am temporarily working around this error in the following way.
1. I modified several test scripts to not execute known vlen tests
   that fail as described above.
2. Added an environment variable called NC_VLEN_NOTEST.
   If set, then those specific tests are suppressed.

This should mean that the --disable-utilities option to
./configure should not need to be set to get a memory leak clean
build.  This should allow for detection of any new leaks.

Note: I used an environment variable rather than a ./configure
option to control the vlen tests. This is because it is
temporary (I hope) and because it is a bit tricky for shell
scripts to access ./configure options.

Finally, as before, this only been tested with netcdf-4 and hdf5 support.

dap4_test/baselineremote/test_atomic_array.syn.dmp

@@ -49,4 +49,5 @@ data:
 
  primary_cloud = Stratus, Cirrocumulus, Nimbostratus, Cirrostratus, 
     Stratocumulus ;
+
 }

docs/CMakeLists.txt

@@ -84,6 +84,6 @@ install-fortran.md all-error-codes.md credits.md auth.md
 obsolete/fan_utils.html bestpractices.md filters.md indexing.md
 inmemory.md DAP2.dox attribute_conventions.md FAQ.md
 file_format_specifications.md known_problems.md
-COPYRIGHT.dox user_defined_formats.md)
+COPYRIGHT.dox user_defined_formats.md DAP4.md DAP4.dox)
 
 ADD_EXTRA_DIST("${CUR_EXTRA_DIST}")

docs/DAP4.dox

@@ -1,55 +1,59 @@
-/*! \page dap4 DAP4 Support
+/*!
+\page dap4 DAP4 Protocol Support
 
-\section dap4_accessing The DAP4 Prototocol
+
+<!-- Note that this file has the .dox extension, but is mostly markdown -->
+<!-- Begin MarkDown -->
 
 \tableofcontents
 
-\subsection dap4_introduction DAP4 Introduction
+# DAP4 Introduction {#dap4_introduction}
 
-Beginning with netCDF version 4.2.2, optional support is provided for
-accessing data through DAP4 servers using the DAP4 protocol.
+Beginning with netCDF version 4.5.0, optional support is provided for
+accessing data from servers supporting the DAP4 protocol.
 
-DAP4 support is automatically enabled if a usable curl library can be
-set using the LDFLAGS environment variable (similar to the way
-that the HDF5 libraries are referenced).
-DAP2 plus DAP4 support can forcibly be enabled or disabled using the --enable-dap
-flag or the --disable-dap flag, respectively.  If enabled,
-then DAP4 support requires access to the curl library.
-Refer to the installation manual for details
+DAP4 support is enabled if the _--enable-dap_ option
+is used with './configure'. If DAP4 support is enabled, then
+a usable version of _libcurl_ must be specified
+using the _LDFLAGS_ environment variable (similar to the way
+that the _HDF5_ libraries are referenced).
+Refer to the installation manual for details.
+By default DAP4 support is enabled if _libcurl_ is found.
+DAP4 support can be disabled using the _--disable-dap_.
 
-DAP4 uses a data model that is similar to, but not identical with,
+DAP4 uses a data model that is, by design, similar to,
+but -- for historical reasons -- not identical with,
 the netCDF-Enhanced (aka netcdf-4) model.
 Generically, the DAP4 data model is encoded in XML document
 called a DMR.
 For detailed information about the DAP4 DMR, refer to
 the DAP4 specification Volume 1:
 http://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
 
-\subsection dap4_access Accessing Data Using the DAP4 Prototocol
+# Accessing Data Using the DAP4 Prototocol {#dap4_accessing_data}
 
 In order to access a DAP4 data source through the netCDF API, the
 file name normally used is replaced with a URL with a specific
 format. The URL is composed of three parts.
- -  URL - this is a standard form URL with specific markers to indicate that
-    it refers to a DAP4 encoded dataset. The markers can be one of the following
-    examples.
-    +  [dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
-    +  [protocol=dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
-    +  http://remotetest.unidata.ucar.edu/d4ts/test.01#dap4
-    +  http://remotetest.unidata.ucar.edu/d4ts/test.01#protocol=dap4
-    +  dap4://remotetest.unidata.ucar.edu/d4ts/test.01
-
-    Note that the last case is converted internally to one of the other forms.
+-  URL - this is a standard form URL with specific markers to indicate that
+   it refers to a DAP4 encoded dataset. The markers are of the form
+   "dap4", "protocol=dap4", or "/thredds/dap4". The following
+   examples show how they are specified. Note that the "/thredds/dap4"
+   case will only work when accessing a Thredds-based server.
+   +  [dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
+   +  [protocol=dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01
+   +  http://remotetest.unidata.ucar.edu/d4ts/test.01#dap4
+   +  http://remotetest.unidata.ucar.edu/d4ts/test.01#protocol=dap4
+   +  http://thredds.ucar.edu/thredds/dap4/...
 
  -  Constraints - these are suffixed to the URL and take the form
     “?dap4.ce=\<expression\>”. The form of the constraint expression
     is somewhat complicated, and the specification should be consulted.
 
-
  - Client parameters - these may be specified in either of
    two ways.  The older, deprecated form prefixes text to the
    front of the url and is of the the general form [\<name>]
-   or [\<name>=value].  Examples include [show=fetch].
+   or [\<name>=value].  Examples include [show=fetch] or [noprefetch].
    The newer, preferred form prefixes the
    parameters to the end of the url using the semi-standard '#'
    format: e.g. http://....#show=fetch&noprefetch.
@@ -61,19 +65,19 @@ the translation using the "ncdump -h" command to see the
 corresponding netCDF-4 representation.
 
 For example, if a web browser is given the following (fictional) URL,
-it will return the DMR for the specified dataset
+it will return the DMR for the specified dataset (in XML format).
 \code
-     http://remotetest.unidata.ucar.edu/d4ts/test.01.dmr#dap4
+     http://remotetest.unidata.ucar.edu/d4ts/test.01.dmr.xml#dap4
 \endcode
 
 By using the following ncdump command, it is possible to see the
 equivalent netCDF-4 translation.
 
 \code
-     ncdump -h http://remotetest.unidata.ucar.edu/d4ts/test.01#dap4
+     ncdump -h '[dap4]http://remotetest.unidata.ucar.edu/d4ts/test.01'
 \endcode
 
-\subsection dap4_defined_params Defined Client Parameters
+# Defined Client Parameters {#dap4_defined_params}
 
 Currently, a limited set of client parameters is
 recognized. Parameters not listed here are
@@ -87,12 +91,23 @@ Parameter Name Legal Values Semantics
   of the complete url for every HTTP get request. If logging is
   enabled, then this can be helpful in checking to see the access
   behavior of the netCDF code.
-
-\subsection dap4_debug Notes on Debugging DAP4 Access
+- "translate=nc4" - This parameter causes the netCDF code to look
+  for specially named elements in the DMR XML in order to
+  achieve a better translation of the DAP4 meta-data to NetCDF enhanced
+  metadata.
+- "opaquesize=<integer>" - This parameter causes the netCDF code to 
+  convert DAP4 variable size OPAQUE objects to netcdf-4 fixed size
+  objects and forces all of them to be of the size specified.
+- "fillmismatch" - Unfortunately, a number of servers sometimes
+  fail to make sure that the type of the "_FillValue" attribute of a variable
+  is the same as the type of the containing variable. Setting this tag
+  caused the netcdf translation to attempt to fix this mismatch. If not set,
+  then an error will occur.
+
+# Notes on Debugging DAP4 Access {#dap4_debug}
 
 The DAP4 support has a logging facility.
-Note that this is currently separate from the
-existing netCDF logging facility. Turning on this logging can
+Turning on this logging can
 sometimes give important information. Logging can be enabled by
 using the client parameter "log" or "log=filename",
 where the first case will send log output to standard error and the
@@ -102,4 +117,138 @@ Users should also be aware that if one is
 accessing data over an NFS mount, one may see some .nfsxxxxx files;
 those can be ignored.
 
+## HTTP Configuration. {#dap4_http2_config}
+
+Limited support for configuring the http connection is provided via
+parameters in the “.daprc” configuration file (aka ".dodsrc").
+The relevant .daprc file is
+located by first looking in the current working directory, and if not
+found, then looking in the directory specified by the “$HOME”
+environment variable.
+
+Entries in the .daprc file are of the form:
+````
+     ['['<url>']']<key>=<value>
+````
+
+That is, it consists of a key name and value pair and optionally
+preceded by a url enclosed in square brackets.
+
+For given KEY and URL strings, the value chosen is as follows:
+
+If URL is null, then look for the .daprc entry that has no url prefix
+and whose key is same as the KEY for which we are looking.
+
+If the URL is not null, then look for all the .daprc entries that
+have a url, URL1, say, and for which URL1 has the same host and port
+as URL. All parts of the url's except host and port are ignored.
+For example, if URL = http//x.y/a, then it will match
+entries of the form
+_[http//x.y/a]KEY=VALUE_ or _[http//x.y/b]KEY=VALUE_.
+It will not match an entry of the form _[http//x.y:8080]KEY=VALUE
+because the second has a port number (8080) different than the URL.
+Finally from the set so constructed, choose the first matching entry.
+
+Currently, the supported set of keys (with descriptions) are as
+follows.
+
+-# HTTP.VERBOSE  
+        Type: boolean ("1"/"0")  
+        Description: Produce verbose output, especially using SSL.  
+        Related CURL Flags: CURLOPT_VERBOSE  
+
+-# HTTP.DEFLATE  
+        Type: boolean ("1"/"0")  
+        Description: Allow use of compression by the server.  
+        Related CURL Flags: CURLOPT_ENCODING  
+
+-# HTTP.COOKIEJAR  
+        Type: String representing file path  
+        Description: Specify the name of file into which to store cookies. Defaults to in-memory storage.  
+        Related CURL Flags:CURLOPT_COOKIEJAR  
+
+-# HTTP.CREDENTIALS.USER  
+        Type: String representing user name  
+        Description: Specify the user name for Digest and Basic authentication.  
+        Related CURL Flags:  
+
+-# HTTP.CREDENTIALS.PASSWORD  
+        Type: String representing password  
+        Type: boolean ("1"/"0")  
+        Description: Specify the password for Digest and Basic authentication.  
+        Related CURL Flags:  
+
+-# HTTP.SSL.CERTIFICATE  
+        Type: String representing file path  
+        Description: Path to a file containing a PEM cerficate.  
+        Related CURL Flags: CURLOPT_CERT  
+
+-# HTTP.SSL.KEY  
+        Type: String representing file path  
+        Description: Same as HTTP.SSL.CERTIFICATE, and should usually have the same value.  
+        Related CURL Flags: CURLOPT_SSLKEY  
+
+-# HTTP.SSL.KEYPASSWORD  
+        Type: String representing password  
+        Description: Password for accessing the HTTP.SSL.KEY/HTTP.SSL.CERTIFICATE  
+        Related CURL Flags: CURLOPT_KEYPASSWORD  
+
+-# HTTP.SSL.CAPATH  
+        Type: String representing directory  
+        Description: Path to a directory containing trusted certificates for validating server certificates.  
+        Related CURL Flags: CURLOPT_CAPATH  
+
+-# HTTP.SSL.VALIDATE  
+        Type: boolean ("1"/"0")  
+        Description: Cause the client to verify the server's presented certificate.  
+        Related CURL Flags: CURLOPT_SSL_VERIFYPEER, CURLOPT_SSL_VERIFYHOST  
+
+-# HTTP.TIMEOUT  
+        Type: String ("dddddd")  
+        Description: Specify the maximum time in seconds that you allow the http transfer operation to take.  
+        Related CURL Flags: CURLOPT_TIMEOUT, CURLOPT_NOSIGNAL  
+
+-# HTTP.PROXY_SERVER  
+        Type: String representing url to access the proxy: (e.g.http://[username:password@]host[:port])  
+        Description: Specify the needed information for accessing a proxy.  
+        Related CURL Flags: CURLOPT_PROXY, CURLOPT_PROXYHOST, CURLOPT_PROXYUSERPWD  
+
+-# HTTP.READ.BUFFERSIZE  
+        Type: String ("dddddd")  
+        Description: Specify the the internal buffer size for curl reads.  
+        Related CURL Flags: CURLOPT_BUFFERSIZE, CURL_MAX_WRITE_SIZE (16kB),  
+                            CURL_MAX_READ_SIZE (512kB).  
+
+-# HTTP.KEEPALIVE  
+        Type: String ("on|n/m")  
+        Description: Specify that TCP KEEPALIVE should be enabled and that the associated idle wait time is n and that the associated repeat interval is m. If the value is of the form is the string "on", then turn on keepalive, but do not set idle or interval.  
+        Related CURL Flags: CURLOPT_TCP_KEEPALIVE, CURLOPT_TCP_KEEPIDLE,  
+                            CURLOPT_TCP_KEEPINTVL.  
+
+The related curl flags line indicates the curl flags modified by this
+key. See the libcurl documentation of the _curl_easy_setopt()_ function
+for more detail (http://curl.haxx.se/libcurl/c/curl_easy_setopt.html).
+
+For servers that require client authentication, as a rule, the following
+.daprc entries should be specified.
+````
+HTTP.SSL.VALIDATE
+HTTP.COOKIEJAR
+HTTP.SSL.CERTIFICATE
+HTTP.SSL.KEY
+HTTP.SSL.CAPATH
+````
+
+Additionally, the _HTTP.SSL.CERTIFICATE_ and _HTTP.SSL.KEY_
+entries should have same value, which is the file path for a
+certificate. The HTTP.SSL.CAPATH entry should
+be the path to a directory containing validation "certificates".
+
+# Point of Contact {#dap4_poc}
+
+__Author__: Dennis Heimbigner<br>
+__Email__: dmh at ucar dot edu<br>
+__Initial Version__: 6/5/2017<br>
+__Last Revised__: 11/7/2018
+
 */