Provide Documentation for the .rc File Mechanism and API

RELEASE_NOTES.md

@@ -7,14 +7,17 @@ This file contains a high-level description of this package's evolution. Release
 
 ## 4.9.3 - TBD
 
+* Provide better documentation for the .rc file mechanism and API. See [Github #????](https://github.com/Unidata/netcdf-c/pull/????) for more information.
+* Convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys. The price to be paid is that lazy attribute reading cannot be supported. See [Github #2836](https://github.com/Unidata/netcdf-c/pull/2936) for more information.
+* Cleanup the option code for NETCDF_ENABLE_SET_LOG_LEVEL\[_FUNC\] See [Github #2931](https://github.com/Unidata/netcdf-c/pull/2931) for more information.
+* 
 ### Release Candidate 1 - July 26, 2024
 
 * Convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys. The price to be paid is that lazy attribute reading cannot be supported. See [Github #2836](https://github.com/Unidata/netcdf-c/issues/2936) for more information.
-
 * Cleanup the option code for NETCDF_ENABLE_SET_LOG_LEVEL\[_FUNC\] See [Github #2931](https://github.com/Unidata/netcdf-c/issues/2931) for more information.
 * Fix duplicate definition when using aws-sdk-cpp. See [Github #2928](https://github.com/Unidata/netcdf-c/issues/2928) for more information.
-* Cleanup various obsolete options and do some code refactoring. See [Github #2926](https://github.com/Unidata/netcdf-c/issues/2926) for more information.
-* Convert the Zarr-related ENABLE_XXX options to NETCDF_ENABLE_XXX options (part of the cmake overhaul). See [Github #2923](https://github.com/Unidata/netcdf-c/issues/2923) for more information.
+* Cleanup various obsolete options and do some code refactoring. See [Github #2926](https://github.com/Unidata/netcdf-c/pull/2926) for more information.
+* Convert the Zarr-related ENABLE_XXX options to NETCDF_ENABLE_XXX options (part of the cmake overhaul). See [Github #2923](https://github.com/Unidata/netcdf-c/pull/2923) for more information.
 * Refactor macro `_FillValue` to `NC_FillValue` to avoid conflict with libc++ headers. See [Github #2858](https://github.com/Unidata/netcdf-c/issues/2858) for more information.
 * Changed `cmake` build options to be prefaced with `NETCDF`, to bring things in to line with best practices.  This will permit a number of overall quality of life improvements to netCDF, in terms of allowing it to be more easily integrated with upstream projects via `FetchContent()`, `subdirectory()`, etc. Currently, the naming convention in use thus far will still work, but will result in warning messages about deprecation, and instructions on how to update your workflow. See [Github #2895](https://github.com/Unidata/netcdf-c/pull/2895) for more information.
 * Incorporate a more modern look and feel to user documentation generated by Doxygen.  See [Doxygen Awesome CSS](https://github.com/jothepro/doxygen-awesome-css) and [Github #2864](https://github.com/Unidata/netcdf-c/pull/2864) for more information.

docs/CMakeLists.txt

@@ -98,6 +98,6 @@ inmemory.md DAP2.dox FAQ.md
 known_problems.md
 COPYRIGHT.dox user_defined_formats.md DAP4.md DAP4.dox
 testserver.dox byterange.md filters.md nczarr.md auth.md quantize.md
-quickstart_paths.md cloud.md)
+quickstart_paths.md quickstart_filters.md quickstart_env.md cloud.md)
 
 ADD_EXTRA_DIST("${CUR_EXTRA_DIST}")

docs/Doxyfile.in

@@ -966,6 +966,7 @@ INPUT                  = @abs_top_srcdir@/docs/mainpage.dox \
                          @abs_top_srcdir@/docs/filters.md \
                          @abs_top_srcdir@/docs/quickstart_filters.md \
                          @abs_top_srcdir@/docs/quickstart_paths.md \
+                         @abs_top_srcdir@/docs/quickstart_env.md \
                          @abs_top_srcdir@/include/netcdf.h \
                          @abs_top_srcdir@/include/netcdf_mem.h \
                          @abs_top_srcdir@/include/netcdf_par.h \

docs/Makefile.am

@@ -15,7 +15,7 @@ obsolete/fan_utils.html indexing.dox inmemory.md FAQ.md			\
 known_problems.md COPYRIGHT.md inmeminternal.dox testserver.dox		\
 byterange.md nczarr.md quantize.md all-error-codes.md                   \
 quickstart_paths.md cloud.md header.html attribute_conventions.md \
-file_format_specifications.md quickstart_filters.md \
+file_format_specifications.md quickstart_filters.md quickstart_env.md \
 doxygen-awesome-css netcdf-50x50.png
 
 # Turn off parallel builds in this directory.

docs/auth.md

@@ -41,67 +41,12 @@ how to properly escape the user and password.
 
 ## RC File Authentication {#auth_dodsrc}
 The netcdf library supports an _rc_ file mechanism to allow the passing
-of a number of parameters to libnetcdf and libcurl.
-Locating the _rc_ file is a multi-step process.
-
-### Search Order
-
-The netcdf-c library searches for, and loads from, the following files,
-in this order:
-1. $HOME/.ncrc
-2. $HOME/.dodsrc
-3. $CWD/.ncrc
-4. $CWD/.dodsrc
-
-*$HOME* is the user's home directory and *$CWD* is the current working directory. Entries in later files override any of the earlier files
-
-It is strongly suggested that you pick a uniform location and a uniform name
-and use them always. Otherwise you may observe unexpected results
-when the netcdf-c library loads an rc file you did not expect.
-
-### RC File Format
-
-The rc file format is a series of lines of the general form:
-
-    [<host:port>]<key>=<value>
-
-where the bracket-enclosed host:port is optional.
-
-### URL Constrained RC File Entries
-
-Each line of the rc file can begin with a host+port enclosed in
-square brackets.  The form is "host:port".  If the port is not
-specified then the form is just "host".  The reason that more of
-the url is not used is that libcurl's authorization grain is not
-any finer than host level.
-
-Here are some examples.
-````
-    [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1
-or
-    [fake.ucar.edu:9090]HTTP.VERBOSE=0
-````
-If the url request from, say, the _netcdf_open_ method
-has a host+port matching one of the prefixes in the rc file, then
-the corresponding entry will be used, otherwise ignored.
-This means that an entry with a matching host+port will take
-precedence over an entry without a host+port.
-
-For example, the URL
-````
-    http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
-````
-will have HTTP.VERBOSE set to 1 because its host matches the example above.
-
-Similarly,
-````
-    http://fake.ucar.edu:9090/dts/test.01
-````
-will have HTTP.VERBOSE set to 0 because its host+port matches the example above.
+of a number of run-time parameters to libnetcdf and libcurl.
+This is described in the file "quickstart_env.md". 
 
 ## Authorization-Related Keys {#auth_keys}
 
-The currently defined set of authorization-related keys are as follows.
+The currently defined set of authorization-related keys recognized in .netrc are as follows.
 The second column is the affected curl_easy_setopt option(s), if any
 (see reference #1).
 <table>

docs/quickstart_env.md

@@ -0,0 +1,155 @@
+Appendix D.3. Environment Variables and .RC files {#nc_env_quickstart}
+==============================
+
+The netCDF-c library provides several parameterization mechanisms to
+control its behavior at run-time. The term _run-time_ means that the
+library's behavior can be changed every time the library is initialized
+
+The most commonly used parameterization mechanisms are:
+1. Environment Variables -- accessed by the getenv() function.
+2. .RC files -- accessed from the file system.
+
+## Environment Variables {#nc_env_vars}
+
+The following table describes (most of) the environment variables
+used by the netcdf-c library. There are some not listed that are only
+used for specialized debugging.
+
+<table>
+<tr><th>Name<th>Description
+<tr><td>ALLUSERSPROFILE<td>This is more-or-less the Windows equivalent of "HOME"
+<tr><td>AWS_ACCESS_KEY_ID<td>Used by the aws access libraries; overrides ${HOME}/.aws/config entries.
+<tr><td>AWS_CONFIG_FILE<td>Used by the aws access libraries; specifies absolute path to the config file.
+<tr><td>AWS_DEFAULT_REGION<td>Used by the aws access libraries; specifies default S3 region.
+<tr><td>AWS_PROFILE<td>Used by the aws access libraries; specifies default AWS profile.
+<tr><td>AWS_REGION<td>Used by the aws access libraries; specifies specific region to use.
+<tr><td>AWS_SECRET_ACCESS_KEY<td>Used by the aws access libraries; overrides ${HOME}/.aws/config entries.
+<tr><td>CURLOPT_VERBOSE<td>Causes libcurl to produce verbose output.
+<tr><td>HOME<td>The user's home directory.
+<tr><td>MSYS2_PREFIX<td>If platform is MSYS2, then specify the root prefix.
+<tr><td>NC_DEFAULT_CREATE_PERMS<td>For NCZarr, specify the default creation permissions for a file.
+<tr><td>NC_DEFAULT_DIR_PERMS<td>For NCZarr, specify the default creation permissions for a directory.
+<tr><td>NCLOGGING<td>Specify the log level: one of "OFF","ERR","WARN","NOTE","DEBUG".
+<tr><td>NCPATHDEBUG<td>Causes path manager to output debugging information.
+<tr><td>NCRCENV_HOME<td>Overrides ${HOME} as the location of the .rc file.
+<tr><td>NCRCENV_IGNORE<td>Causes any .rc files to be ignored.
+<tr><td>NCRCENV_RC<td>The absolute path to use for the .rc file.
+<tr><td>NCTRACING<td>Specify the level of tracing detail.
+<tr><td>NCZARRFORMAT<td>Force use of a specific Zarr format version: 2 or 3.
+<tr><td>NETCDF_LOG_LEVEL<td>Specify the log level for HDF5 logging (separate from e.g. NCLOGGING).
+<tr><td>TEMP<td>For Windows platform, specifies the location of a directory to store temporary files.
+<tr><td>USERPROFILE<td>For Windows platform, overrides ${HOME}.
+</table>
+
+## Resource Control (.rc) Files {#nc_env_rc}
+
+In addition to using environment variables,
+the netcdf-c library supports run-time configuration
+of the library using the so-called ".rc" file mechanism.
+This means that as part of its initialization, the netcdf-c
+library will search for and process a set of files where
+these files contain entries specifying (key,value) pairs.
+These pairs are compiled into a single internal database
+that can be queried by other parts of the netcdf-c library.
+
+### Locating The _.rc_ Files
+
+For historical reasons, multiple .rc files are allowed.
+
+### Search Order
+
+The netcdf-c library searches for, and loads from, the following files,
+in this order:
+1. $HOME/.ncrc
+2. $HOME/.dodsrc
+3. $CWD/.ncrc
+4. $CWD/.dodsrc
+
+*$HOME* is the user's home directory and *$CWD* is the current working directory.
+Entries in later files override any of the earlier files
+
+It is strongly suggested that you pick a uniform location and a uniform name
+and use them always. Otherwise you may observe unexpected results
+when the netcdf-c library loads an rc file you did not expect.
+
+### RC File Format
+
+The rc file format is a series of lines of the general form:
+````
+    [<URL>]<key>=<value>
+````
+where the bracket-enclosed URL is optional. Note that the brackets
+are part of the line.
+
+### URL Constrained RC File Entries
+
+Each line of the rc file can begin with a URL enclosed in
+square brackets. The parts of the URL that are used for choosing
+an entry are the host, the port, and the URL path.
+Note that the host+port is the only part used when searching for
+libcurl related entries. This is because libcurl's authorization grain is not
+any finer than host+port level.
+The URL path may be used for non-curl related entries.
+Also note that the protocol is ignored.
+
+Here are some examples.
+````
+    [https://remotetest.unidata.ucar.edu/thredds]HTTP.VERBOSE=1
+or
+    [https://fake.ucar.edu:9090]HTTP.VERBOSE=0
+````
+
+For selection purposes, the host+port+path is used when the path argument
+for _nc_open()_ or _nc_create()_ takes the form of a URL.
+If the url request from, say, the _netcdf_open_ method
+has a host,port, and path matching one of the prefixes in the rc file, then
+the corresponding entry will be used, otherwise ignored.
+This means that an entry with a matching host+port+path will take
+precedence over an entry without a host+port+path.
+
+For example, passing this URL to _nc_open_
+````
+    http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc
+````
+will have HTTP.VERBOSE set to 1 because its host and path match the example above.
+
+Similarly, using this path
+````
+    http://fake.ucar.edu:9090/dts/test.01
+````
+will have HTTP.VERBOSE set to 0 because its host+port matches the example above.
+
+### Programmatic Access to .rc File
+
+It is possible for client programs to query and modify the internal .rc database
+through the following API.
+* ````char* nc_rc_get(const char* key);````
+    Get the value corresponding to key or return NULL if not defined. The caller must free the resulting value.
+* ````int nc_rc_set(const char* key, const char* value);````
+    Set/overwrite the value corresponding to the specified key.
+
+Note that this API does not (currently) support URL prefixed keys, so the client will need to take this into account.
+
+### Defined .rc file keys
+
+There a a number of keys used by the netcdf-c library. Most of them
+are authorization-related. The file "auth.md" describes these keys.
+
+Other keys are as follows:
+* libdap4/d4curlfunctions.c and oc2/ocinternal.c
+    - HTTP.READ.BUFFERSIZE -- set the read buffer size for DAP2/4 connection
+    - HTTP.KEEPALIVE -- turn on keep-alive for DAP2/4 connection
+* libdispatch/ds3util.c
+    - AWS.PROFILE -- alternate way to specify the default AWS profile
+    - AWS.REGION --  alternate way to specify the default AWS region
+* libnczarr/zinternal.c
+    - ZARR.DIMENSION_SEPARATOR -- alternate way to specify the Zarr dimension separator character
+* oc2/occurlfunctions.c
+    - HTTP.NETRC -- alternate way to specify the path of the .netrc file
+
+## Point of Contact {#nc_env_poc}
+
+__Author__: Dennis Heimbigner<br>
+__Email__: dmh at ucar dot edu<br>
+__Initial Version__: 01/09/2023<br>
+__Last Revised__: 07/30/2024

libdispatch/drc.c

@@ -71,6 +71,7 @@ where the .rc entry has the simple form "key=value".
 If that entry is not found, then return NULL.
 
 @param key table entry key field
+@param  table entry key field
 @return value matching the key -- caller frees
 @return NULL if no entry of the form key=value exists
 */
@@ -269,8 +270,12 @@ NC_rcload(void)
 }
 
 /**
- * Locate a entry by property key and host+port (may be null)
+ * Locate a entry by property key.
  * If duplicate keys, first takes precedence.
+ * @param key to lookup
+ * @param hostport to use for lookup
+ * @param urlpath to use for lookup
+ * @return the value of the key or NULL if not found.
  */
 char*
 NC_rclookup(const char* key, const char* hostport, const char* urlpath)
@@ -468,7 +473,7 @@ rccompile(const char* filepath)
             }
             line = rtag + 1;
             *rtag = '\0';
-            /* compile the url and pull out the host and protocol */
+            /* compile the url and pull out the host, port, and path */
             if(uri) ncurifree(uri);
             if(ncuriparse(url,&uri)) {
                 nclog(NCLOGERR, "Malformed [url] in %s entry: %s",filepath,line);