From 78f259ab48e24710fe3b9195396b49282e7e3e7e Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Tue, 30 Jul 2024 13:51:32 -0600 Subject: [PATCH] Provide Documentation for the .rc File Mechanism and API re: Issue https://github.com/Unidata/netcdf-c/issues/2952 The documentation -- such as it is -- is lost inside docs/auth.md. So, create docs/quickstart_env.md to describe how .rc file mechanism is used, and include the netcdf-c API. Also add list of environment variables used by the netcdf-c library. --- RELEASE_NOTES.md | 9 +-- docs/CMakeLists.txt | 2 +- docs/Doxyfile.in | 1 + docs/Makefile.am | 2 +- docs/auth.md | 61 +--------------- docs/quickstart_env.md | 155 +++++++++++++++++++++++++++++++++++++++++ libdispatch/drc.c | 9 ++- 7 files changed, 173 insertions(+), 66 deletions(-) create mode 100644 docs/quickstart_env.md diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 12bfa7bda7..b29e7babf5 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -7,11 +7,12 @@ This file contains a high-level description of this package's evolution. Release ## 4.9.3 - TBD -* Convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys. The price to be paid is that lazy attribute reading cannot be supported. See [Github #2836](https://github.com/Unidata/netcdf-c/issues/2936) for more information. -* Cleanup the option code for NETCDF_ENABLE_SET_LOG_LEVEL\[_FUNC\] See [Github #2931](https://github.com/Unidata/netcdf-c/issues/2931) for more information. +* Provide better documentation for the .rc file mechanism and API. See [Github #????](https://github.com/Unidata/netcdf-c/pull/????) for more information. +* Convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys. The price to be paid is that lazy attribute reading cannot be supported. See [Github #2836](https://github.com/Unidata/netcdf-c/pull/2936) for more information. +* Cleanup the option code for NETCDF_ENABLE_SET_LOG_LEVEL\[_FUNC\] See [Github #2931](https://github.com/Unidata/netcdf-c/pull/2931) for more information. * Fix duplicate definition when using aws-sdk-cpp. See [Github #2928](https://github.com/Unidata/netcdf-c/issues/2928) for more information. -* Cleanup various obsolete options and do some code refactoring. See [Github #2926](https://github.com/Unidata/netcdf-c/issues/2926) for more information. -* Convert the Zarr-related ENABLE_XXX options to NETCDF_ENABLE_XXX options (part of the cmake overhaul). See [Github #2923](https://github.com/Unidata/netcdf-c/issues/2923) for more information. +* Cleanup various obsolete options and do some code refactoring. See [Github #2926](https://github.com/Unidata/netcdf-c/pull/2926) for more information. +* Convert the Zarr-related ENABLE_XXX options to NETCDF_ENABLE_XXX options (part of the cmake overhaul). See [Github #2923](https://github.com/Unidata/netcdf-c/pull/2923) for more information. * Refactor macro `_FillValue` to `NC_FillValue` to avoid conflict with libc++ headers. See [Github #2858](https://github.com/Unidata/netcdf-c/issues/2858) for more information. * Changed `cmake` build options to be prefaced with `NETCDF`, to bring things in to line with best practices. This will permit a number of overall quality of life improvements to netCDF, in terms of allowing it to be more easily integrated with upstream projects via `FetchContent()`, `subdirectory()`, etc. Currently, the naming convention in use thus far will still work, but will result in warning messages about deprecation, and instructions on how to update your workflow. See [Github #2895](https://github.com/Unidata/netcdf-c/pull/2895) for more information. * Incorporate a more modern look and feel to user documentation generated by Doxygen. See [Doxygen Awesome CSS](https://github.com/jothepro/doxygen-awesome-css) and [Github #2864](https://github.com/Unidata/netcdf-c/pull/2864) for more information. diff --git a/docs/CMakeLists.txt b/docs/CMakeLists.txt index 50f9285c19..c82b1b0190 100644 --- a/docs/CMakeLists.txt +++ b/docs/CMakeLists.txt @@ -96,6 +96,6 @@ inmemory.md DAP2.dox FAQ.md known_problems.md COPYRIGHT.dox user_defined_formats.md DAP4.md DAP4.dox testserver.dox byterange.md filters.md nczarr.md auth.md quantize.md -quickstart_paths.md cloud.md) +quickstart_paths.md quickstart_filters.md quickstart_env.md cloud.md) ADD_EXTRA_DIST("${CUR_EXTRA_DIST}") diff --git a/docs/Doxyfile.in b/docs/Doxyfile.in index c93fe1a9cd..87adb8c6c1 100644 --- a/docs/Doxyfile.in +++ b/docs/Doxyfile.in @@ -965,6 +965,7 @@ INPUT = @abs_top_srcdir@/docs/mainpage.dox \ @abs_top_srcdir@/docs/filters.md \ @abs_top_srcdir@/docs/quickstart_filters.md \ @abs_top_srcdir@/docs/quickstart_paths.md \ + @abs_top_srcdir@/docs/quickstart_env.md \ @abs_top_srcdir@/include/netcdf.h \ @abs_top_srcdir@/include/netcdf_mem.h \ @abs_top_srcdir@/include/netcdf_par.h \ diff --git a/docs/Makefile.am b/docs/Makefile.am index ba3e41cbb7..c445d5e1e2 100644 --- a/docs/Makefile.am +++ b/docs/Makefile.am @@ -15,7 +15,7 @@ obsolete/fan_utils.html indexing.dox inmemory.md FAQ.md \ known_problems.md COPYRIGHT.md inmeminternal.dox testserver.dox \ byterange.md nczarr.md quantize.md all-error-codes.md \ quickstart_paths.md cloud.md header.html attribute_conventions.md \ -file_format_specifications.md quickstart_filters.md \ +file_format_specifications.md quickstart_filters.md quickstart_env.md \ doxygen-awesome-css netcdf-50x50.png # Turn off parallel builds in this directory. diff --git a/docs/auth.md b/docs/auth.md index dfb49b3265..d15e60a1bc 100644 --- a/docs/auth.md +++ b/docs/auth.md @@ -44,67 +44,12 @@ how to properly escape the user and password. ## RC File Authentication {#auth_dodsrc} The netcdf library supports an _rc_ file mechanism to allow the passing -of a number of parameters to libnetcdf and libcurl. -Locating the _rc_ file is a multi-step process. - -### Search Order - -The netcdf-c library searches for, and loads from, the following files, -in this order: -1. $HOME/.ncrc -2. $HOME/.dodsrc -3. $CWD/.ncrc -4. $CWD/.dodsrc - -*$HOME* is the user's home directory and *$CWD* is the current working directory. Entries in later files override any of the earlier files - -It is strongly suggested that you pick a uniform location and a uniform name -and use them always. Otherwise you may observe unexpected results -when the netcdf-c library loads an rc file you did not expect. - -### RC File Format - -The rc file format is a series of lines of the general form: - - []= - -where the bracket-enclosed host:port is optional. - -### URL Constrained RC File Entries - -Each line of the rc file can begin with a host+port enclosed in -square brackets. The form is "host:port". If the port is not -specified then the form is just "host". The reason that more of -the url is not used is that libcurl's authorization grain is not -any finer than host level. - -Here are some examples. -```` - [remotetest.unidata.ucar.edu]HTTP.VERBOSE=1 -or - [fake.ucar.edu:9090]HTTP.VERBOSE=0 -```` -If the url request from, say, the _netcdf_open_ method -has a host+port matching one of the prefixes in the rc file, then -the corresponding entry will be used, otherwise ignored. -This means that an entry with a matching host+port will take -precedence over an entry without a host+port. - -For example, the URL -```` - http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc -```` -will have HTTP.VERBOSE set to 1 because its host matches the example above. - -Similarly, -```` - http://fake.ucar.edu:9090/dts/test.01 -```` -will have HTTP.VERBOSE set to 0 because its host+port matches the example above. +of a number of run-time parameters to libnetcdf and libcurl. +This is described in the file "quickstart_env.md". ## Authorization-Related Keys {#auth_keys} -The currently defined set of authorization-related keys are as follows. +The currently defined set of authorization-related keys recognized in .netrc are as follows. The second column is the affected curl_easy_setopt option(s), if any (see reference #1). diff --git a/docs/quickstart_env.md b/docs/quickstart_env.md new file mode 100644 index 0000000000..9d0f4f812e --- /dev/null +++ b/docs/quickstart_env.md @@ -0,0 +1,155 @@ +Appendix D.3. Environment Variables and .RC files {#nc_env_quickstart} +============================== + +The netCDF-c library provides several parameterization mechanisms to +control its behavior at run-time. The term _run-time_ means that the +library's behavior can be changed every time the library is initialized + +The most commonly used parameterization mechanisms are: +1. Environment Variables -- accessed by the getenv() function. +2. .RC files -- accessed from the file system. + +## Environment Variables {#nc_env_vars} + +The following table describes (most of) the environment variables +used by the netcdf-c library. There are some not listed that are only +used for specialized debugging. + +
+
NameDescription +
ALLUSERSPROFILEThis is more-or-less the Windows equivalent of "HOME" +
AWS_ACCESS_KEY_IDUsed by the aws access libraries; overrides ${HOME}/.aws/config entries. +
AWS_CONFIG_FILEUsed by the aws access libraries; specifies absolute path to the config file. +
AWS_DEFAULT_REGIONUsed by the aws access libraries; specifies default S3 region. +
AWS_PROFILEUsed by the aws access libraries; specifies default AWS profile. +
AWS_REGIONUsed by the aws access libraries; specifies specific region to use. +
AWS_SECRET_ACCESS_KEYUsed by the aws access libraries; overrides ${HOME}/.aws/config entries. +
CURLOPT_VERBOSECauses libcurl to produce verbose output. +
HOMEThe user's home directory. +
MSYS2_PREFIXIf platform is MSYS2, then specify the root prefix. +
NC_DEFAULT_CREATE_PERMSFor NCZarr, specify the default creation permissions for a file. +
NC_DEFAULT_DIR_PERMSFor NCZarr, specify the default creation permissions for a directory. +
NCLOGGINGSpecify the log level: one of "OFF","ERR","WARN","NOTE","DEBUG". +
NCPATHDEBUGCauses path manager to output debugging information. +
NCRCENV_HOMEOverrides ${HOME} as the location of the .rc file. +
NCRCENV_IGNORECauses any .rc files to be ignored. +
NCRCENV_RCThe absolute path to use for the .rc file. +
NCTRACINGSpecify the level of tracing detail. +
NCZARRFORMATForce use of a specific Zarr format version: 2 or 3. +
NETCDF_LOG_LEVELSpecify the log level for HDF5 logging (separate from e.g. NCLOGGING). +
TEMPFor Windows platform, specifies the location of a directory to store temporary files. +
USERPROFILEFor Windows platform, overrides ${HOME}. +
+ +## Resource Control (.rc) Files {#nc_env_rc} + +In addition to using environment variables, +the netcdf-c library supports run-time configuration +of the library using the so-called ".rc" file mechanism. +This means that as part of its initialization, the netcdf-c +library will search for and process a set of files where +these files contain entries specifying (key,value) pairs. +These pairs are compiled into a single internal database +that can be queried by other parts of the netcdf-c library. + +### Locating The _.rc_ Files + +For historical reasons, multiple .rc files are allowed. + +### Search Order + +The netcdf-c library searches for, and loads from, the following files, +in this order: +1. $HOME/.ncrc +2. $HOME/.dodsrc +3. $CWD/.ncrc +4. $CWD/.dodsrc + +*$HOME* is the user's home directory and *$CWD* is the current working directory. +Entries in later files override any of the earlier files + +It is strongly suggested that you pick a uniform location and a uniform name +and use them always. Otherwise you may observe unexpected results +when the netcdf-c library loads an rc file you did not expect. + +### RC File Format + +The rc file format is a series of lines of the general form: +```` + []= +```` +where the bracket-enclosed URL is optional. Note that the brackets +are part of the line. + +### URL Constrained RC File Entries + +Each line of the rc file can begin with a URL enclosed in +square brackets. The parts of the URL that are used for choosing +an entry are the host, the port, and the URL path. +Note that the host+port is the only part used when searching for +libcurl related entries. This is because libcurl's authorization grain is not +any finer than host+port level. +The URL path may be used for non-curl related entries. +Also note that the protocol is ignored. + +Here are some examples. +```` + [https://remotetest.unidata.ucar.edu/thredds]HTTP.VERBOSE=1 +or + [https://fake.ucar.edu:9090]HTTP.VERBOSE=0 +```` + +For selection purposes, the host+port+path is used when the path argument +for _nc_open()_ or _nc_create()_ takes the form of a URL. +If the url request from, say, the _netcdf_open_ method +has a host,port, and path matching one of the prefixes in the rc file, then +the corresponding entry will be used, otherwise ignored. +This means that an entry with a matching host+port+path will take +precedence over an entry without a host+port+path. + +For example, passing this URL to _nc_open_ +```` + http://remotetest.unidata.ucar.edu/thredds/dodsC/testdata/testData.nc +```` +will have HTTP.VERBOSE set to 1 because its host and path match the example above. + +Similarly, using this path +```` + http://fake.ucar.edu:9090/dts/test.01 +```` +will have HTTP.VERBOSE set to 0 because its host+port matches the example above. + +### Programmatic Access to .rc File + +It is possible for client programs to query and modify the internal .rc database +through the following API. +* ````char* nc_rc_get(const char* key);```` + Get the value corresponding to key or return NULL if not defined. The caller must free the resulting value. +* ````int nc_rc_set(const char* key, const char* value);```` + Set/overwrite the value corresponding to the specified key. + +Note that this API does not (currently) support URL prefixed keys, so the client will need to take this into account. + +### Defined .rc file keys + +There a a number of keys used by the netcdf-c library. Most of them +are authorization-related. The file "auth.md" describes these keys. + +Other keys are as follows: +* libdap4/d4curlfunctions.c and oc2/ocinternal.c + - HTTP.READ.BUFFERSIZE -- set the read buffer size for DAP2/4 connection + - HTTP.KEEPALIVE -- turn on keep-alive for DAP2/4 connection +* libdispatch/ds3util.c + - AWS.PROFILE -- alternate way to specify the default AWS profile + - AWS.REGION -- alternate way to specify the default AWS region +* libnczarr/zinternal.c + - ZARR.DIMENSION_SEPARATOR -- alternate way to specify the Zarr dimension separator character +* oc2/occurlfunctions.c + - HTTP.NETRC -- alternate way to specify the path of the .netrc file + +## Point of Contact {#nc_env_poc} + +__Author__: Dennis Heimbigner
+__Email__: dmh at ucar dot edu
+__Initial Version__: 01/09/2023
+__Last Revised__: 07/30/2024 diff --git a/libdispatch/drc.c b/libdispatch/drc.c index 9bfaa35a7c..351a5a4777 100644 --- a/libdispatch/drc.c +++ b/libdispatch/drc.c @@ -71,6 +71,7 @@ where the .rc entry has the simple form "key=value". If that entry is not found, then return NULL. @param key table entry key field +@param table entry key field @return value matching the key -- caller frees @return NULL if no entry of the form key=value exists */ @@ -269,8 +270,12 @@ NC_rcload(void) } /** - * Locate a entry by property key and host+port (may be null) + * Locate a entry by property key. * If duplicate keys, first takes precedence. + * @param key to lookup + * @param hostport to use for lookup + * @param urlpath to use for lookup + * @return the value of the key or NULL if not found. */ char* NC_rclookup(const char* key, const char* hostport, const char* urlpath) @@ -468,7 +473,7 @@ rccompile(const char* filepath) } line = rtag + 1; *rtag = '\0'; - /* compile the url and pull out the host and protocol */ + /* compile the url and pull out the host, port, and path */ if(uri) ncurifree(uri); if(ncuriparse(url,&uri)) { nclog(NCLOGERR, "Malformed [url] in %s entry: %s",filepath,line);