doc: implementation: add configuration section to implementation guide

The recent addition of a formal configuration file and override
mechanism to ODP highlights the need to discuss implementation
configuration considerations. Add this section to the ODP Implementation
Guide and discuss the various types of static and dynamic configuration
options available, with illustrations from how they are used in
odp-linux.

Signed-off-by: Bill Fischofer <bill.fischofer@linaro.org>
Reviewed-by: Petri Savolainen <petri.savolainen@linaro.org>
Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>

doc/implementers-guide/implementers-guide.adoc

@@ -684,4 +684,167 @@ creating a managed binary is itself a secondary compilation and optimization
 step. The difference is that performing this step is a system rather than a
 developer responsibility.
 
+== Configuration
+Each ODP implementation will choose various sizes, limits, and similar
+internal parameters that are well matched to its design and platform
+capabilities. However, it is often useful to expose at least some of these
+parameters and allow users to select specific values to use either
+at compile time or runtime. This section discusses options for doing this,
+using the configuration options offered in the `odp-linux` reference
+implementation as an example.
+
+=== Static Configuration
+Static configuration requires the ODP implementation to be recompiled. The
+reasons for choosing static configuration vary but can involve both design
+simplicity (_e.g.,_ arrays can be statically configured) or performance
+considerations (_e.g.,_ including optional debug code). Two approaches to
+static configuration are `#define` statements and use of autotools.
+
+==== `#define` Statements
+Certain implementation limits can best be represented by `#define` statements
+that are set at compile time. Examples of this can be seen in the `odp-linux`
+reference implementation in the file
+`platform/linux-generic/include/odp_config_internal.h`.
+
+.Compile-time implementation limits (excerpt)
+[source,c]
+-----
+/*
+ * Maximum number of CPUs supported. Maximum CPU ID is CONFIG_NUM_CPU - 1.
+ */
+#define CONFIG_NUM_CPU 256
+
+/*
+ * Maximum number of pools
+ */
+#define ODP_CONFIG_POOLS 64
+-----
+
+Here two fundamental limits, the number of CPUs supported and the maximum
+number of pools that can be created via the `odp_pool_create()` API are
+defined. By using `#define`, the implementation can configure supporting
+structures (bit strings and arrays) statically, and can also allow static
+compile-time validation/consistency checks to be done using facilities like
+`ODP_STATIC_ASSERT()`. This results in more efficient code since these limits
+need not be computed at runtime.
+
+Users are able to change these limits (potentially within documented absolute
+bounds) by changing the relevant source file and recompiling that ODP
+implementation.
+
+==== Use of `autotools configure`
+The ODP reference implementation, like many open source projects, makes use of
+https://www.gnu.org/software/automake/faq/autotools-faq.html[autotools]
+to simplify project configuration and support for various build targets.
+These same tools permit compile-time configuration options to be specified
+without requiring changes to source files.
+
+In addition to the "standard" `configure` options for specifying prefixes,
+target install paths, etc., the `odp-linux` reference implementation supports
+a large number of static configuration options that control how ODP is
+built. Use the `./configure --help` command for a complete list. Here we
+discuss simply a few for illustrative purposes:
+
+`--enable-debug`::
+The ODP API specification simply says that "results are undefined" when
+invalid parameters are passed to ODP APIs. This is done for performance
+reasons so that implementations don't need to insert extraneous parameter
+checking that would impact runtime performance in fast-path operations. While
+this is a reasonable trade off, it can complicate application debugging.
+To address this, the ODP implementation makes use of the `ODP_ASSERT()` macro
+that by default disappears at compile time unless the `--enable-debug`
+configuration option was specified. Running with a debug build of ODP trades
+off performance for improved parameter/bounds checking to make application
+debugging easier.
+
+`--enable-user-guides`::
+By default, building ODP only builds the code. When this option is specified,
+the supporting user documentation (including this file) is also built.
+
+`--disable-abi-compat`::
+By default ODP builds with support for the ODP ABI, which permits application
+binary portability across different ODP implementations targeting the same
+Instruction Set Architecture (ISA). While this is useful in cloud/host
+environments, it does involve some performance cost to provide binary
+compatibility. For embedded use of ODP, disabling ABI compatibility means
+tighter code can be generated by inlining more of the ODP implementation
+into the calling application code. When built without ABI compatibility,
+moving an application to another ODP implementation requires that the
+application be recompiled. For most embedded uses this is a reasonable
+trade off in exchange for better application performance on a specific
+target platform.
+
+=== Dynamic Configuration
+While compile-time limits have the advantage of simplicity, they are also
+not very flexible since they require an ODP implementation to be regenerated
+to change them. The alternative is for implementations to support _dynamic
+configuration_ that enables ODP to change implementation behavior without
+source changes or recompilation.
+
+The options for dynamic configuration include: command line parameters,
+environment variables, and configuration files.
+
+==== Command line parameters
+Applications that accept a command line passed to their `main()` function can
+use this to tailor how they use ODP. This may involve self-imposed limits
+driven by the application or these can specify arguments that are to be
+passed to ODP initialization via the `odp_init_global()` API. The syntax of
+that API is:
+[source,c]
+-----
+int odp_init_global(odp_instance_t *instance,
+		    const odp_init_t *params,
+		    const odp_platform_init_t *platform_params);
+-----
+and the `odp_init_t` struct is used to pass platform-independent parameters
+that control ODP behavior while the `odp_platform_init_t` struct is used to
+pass platform-specific parameters. The `odp-linux` reference platform does
+not make use of these platform-specific parameters, however the `odp-dpdk`
+reference implementation uses these to allow applications to pass DPDK
+initialization parameters to it via these params.
+
+ODP itself uses the `odp_init_t` parameters to allow applications to specify
+override logging and abort functions. These routines are called to perform
+these functions on behalf of the ODP implementation, thus better allowing
+ODP to interoperate with application-defined logging and error handling
+facilities.
+
+==== Environment variables
+Linux environment variables set via the shell provide a convenient means of
+passing dynamic configuration values. Each ODP implementation defines which
+environment variables it looks for and how they are used. For example, the
+`odp-dpdk` implementation uses the variable `ODP_PLATFORM_PARAMS` as an
+alternate means of passing DPDK initialization parameters.
+
+Another important environment variable that ODP uses is `ODP_CONFIG_FILE`
+that is used to specify the file path of a _configuration override file_, as
+described in the next section.
+
+==== Configuration files
+The https://hyperrealm.github.io/libconfig/[libconfig] library provides a
+standard set of APIs and tools for parsing configuration files. ODP uses this
+to provide a range of dynamic configuration options that users may
+wish to specify.
+
+ODP uses a _base configuration file_ that contains system-wide defaults, and
+is located in the `config/odp-linux-generic.conf` file within the ODP
+distribution. This specifies a range of overridable configuration options that
+control things like shared memory usage, queue and scheduler limits and tuning
+parameters, timer processing options, as well as I/O parameters for various
+pktio classes.
+
+While users of ODP may modify this base file before building it, users can
+also supply an override configuration file that sets specific values of
+interest while leaving other parameters set to their defaults as defined by
+the base configuration file. As noted earlier, the `ODP_CONFIG_FILE`
+environment variable is used to point to the override file to be used.
+
+=== Summary
+There is a place for both static and dynamic configuration in any ODP
+implementation. This section described some of the most common and
+discussed how the ODP-supplied reference implementations make use of them.
+Other ODP implementations are free to copy and/or build on these, or use
+whatever other mechanisms are native to the platforms supported by those ODP
+implementations.
+
 include::../glossary.adoc[]