Skip to content

Commit

Permalink
UW-513 Add environment-variable support for template render (#425)
Browse files Browse the repository at this point in the history
  • Loading branch information
@maddenp-noaa
maddenp-noaa committed Mar 4, 2024
1 parent ce389ed commit 4039dcd
Show file tree
Hide file tree
Showing 7 changed files with 262 additions and 113 deletions.
85 changes: 48 additions & 37 deletions docs/sections/user_guide/cli/tools/mode_template.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ The ``uw`` mode for handling :jinja2:`Jinja2 templates<templates>`.
$ uw template render --help
usage: uw template render [-h] [--input-file PATH] [--output-file PATH] [--values-file PATH]
[--values-format {ini,nml,sh,yaml}] [--values-needed] [--partial]
[--dry-run] [--debug] [--quiet] [--verbose]
[--values-format {ini,nml,sh,yaml}] [--env] [--values-needed]
[--partial] [--dry-run] [--quiet] [--verbose]
[KEY=VALUE ...]
Render a template
Expand All @@ -47,14 +47,14 @@ The ``uw`` mode for handling :jinja2:`Jinja2 templates<templates>`.
Path to file providing override or interpolation values
--values-format {ini,nml,sh,yaml}
Values format
--env
Use environment variables
--values-needed
Print report of values needed to render template
--partial
Permit partial template rendering
--dry-run
Only log info, making no changes
--debug
Print all log messages, plus any unhandled exception's stack trace (implies --verbose)
--quiet, -q
Print no logging messages
--verbose, -v
Expand Down Expand Up @@ -84,8 +84,8 @@ and a YAML file called ``values.yaml`` with the following contents:
$ uw template render --input-file template --values-needed
[2023-12-18T19:16:08] INFO Value(s) needed to render this template are:
[2023-12-18T19:16:08] INFO greeting
[2023-12-18T19:16:08] INFO recipient
[2023-12-18T19:16:08] INFO greeting
[2023-12-18T19:16:08] INFO recipient
* To render the template to ``stdout``:

Expand Down Expand Up @@ -136,35 +136,6 @@ and a YAML file called ``values.yaml`` with the following contents:
$ uw template render --input-file template --values-file values.txt --values-format yaml
Hello, World!
* It is an error to render a template without providing all needed values. For example, with ``recipient: World`` removed from ``values.yaml``:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml
[2023-12-18T19:30:05] ERROR Required value(s) not provided:
[2023-12-18T19:30:05] ERROR recipient
But the ``--partial`` switch may be used to render as much as possible while passing expressions containing missing values through unchanged:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml --partial
Hello, {{ recipient }}!
Values may also be supplemented by ``key=value`` command-line arguments. For example:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml recipient=Reader
Hello, Reader!
Such ``key=value`` arguments may also be used to *override* file-based values:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml recipient=Reader greeting="Good day"
Good day, Reader!
* To request verbose log output:

.. code-block:: text
Expand Down Expand Up @@ -215,6 +186,48 @@ and a YAML file called ``values.yaml`` with the following contents:
[2023-12-18T23:27:04] DEBUG ---------------------------------------------------------------------
[2023-12-18T23:27:04] DEBUG Read initial values from values.yaml
**NB**: The following examples are based on a ``values.yaml`` file with ``recipient: World`` removed.

* It is an error to render a template without providing all needed values.

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml
[2024-03-02T16:42:48] ERROR Required value(s) not provided:
[2024-03-02T16:42:48] ERROR recipient
[2024-03-02T16:42:48] ERROR Template could not be rendered.
But the ``--partial`` switch may be used to render as much as possible while passing expressions containing missing values through unchanged:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml --partial
Hello, {{ recipient }}!
Values may also be supplemented by ``key=value`` command-line arguments:

.. code-block:: text
$ uw template render --input-file template --values-file values.yaml recipient=Reader
Hello, Reader!
The optional ``-env`` switch allows environment variables to be used to supply values:

.. code-block:: text
$ export recipient=You
$ uw template render --input-file template --values-file values.yaml --env
Hello, You!
Values from ``key=value`` arguments override values from file, and environment variables override both:

.. code-block:: text
$ recipient=Sunshine uw template render --input-file template --values-file values.yaml recipient=Reader greeting="Good day" --env
Good day, Sunshine!
Note that ``recipient=Sunshine`` is shell syntax for exporting environment variable ``recipient`` only for the duration of the command that follows. It should not be confused with the two ``key=value`` pairs later on the command line, which are arguments to ``uw``.

* Non-YAML-formatted files may also be used as value sources. For example, ``template``

.. code-block:: jinja
Expand All @@ -237,8 +250,6 @@ and a YAML file called ``values.yaml`` with the following contents:
$ uw template render --input-file template --values-file values.nml
Hello, World!
Note that ``ini`` and ``nml`` configs are, by definition, depth-2 configs, while ``sh`` configs are depth-1, and ``yaml`` configs have arbitrary depth.

.. _cli_template_translate_examples:

``translate``
Expand Down
12 changes: 8 additions & 4 deletions src/uwtools/api/template.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,12 @@


def render(
values: Union[dict, Path],
values_src: Optional[Union[dict, Path]] = None,
values_format: Optional[str] = None,
input_file: Optional[Path] = None,
output_file: Optional[Path] = None,
overrides: Optional[Dict[str, str]] = None,
env: bool = False,
values_needed: bool = False,
partial: bool = False,
dry_run: bool = False,
Expand All @@ -29,25 +30,27 @@ def render(
primary values source. If no input file is specified, ``stdin`` is read. If no output file is
specified, ``stdout`` is written to.
:param values: Source of values to render the template
:param values_src: Source of values to render the template
:param values_format: Format of values when sourced from file
:param input_file: Path to read raw Jinja2 template from (``None`` or unspecified => read
``stdin``)
:param output_file: Path to write rendered Jinja2 template to (``None`` or unspecified => write
to ``stdout``)
:param overrides: Supplemental override values
:param env: Supplement values with environment variables?
:param values_needed: Just report variables needed to render the template?
:param partial: Permit unrendered Jinja2 variables/expressions in output?
:param dry_run: Run in dry-run mode?
:return: The rendered template string
:raises: UWTemplateRenderError if template could not be rendered
"""
result = _render(
values=values,
values_src=values_src,
values_format=values_format,
input_file=input_file,
output_file=output_file,
overrides=overrides,
env=env,
values_needed=values_needed,
partial=partial,
dry_run=dry_run,
Expand All @@ -58,10 +61,11 @@ def render(


def render_to_str( # pylint: disable=unused-argument
values: Union[dict, Path],
values_src: Optional[Union[dict, Path]] = None,
values_format: Optional[str] = None,
input_file: Optional[Path] = None,
overrides: Optional[Dict[str, str]] = None,
env: bool = False,
values_needed: bool = False,
partial: bool = False,
dry_run: bool = False,
Expand Down
59 changes: 39 additions & 20 deletions src/uwtools/cli.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from dataclasses import dataclass
from functools import partial
from pathlib import Path
from typing import Any, Callable, Dict, List, Tuple
from typing import Any, Callable, Dict, List, NoReturn, Tuple

import uwtools.api.config
import uwtools.api.fv3
Expand All @@ -20,7 +20,7 @@
import uwtools.api.template
import uwtools.config.jinja2
import uwtools.rocoto
from uwtools.exceptions import UWConfigRealizeError, UWTemplateRenderError
from uwtools.exceptions import UWConfigRealizeError, UWError, UWTemplateRenderError
from uwtools.logging import log, setup_logging
from uwtools.utils.file import FORMAT, get_file_format

Expand All @@ -42,20 +42,27 @@ def main() -> None:
# defined checks for the appropriate [sub]mode. Reconfigure logging after quiet/verbose choices
# are known, then dispatch to the [sub]mode handler.

setup_logging(quiet=True)
args, checks = _parse_args(sys.argv[1:])
for check in checks[args[STR.mode]][args[STR.action]]:
check(args)
setup_logging(quiet=args[STR.quiet], verbose=args[STR.verbose])
log.debug("Command: %s %s", Path(sys.argv[0]).name, " ".join(sys.argv[1:]))
modes = {
STR.config: _dispatch_config,
STR.fv3: _dispatch_fv3,
STR.rocoto: _dispatch_rocoto,
STR.sfcclimogen: _dispatch_sfc_climo_gen,
STR.template: _dispatch_template,
}
sys.exit(0 if modes[args[STR.mode]](args) else 1)
try:
setup_logging(quiet=True)
args, checks = _parse_args(sys.argv[1:])
for check in checks[args[STR.mode]][args[STR.action]]:
check(args)
setup_logging(quiet=args[STR.quiet], verbose=args[STR.verbose])
except UWError as e:
_abort(str(e))
try:
log.debug("Command: %s %s", Path(sys.argv[0]).name, " ".join(sys.argv[1:]))
modes = {
STR.config: _dispatch_config,
STR.fv3: _dispatch_fv3,
STR.rocoto: _dispatch_rocoto,
STR.sfcclimogen: _dispatch_sfc_climo_gen,
STR.template: _dispatch_template,
}
sys.exit(0 if modes[args[STR.mode]](args) else 1)
except UWError as e:
log.error(str(e))
sys.exit(1)


# Mode config
Expand Down Expand Up @@ -434,6 +441,7 @@ def _add_subparser_template_render(subparsers: Subparsers) -> ActionChecks:
_add_arg_output_file(optional)
_add_arg_values_file(optional)
_add_arg_values_format(optional, choices=FORMATS)
_add_arg_env(optional)
_add_arg_values_needed(optional)
_add_arg_partial(optional)
_add_arg_dry_run(optional)
Expand Down Expand Up @@ -464,18 +472,20 @@ def _dispatch_template_render(args: Args) -> bool:
"""
try:
uwtools.api.template.render(
values=args[STR.valsfile],
values_src=args[STR.valsfile],
values_format=args[STR.valsfmt],
input_file=args[STR.infile],
output_file=args[STR.outfile],
overrides=_dict_from_key_eq_val_strings(args[STR.keyvalpairs]),
env=args[STR.env],
values_needed=args[STR.valsneeded],
partial=args[STR.partial],
dry_run=args[STR.dryrun],
)
except UWTemplateRenderError:
msg = "Template could not be rendered. Try with %s for details." % _switch(STR.valsneeded)
log.error(msg)
if args[STR.valsneeded]:
return True
log.error("Template could not be rendered.")
return False
return True

Expand Down Expand Up @@ -534,6 +544,14 @@ def _add_arg_dry_run(group: Group) -> None:
)


def _add_arg_env(group: Group) -> None:
group.add_argument(
_switch(STR.env),
action="store_true",
help="Use environment variables",
)


def _add_arg_file_format(
group: Group, switch: str, helpmsg: str, choices: List[str], required: bool = False
) -> None:
Expand Down Expand Up @@ -704,7 +722,7 @@ def _add_arg_verbose(group: Group) -> None:
# Support


def _abort(msg: str) -> None:
def _abort(msg: str) -> NoReturn:
"""
Exit with an informative message and error status.
Expand Down Expand Up @@ -854,6 +872,7 @@ class STR:
config: str = "config"
cycle: str = "cycle"
dryrun: str = "dry_run"
env: str = "env"
file1fmt: str = "file_1_format"
file1path: str = "file_1_path"
file2fmt: str = "file_2_format"
Expand Down
Loading

0 comments on commit 4039dcd

Please sign in to comment.