Switch CLI args/named-args for attr-{get,set}

- Switch 'stanza' and 'attribute' as required named arguments instead of
  positional arguments.  This allows the interface between set/get subcommands
  to more naturally align and it should make scripts more readable.
- Add some additional changelog and cheatsheet content.
- Add some API usage examples showing some alternate easy conf update
  processes.
- Cleanup get/set code a bit (remove unnecessary import, fix some comments, and
  so on)

docs/source/changelog.rst

@@ -32,12 +32,12 @@ Ksconf 0.12
 Ksconf v0.12.1 (DRAFT)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-*  Introducing `ksconf_cmd_attr-get` and `ksconf_cmd_attr-set` - the newest and simplest ksconf commands ever!
+*  Introducing :ref:`ksconf_cmd_attr-get` and :ref:`ksconf_cmd_attr-set` - the newest and simplest ksconf commands ever!
    Use this to quickly grab and/or update a specific stanzas, attribute combination from a conf file.
 *  Add new attribute-level matching logic to ``ksconf filter``.
    Use ``--attr-matches`` and/or ``--attr-not-matches`` to match specific attribute and value combinations for stanza matching.
    This can be used to find props with a specific ``KV_MODE``, find saved search containing a specific search command, or list indexes not using ``volume:`` designation.
-   See the `ksconf_cmd_filter` docs for example usage.
+   See the :ref:`ksconf_cmd_filter` docs for example usage.
 *  Add ``--in-place`` processing behavior for :ref:`ksconf_cmd_merge` to simplify the process of merging new content into an existing conf file.
 *  Fixed documentation generation bug that prevented command line options from showing up in the docs.
 *  Fixed some CLI file handling bug that resulted in broken use of ``-`` (stdin) and/or fancy shell commands involving ``<(some command)`` syntax, which can be a helpful trick to reduce the number of temporary files.

docs/source/cheatsheet.rst

@@ -27,6 +27,31 @@ General purpose
 ---------------
 
 
+Extracting a single value
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Grabbing the definition of a single macro using :ref:`ksconf_cmd_attr-get`.
+Note in the case of a complex or multi-line expression, any line continuation characters will be removed.
+
+    .. code-block:: sh
+
+        ksconf attr-get macros.conf --stanza 'unroll_json_array(6)' --attribute definition
+
+
+Updating a single value
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Suppose you have a macro called ``mydata_index`` that defines the source indexes for your dashboards.
+The following command uses :ref:`ksconf_cmd_attr-set` to update that macro directly from the CLI without opening an editor.
+
+    .. code-block:: sh
+
+        ksconf attr-set macros.conf --stanza mydata_index --attribute definition --value 'index=mydata1 OR index=otheridx'
+
+In this case the definition is a single line, but multi-line input is handled automatically.
+It's also possible to pull a vale from an existing file or from an environment variable, should that be useful.
+
+
 Comparing files
 ~~~~~~~~~~~~~~~~
 
@@ -78,7 +103,7 @@ Say you want to remove ``vsid`` from a legacy savedsearches file:
         ksconf filter search/default/savedsearches.conf --reject-attrs "vsid"
 
 
-To see just to the schedule and scheduler status of scheduled searches, run:
+To see just to the scheduled time and enablement status of scheduled searches, run:
 
     .. code-block:: sh
 
@@ -97,6 +122,16 @@ List apps configured in the deployment server
             cut -d: -f4 | sort | uniq
 
 
+Find saved searches with earliest=-1d@d
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    .. code-block:: sh
+
+        ksconf filter apps/*/default/savedsearches.conf \
+            --attr-eq dispatch.earliest_time "-1d@d"
+
+
+
 Cleaning up
 -----------
 

docs/source/cmd_attr-get.rst

@@ -16,18 +16,19 @@ ksconf attr-get
 Example
 ^^^^^^^
 
-
 Show the version of the Splunk AWS technology addon:
 
 ..  code-block:: sh
 
-    ksconf attr-get launcher version etc/apps/Splunk_TA_AWS/default/app.conf
+    ksconf attr-get etc/apps/Splunk_TA_AWS/default/app.conf --stanza launcher --attribute version
+
 
 
-Fetch the "live" (prefer local over default) search string called "Internal Server Errors" from my_app.
-The search string will be saved to your text file without any additional metadata or continuation markers.
+Fetch the search string for the "Internal Server Errors" search in the from my_app.
+The search is saved to a text file without any metadata or line continuation markers (trailing ```\\`` characters.)
+Note that ``kconf merge`` is used here to ensure that the "live" version of the search is shown, so ``local`` will be used if present, otherwise ``default`` will be shown.
 
 ..  code-block:: sh
 
     ksconf merge $SPLUNK_HOME/etc/apps/my_app/{default,local}/savedsearches.conf \
-    | ksconf attr-get "Internal System Errors" search - -o errors_search.txt
+    | ksconf attr-get - -s "Internal System Errors" -a search -o errors_search.txt

docs/source/cmd_attr-set.rst

@@ -21,14 +21,55 @@ Update build during CI/CD
 
 ..  code-block:: sh
 
-    ksconf attr-set build/default.app launcher version 1.1.2
-    ksconf attr-set build/default.app launcher build --value-type env GITHUB_RUN_NUMBER
-
+    ksconf attr-set build/default.app -s launcher -a version 1.1.2
+    ksconf attr-set build/default.app -s launcher -a build --value-type env GITHUB_RUN_NUMBER
 
 Rewrite a saved search to match the new cooperate initiative to relabel all "CRITICAL" messages as "WHOOPSIES".
 
 ..  code-block:: sh
 
-    ksconf attr-get "Internal System Errors" search savedsearches.conf \
+    ksconf attr-get savedsearches.conf -s "Internal System Errors" -a search \
         | sed -re 's/CRITICAL/WHOOPSIES/g' \
-        | ksconf attr-set savedsearches.conf "Internal System Errors" search --value-type file -
+        | ksconf attr-set savedsearches.conf -s "Internal System Errors" -a search --value-type file -
+
+
+
+..  note::  What if you want to write multiple stanza/attributes at once?
+
+    Of course it's possible to call ``ksconf attr-set`` multiple times, but that may be awkward or inefficient if many updates are needed.
+    In the realm of shell scripting, another option is to use :ref:`ksconf_cmd_merge` which is designed to merge multiple stanzas, or even multiple files, at once.
+    With a little bit of creatively, it's possible to add (or update) and entire new stanza in-line using a single command like so:
+
+    ..  code-block:: sh
+
+        printf '[drop_field(1)]\ndefinition=| fields - $field$\nargs=field\niseval=0\n' | ksconf merge --in-place --target macros.conf -
+
+        # which is identical to:
+        ksconf merge --in-place --target macros.conf <(printf '[drop_field(1)]\ndefinition=| fields - $field$\nargs=field\niseval=0\n')
+
+    Of course, neither of these are super easy to read.  If your content is static, then an easy answer it to use a static conf file.
+    However, at some point it may be easier to just edit these using Python where any arbitrary level of complexity is possible.
+
+    Ksconf has some built in utility functions to make this kind of simple update-in-place workflow super simple.
+    For example, the :py:class:`~ksconf.conf.parser.update_conf` context manager allows access to existing conf values and quick modification.
+    If no modification is necessary, then the file is left untouched.
+
+    ..  code-block:: py
+
+            from ksconf.conf.parser import update_conf, conf_attr_boolean
+
+            # Update app.conf for a build release
+            with update_conf("app.conf") as conf:
+                conf["launcher"]["version"] = "1.0.2"
+                conf["install"]["build"] = "33"
+
+            # Update sourcetype references in all saved searches; place marker in description
+            with update_conf("savedsearches.conf") as conf:
+                for report in conf:
+                    if not conf_attr_boolean(conf[report].get("disabled", "0")):
+                        # Update enabled search
+                        search = conf[report].get("search", "")
+                        conf[report]["search"] = search.replace("cisco:old-understood-tech", "cisco:new-fangled-tech")
+                        conf[report]["description"] = f"We did an update.\n Old description: {conf[report].get('description', '')}"
+
+    ..  Yes, we need an API intro for simple use cases like this.  For now, I guess this is it!?!

docs/source/cmd_merge.rst

@@ -9,6 +9,12 @@ ksconf merge
     :path: merge
     :nodefault:
 
+    --in-place: @after
+
+    The ``--in-place`` option was added in v0.12.1.
+    In earlier version of ksconf, and move forward, this same behavior can be accomplished by simply listing the target twice.
+    Once as in the ``--target`` option, and then a second time as the first CONF file.
+
 
 Examples
 ---------

ksconf/commands/attr.py

@@ -1,19 +1,19 @@
-""" SUBCOMMAND:  ``ksconf attr-get <CONF>``
+""" SUBCOMMAND:  ``ksconf attr-get <CONF> --stanza STANZA --attribute ATTR``
 
 .. code-block:: sh
 
-    ksconf attr-get launcher version $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/app.conf
+    ksconf attr-get $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/app.conf --stanza launcher --attribute version
 
 
-SUBCOMMAND:  ``ksconf attr-set <CONF>``
+SUBCOMMAND:  ``ksconf attr-set <CONF> --stanza STANZA --attribute ATTR --value VALUE``
 
-    ksconf attr-set launcher version $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf --value 9.9.9
+    ksconf attr-set $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf --stanza launcher --attribute version --value 9.9.9
 
     echo "9.9.9" > /tmp/new_version
-    ksconf attr-set launcher version $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf -t file /tmp/new_version
+    ksconf attr-set $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf --stanza launcher --attribute version  -t file /tmp/new_version
 
     export NEW_VERSION=1.2.3
-    ksconf attr-set launcher version $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf -t env NEW_VERSION
+    ksconf attr-set $SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/app.conf --stanza launcher --attribute version -t env NEW_VERSION
 
 
 """
@@ -25,12 +25,9 @@
 from pathlib import Path
 
 from ksconf.commands import KsconfCmd, dedent
-from ksconf.conf.parser import (PARSECONF_STRICT, ConfParserException,
-                                parse_conf, update_conf, write_conf)
-from ksconf.consts import (EXIT_CODE_BAD_CONF_FILE,
-                           EXIT_CODE_CONF_NO_DATA_MATCH, EXIT_CODE_NO_SUCH_FILE,
-                           EXIT_CODE_NOTHING_TO_DO, EXIT_CODE_SORT_APPLIED,
-                           EXIT_CODE_SUCCESS, SMART_NOCHANGE)
+from ksconf.conf.parser import update_conf
+from ksconf.consts import (EXIT_CODE_CONF_NO_DATA_MATCH, EXIT_CODE_NO_SUCH_FILE,
+                           EXIT_CODE_NOTHING_TO_DO, EXIT_CODE_SUCCESS)
 from ksconf.util.completers import conf_files_completer
 from ksconf.util.file import expand_glob_list
 
@@ -44,16 +41,17 @@ class AttrGetCmd(KsconfCmd):
     maturity = "beta"
 
     def register_args(self, parser):
-        parser.add_argument("stanza", metavar="STANZA",
-                            help="Name of the conf file stanza to retrieve.")
-        parser.add_argument("attribute", metavar="ATTR",
-                            help="Name of the conf file attribute to retrieve.")
-
         parser.add_argument("conf", metavar="FILE", nargs="+",
                             default=["-"],
                             help="Input file to sort, or standard input."
                             ).completer = conf_files_completer
 
+        parser.add_argument("--stanza", "-s", metavar="STANZA", required=True,
+                            help="Name of the conf file stanza to retrieve.")
+        parser.add_argument("--attribute", "--attr", "-a",
+                            metavar="ATTR", required=True,
+                            help="Name of the conf file attribute to retrieve.")
+
         parser.add_argument("--missing-okay", action="store_true", default=False,
                             help="Ignore missing stanzas and attributes.  ")
         '''
@@ -71,7 +69,7 @@ def pre_run(self, args):
         args.conf = list(expand_glob_list(args.conf))
 
     def run(self, args):
-        ''' Sort one or more configuration file. '''
+        ''' For a given conf file, get the 'value' from [stanza] attribute = value '''
         for conf in args.conf:
             if len(args.conf) > 1:
                 args.output.write(f"---------------- [ {conf} ] ----------------\n\n")
@@ -115,10 +113,11 @@ def register_args(self, parser):
         parser.add_argument("conf", metavar="FILE",
                             help="Configuration file to update."
                             ).completer = conf_files_completer
-        parser.add_argument("stanza", metavar="STANZA",
-                            help="Name of the conf file stanza to retrieve.")
-        parser.add_argument("attribute", metavar="ATTR",
-                            help="Name of the conf file attribute to retrieve.")
+        parser.add_argument("--stanza", "-s", metavar="STANZA", required=True,
+                            help="Name of the conf file stanza to set.")
+        parser.add_argument("--attribute", "--attr", "-a",
+                            metavar="ATTR", required=True,
+                            help="Name of the conf file attribute to set.")
 
         parser.add_argument("value", metavar="VALUE",
                             help="Value to apply to the conf file.  Note that this can be a raw "
@@ -166,13 +165,13 @@ def set_conf_value(self, conf_file: Path, stanza: str, attribute: str, value: st
                 if existing_value == value:
                     self.stderr.write(f"No change necessary.  {conf_file} "
                                       f"[{stanza}] {attribute} already has desired value.\n")
-                    conf.abort_update()
+                    conf.cancel()
                     return EXIT_CODE_SUCCESS
 
                 if no_overwrite:
                     self.stderr.write(f"Skipping updating {conf_file} due to --no-overwrite.  "
                                       f"[{stanza}] {attribute} already set.\n")
-                    conf.abort_update()
+                    conf.cancel()
                     return EXIT_CODE_NOTHING_TO_DO
 
             if stanza not in conf:
@@ -183,10 +182,8 @@ def set_conf_value(self, conf_file: Path, stanza: str, attribute: str, value: st
         return EXIT_CODE_SUCCESS
 
     def run(self, args):
-        ''' Sort one or more configuration file. '''
+        ''' For a given conf file, set [stanza] attribute = value '''
         value = self.get_value(args.value, args.value_type)
-
-        print(f"args.conf:  {args.conf}")
         conf_path = Path(args.conf)
         return self.set_conf_value(conf_path, args.stanza, args.attribute, value,
                                    args.create_missing, args.no_overwrite)

ksconf/conf/parser.py

@@ -172,6 +172,10 @@ def _detect_lite(byte_str: bytes) -> Dict[str, str]:
 
 
 def detect_by_bom(path: PathType) -> str:
+    # Refuse to consume a possibly read-once stream.  Could be a shell:  <(cmd)
+    # Using an assert because this really *should* be handled by the caller, but just in case...
+    assert not os.fspath(path).startswith("/dev/fd/"), "Don't bom my pipe!"
+
     with open(path, 'rb') as f:
         raw = f.read(4)    # will read less if the file is smaller
     encoding = _detect_lite(raw)
@@ -576,6 +580,7 @@ def keys(self) -> List[str]:
     def update(self, *args, **kwargs):
         self._data.update(*args, **kwargs)
 
-    def abort_update(self):
-        """ Indicate that no updates were made. """
+    def cancel(self):
+        """ Indicate that no updates were made and all processing is complete.
+        An error will occur if additional read/writes are attempted. """
         self._data = None

ksconf/consts.py

@@ -45,7 +45,7 @@ class _UNSET:
 EXIT_CODE_NOTHING_TO_DO = 1
 EXIT_CODE_USER_QUIT = 2
 EXIT_CODE_NO_SUCH_FILE = 5
-EXIT_CODE_MISSING_ARG = 6
+EXIT_CODE_MISSING_ARG = 6  # Note that argparse returns 2 on it's own for missing arguments
 EXIT_CODE_BAD_ARGS = 7
 
 EXIT_CODE_DIFF_EQUAL = 0