Add 'rest-publish' command

Supports: - Pushing multiple stanzas to a specific config endpoint at once, in a fairly efficient process. - Changes made on the server are show in the form of a diff. (Eventually, this should be configurable, and a dry-run mode should be added.) - Supports pushing to a live Splunk instance (via Splunkd REST API). This should replace the use of 'rest-export' for MOST use cases. - Supports reading and applying metadata (ACLs, owner, shared, app) to imported objects. - Any type of conf entry should be supported. Limitations: - **NO UNIT TESTING!** (This will require some fancy mock work.) - Doesn't support attribute-level metadata. - No SSL validation setup yet. This is primarily because of how Splunk's Python SDK works. For other Python HTTP libraries this is enabled by default. Squashed commit of the following: commit 80855ed Author: Lowell Alleman <lowell@kintyre.co> Date: Tue Feb 26 22:00:00 2019 -0500 Add changelog for rest-publish commit 8c8c60a Author: Lowell Alleman <lowell@kintyre.co> Date: Tue Feb 26 21:14:26 2019 -0500 Fixed small bug in rest-publish Resolved issue where 'diff' output would crash if the Splunkd version of the stanza contained a blank attribute (which was being returned as a None) which was screwing up the show_diff() function. - Replace values of None with '' for attributes returned from Splunkd (REST). This fixes the core issue in rest-publish. - Placed protective code in the show_diff() to be slightly better at handling unexpected None values. This is a secondary protection method that should at least NOT CRASH if somehow a None leaks in from somewhere else. commit 5ef130c Author: Lowell Alleman <lowell@kintyre.co> Date: Mon Feb 25 18:39:54 2019 -0500 Fixed bugs in rest-publish - Fixed issue where rest-publish would fail if '--conf' wasn't explicitly set. Has something to do with encoding. Splunk SDK is looking for a non-unicode value for conf-type. Whatever. - Fixed corner-case thing where ACL has 'read' or 'write' is an empty list. commit d0a3df2 Author: Lowell Alleman <lowell@kintyre.co> Date: Mon Feb 25 18:08:56 2019 -0500 Reduce server round-trips for rest-publish - Fix ACL diff output (requires some manipulation of the 'access' data to some wonky hybrid of (1) what the conf file looks like, (2) what the rest endpoint returns, and (3) what the ACL end-point expect. (Because you know, all 3 have to be different.) - Remove unnecessary ACL endpoint call (due to typo). - Remove the 'exists' check per stanza overhead. - Fixed an always-update bug that caused an extra round-trip. - Improved some output for some error conditions (lots of room for improvement here). Sill no unit tests, and no dry-run mode, ... In testing with a simple 4 stanzas file. I was able to get the best case run (where no changes are needed) down from from 17 REST calls down to 5. Which the theoretical best case possible (1 per stanzas plus 1 for login). First pushes will still be slower, at about 13 calls, but this should still feel much faster then before. (Running a few hundred felt quite painful.) commit c4e7dba Author: Lowell Alleman <lowell@kintyre.co> Date: Mon Feb 25 13:31:54 2019 -0500 Various fixes to rest-publish - Explicitly sort stanzas for additional predicable behavior. - Fixed missing/limited 'path' when publishing stanzas (diff header). - Expand/fine rest-publish CLI docs. - Noted some additional issues in comments. commit e7b4b34 Author: Lowell Alleman <lowell@kintyre.co> Date: Wed Feb 20 22:52:36 2019 -0500 Metadata support for rest-publish - Adds initial support for per-object metadata support for the rest-publish command. - Wonky workaround required for splunksdk limitation where posting to the '/acl' endpoint doesn't work due to kwargs handling. See upstream issue here: splunk/splunk-sdk-python#207 Still no unit-tests for rest-publish, and I'm not sure if mock is enough to handle this complex mess! (I'm really not excited about launching a Splunk instance from within Travis CI, either.) commit 747701e Author: Lowell Alleman <lowell@kintyre.co> Date: Tue Feb 19 15:41:40 2019 -0500 Add '--delete' support to rest-publish - Complete delete support. (Refactoring may still make sense). - Improved 'diff' output by showing REST endpoint on the one side, along with the modification time (when known). commit 0403008 Author: Lowell Alleman <lowell@kintyre.co> Date: Thu Feb 14 22:40:31 2019 -0500 Working rest-publish prototype development snapshot - Completed first pass of 'rest-publish'. Needs work, but functional. commit b745e14 Author: Lowell Alleman <lowell@kintyre.co> Date: Thu Feb 14 14:37:22 2019 -0500 First draft of 'rest-publish' command Development snapshot before direction change. Current implementation is based on 'requests' with custom calls to REST endpoints, but switching over to splunk-sdk. - Moved common splunkd access argparse arguments to shared function. - Split some common REST path handling functions in a shared ksconf.util.rest module. - Changed rest-export interface to use '--owner' (object owner) instead of using '--user' which is used for Splunkd auth. Currently the old argument is still supported with a deprecation warning.
Kintyre · Feb 27, 2019 · f22ac0a · f22ac0a
1 parent cc212ec
commit f22ac0a
Show file tree

Hide file tree

Showing 12 changed files with 526 additions and 8 deletions.
diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst
@@ -16,6 +16,7 @@ Release v0.6.3 (DRAFT)
 
 *General changes*
 
+-   Added new :ref:`ksconf_cmd_rest-publish` command that for nearly every use case supersedes the use of ``rest-export``.
 -   Added :doc:`cheatsheet` to the docs.
 -   Significant improvement to entrypoint handling and support for conditional inclusion of 3rd party libraries with sane behavior on import errors, and improved warnings.  This information is conveniently viewable to the user via ``ksconf --version``.
 -   Refactored internal diff logic and added additional safeties and unit tests.  This includes improvements to TTY colorization which should avoid previous color leaks scenarios that were likely if unhandled exceptions occur.
@@ -26,7 +27,8 @@ Release v0.6.3 (DRAFT)
 
 -   Modified installation of python package installation.  In previous releases, various ``.dist-info`` folders were created with version-specific names leading to a mismatch of package versions after upgrade.
 -   Changed Splunk app install script to ``install.py`` (it was ``bootstrap_bin.py``).  Hopefully this is more intuitive.
--   Add windows support in ``install.py``.
+-   Added windows support in ``install.py``.
+-   Now includes the Splunk Python SDK.  Currently used for ``rest-publish`` but will eventually be used for additional functionally unique to the Splunk app.
 
 
 Release v0.6.2 (2019-02-09)

diff --git a/docs/source/cmd_rest-export.rst b/docs/source/cmd_rest-export.rst
@@ -3,6 +3,16 @@
 ksconf rest-export
 ==================
 
+..  deprecated:: 0.7.0
+
+    You should consider using :ref:`ksconf_cmd_rest-publish` instead of this one.
+    The only remaining valid use case for ``rest-export`` (this command) is for disconnected scenarios.
+    In other words, if you need to push stanzas to a splunkd instance where you don't (an can't) install ``ksconf``,
+    then this command may still be useful to you.
+    In this case, ``ksconf rest-export`` can create a shell script that you can transfer to the correct network,
+    and then run the shell script.
+    But for **ALL** other use cases, the ``rest-publish`` command is superior.
+
 ..  argparse::
     :module: ksconf.__main__
     :func: build_cli_parser

diff --git a/docs/source/cmd_rest-publish.rst b/docs/source/cmd_rest-publish.rst
@@ -0,0 +1,42 @@
+..  _ksconf_cmd_rest-publish:
+
+ksconf rest-publish
+===================
+
+..  note::
+
+    This command effectively replace :ref:`ksconf_cmd_rest-export` for all nearly all use cases.
+    The only thing that ``rest-publish`` can't do that ``rest-export`` is handle a disconnected scenario.
+    But for **ALL** other use cases, the ``rest-publish`` (this command) command is far superior.
+
+..  note:: This commands requires the Splunk Python SDK, which is automatically bundled with the *Splunk app for KSCONF*.
+
+
+..  argparse::
+    :module: ksconf.__main__
+    :func: build_cli_parser
+    :path: rest-publish
+    :nodefault:
+
+--------
+
+
+
+Examples
+---------
+
+A simple example:
+
+.. code-block:: sh
+
+    ksconf rest-publish etc/app/Splunk_TA_aws/local/props.conf \
+        --user admin --password secret --app Splunk_TA_aws --owner nobody --sharing global
+
+
+This command also supports replaying metdata like ACLs:
+
+.. code-block:: sh
+
+    ksconf rest-publish etc/app/Splunk_TA_aws/local/props.conf \
+        --meta etc/app/Splunk_TA_aws/metdata/local.meta \
+        --user admin --password secret --app Splunk_TA_aws
diff --git a/docs/source/dyn/cli.rst b/docs/source/dyn/cli.rst
@@ -12,7 +12,7 @@ ksconf
  .. code-block:: none
 
     usage: ksconf [-h] [--version] [--force-color]
-                  {check,combine,diff,filter,promote,merge,minimize,snapshot,sort,rest-export,unarchive}
+                  {check,combine,diff,filter,promote,merge,minimize,snapshot,sort,rest-export,rest-publish,unarchive}
                   ...
     
     Ksconf: Kintyre Splunk CONFig tool
@@ -25,7 +25,7 @@ ksconf
     "default" (which splunk can't handle natively) are all supported tasks.
     
     positional arguments:
-      {check,combine,diff,filter,promote,merge,minimize,snapshot,sort,rest-export,unarchive}
+      {check,combine,diff,filter,promote,merge,minimize,snapshot,sort,rest-export,rest-publish,unarchive}
         check               Perform basic syntax and sanity checks on .conf files
         combine             Combine configuration files across multiple source
                             directories into a single destination directory. This
@@ -49,6 +49,8 @@ ksconf
                             appropriate for version control
         rest-export         Export .conf settings as a curl script to apply to a
                             Splunk instance later (via REST)
+        rest-publish        Publish .conf settings to a live Splunk instance via
+                            REST
         unarchive           Install or upgrade an existing app in a git-friendly
                             and safe way
     
@@ -501,6 +503,63 @@ ksconf rest-export
 
 
 
+.. _ksconf_cli_rest-publish:
+
+ksconf rest-publish
+*******************
+
+ .. code-block:: none
+
+    usage: ksconf rest-publish [-h] [--conf TYPE] [-m META] [--url URL]
+                               [--user USER] [--pass PASSWORD] [-k] [--app APP]
+                               [--owner OWNER] [--sharing {user,app,global}] [-D]
+                               CONF [CONF ...]
+    
+    Publish stanzas in a .conf file to a running Splunk instance via REST. This
+    requires access to the HTTPS endpoint of splunk. By default, ksconf will
+    handle both the creation of new stanzas and the update of exists stanzas. This
+    can be used to push full configuration stanzas where you only have REST access
+    and can't directly publish an app. Only attributes present in the conf file
+    are pushed. While this may seem obvious, this fact can have profound
+    implications in certain situations, like when using this command for
+    continuous updates. This means that it's possible for the source .conf to
+    ultimately differ from what ends up on the server's .conf file. One way to
+    avoid this is to explicitly remove object using '--delete' mode first, and
+    then insert a new copy of the object. Of course this means that the object
+    will be unavailable. The other impact is that diffs only compares and shows a
+    subset of attribute. Be aware that, for consistency, the configs/conf-TYPE
+    endpoint is used for this command. Therefore, a reload may be required for the
+    server to use the published config settings.
+    
+    positional arguments:
+      CONF                  Configuration file(s) to export settings from.
+    
+    optional arguments:
+      -h, --help            show this help message and exit
+      --conf TYPE           Explicitly set the configuration file type. By default
+                            this is derived from CONF, but sometime it's helpful
+                            set this explicitly. Can be any valid Splunk conf file
+                            type, example include 'app', 'props', 'tags',
+                            'savedsearches', and so on.
+      -m META, --meta META  Specify one or more '.meta' files to determine the
+                            desired read & write ACLs, owner, and sharing for
+                            objects in the CONF file.
+      --url URL             URL of Splunkd. Default: https://localhost:8089
+      --user USER           Login username Splunkd. Default: admin
+      --pass PASSWORD       Login password Splunkd. Default: changeme
+      -k, --insecure        Disable SSL cert validation.
+      --app APP             Set the namespace (app name) for the endpoint
+      --owner OWNER         Set the user who owns the content. The default of
+                            'nobody' works well for app-level sharing.
+      --sharing {user,app,global}
+                            Set the sharing mode.
+      -D, --delete          Remove existing REST entities. This is a destructive
+                            operation. In this mode, stanzas attributes are
+                            unnecessary. NOTE: This works for 'local' entities
+                            only; the default folder cannot be updated.
+
+
+
 .. _ksconf_cli_unarchive:
 
 ksconf unarchive

diff --git a/docs/source/ksconf.util.rst b/docs/source/ksconf.util.rst
@@ -28,6 +28,14 @@ ksconf.util.file module
     :undoc-members:
     :show-inheritance:
 
+ksconf.util.rest module
+-----------------------
+
+..  automodule:: ksconf.util.rest
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 ksconf.util.terminal module
 ---------------------------
 

diff --git a/ksconf/commands/__init__.py b/ksconf/commands/__init__.py
@@ -25,6 +25,8 @@
     "dedent",
     "get_all_ksconf_cmds",
     "get_entrypoints",
+    "add_splunkd_access_args",
+    "add_splunkd_namespace",
 ]
 
 
@@ -379,6 +381,31 @@ def post_run(self, args, exec_info=None):
         pass
 
 
+def add_splunkd_access_args(parser):
+    # type: (argparse.ArgumentParser) -> argparse.ArgumentParser
+    parser.add_argument("--url", default="https://localhost:8089",
+                        help="URL of Splunkd.  Default:  %(default)s")
+    parser.add_argument("--user", default="admin",
+                        help="Login username Splunkd.  Default:  %(default)s")
+    parser.add_argument("--pass", dest="password", default="changeme",
+                        help="Login password Splunkd.  Default:  %(default)s")
+    parser.add_argument("-k", "--insecure", action="store_true", default=False,
+                        help="Disable SSL cert validation.")
+    return parser
+
+
+def add_splunkd_namespace(parser):
+    # type: (argparse.ArgumentParser) -> argparse.ArgumentParser
+    parser.add_argument("--app", default="$SPLUNK_APP",
+                        help="Set the namespace (app name) for the endpoint")
+    parser.add_argument("--owner", default="nobody",
+                        help="Set the user who owns the content.  "
+                             "The default of 'nobody' works well for app-level sharing.")
+    parser.add_argument("--sharing", default="global", choices=["user", "app", "global"],
+                        help="Set the sharing mode.")
+    return parser
+
+
 def _get_entrypoints_lib(group, name=None):
     import entrypoints
 

diff --git a/ksconf/commands/restexport.py b/ksconf/commands/restexport.py
@@ -23,6 +23,7 @@
 
 from six.moves.urllib.parse import quote
 
+from ksconf.util.rest import build_rest_url
 from ksconf.commands import KsconfCmd, dedent, ConfFileType
 from ksconf.conf.parser import PARSECONF_LOOSE, GLOBAL_STANZA
 from ksconf.consts import EXIT_CODE_SUCCESS
@@ -173,11 +174,10 @@ def register_args(self, parser):
 
     @staticmethod
     def build_rest_url(base, owner, app, conf):
-        # XXX: Quote user & app; however for now we're still allowing the user to pass though an
+        # XXX: Quote owner & app; however for now we're still allowing the user to pass though an
         #  environmental variable as-is and quoting would break that.   Need to make a decision,
         # for now this is not likely to be a big issue given app and user name restrictions.
-        url = "{}/servicesNS/{}/{}/configs/conf-{}".format(base, owner, app, conf)
-        return url
+        return build_rest_url(base, "configs/conf-{}".format(conf), owner, app)
 
     def run(self, args):
         ''' Convert a conf file into a bunch of CURL commands'''