Diff support expansion and refactoring

Expanded existing diff support, and refactored the code to support additional
use cases and improve safety.   The entire commit history is provided, but in
summary:

- Split compare_cfgs() into smaller bits.  Now a diff on individual stanzas is
  supported using the compare_stanzas() function.
- Expand diff header support.  It's now possible to further manipulate headers,
  primarily this is used for setting the modification time for non-file objects.
- Add new TermColor helper to provide context manager support around ANSI color
  manipulation on the terminal.  This reduces the likelihood of bad TTY state.
- Added helper functions:  reduce_stanza() and is_equal()

Squashed commit of the following:

commit 37f15f20c5bafecac303c72b25d7615b755bbff3
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Mon Feb 25 19:45:11 2019 -0500

    Refactor compare_stanza() into private and public parts

    - Split compare_stanzas() into 2 parts.  1 Part is private and used by the
      public version and by compare_cfgs().  The public interface now returns a
      list, not an iterable.

commit 5a0ec24287a08ad5e2ce7ccfda828aad90815f80
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Fri Feb 22 18:44:03 2019 -0500

    Add new TermColor helper

    Created a new context-managing class as a replacement for the simple
    tty_color() function.  I ran into an issue in the field where and unhandled
    exception in the diff library (surely caused by something external) left the
    terminal in a hard-to-read color.   And that should just never happen.  This
    context manager should keep this from being an issue.

    Written at 2AM after a 6 hour client working session.  So, do more testing.

commit 51866ea3d9e36331cd3d67440586a379f5acf233
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Mon Feb 25 16:06:24 2019 -0500

    Fixed diff logic and tests

    - Fixed issues with diff core
    - Added 2 new unit tests since the last break went undetected by all automated
      testing.  ;-(

commit 426a2343e96dd1b67965db9a1f2a55637e6f9a56
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Mon Feb 25 19:42:52 2019 -0500

    Expand diff helper functions

    - Created new shared functions:
      - reduce_stanza(): To filtered down two stanzas to a list of keys
      - is_equal():  Simple boolean test find identical stanzas/configs.

commit 0dbfcbcfcad0a0dc19b85434a087ae65fa3dc300
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Fri Feb 22 01:55:48 2019 -0500

    Expand compare_stanza() algorithm

    - Updated the break down between compare_cfgs() and compare_stanzas() so that
      stanza-level changes, such as an added or removed stanza, can be detected by
      compare_stanzas() directly.

commit ccae564e21a6b9fae21e770ad6e2ce69a7556ac9
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Tue Feb 19 15:21:59 2019 -0500

    Expand diff header support

    - Revamp support for diff headers (first 2 lines showing which files are being
      compared).   Now these can be explicitly given a timestamp.

commit b8e2f667d6018b86a2b6b5ae3144b11a36ffb66d
Author: Lowell Alleman <lowell@kintyre.co>
Date:   Mon Feb 25 19:36:01 2019 -0500

    Refactor compare_cfg() into smaller bits

    - Refactor config comparisons into smaller bits.  Now individual stanzas can be
      compared using the compare_stanza() function, previously only a set of
      stanzas could be compared.
    - Created boolean munging function.  (To eventually should support comparing
      booleans in a type-aware way)

ksconf/conf/delta.py

@@ -11,7 +11,7 @@
 from ksconf.conf.parser import GLOBAL_STANZA, _format_stanza, default_encoding
 from ksconf.consts import EXIT_CODE_DIFF_EQUAL, EXIT_CODE_DIFF_CHANGE, EXIT_CODE_DIFF_NO_COMMON
 from ksconf.util.compare import _cmp_sets
-from ksconf.util.terminal import ANSI_RESET, ANSI_GREEN, ANSI_RED, tty_color, ANSI_YELLOW, ANSI_BOLD
+from ksconf.util.terminal import TermColor, ANSI_RESET, ANSI_GREEN, ANSI_RED, ANSI_YELLOW, ANSI_BOLD
 
 ####################################################################################################
 ## Diff logic
@@ -27,6 +27,63 @@
 DiffStzKey = namedtuple("DiffStzKey", ("type", "stanza", "key"))
 
 
+class DiffHeader(object):
+    def __init__(self, name, mtime=None):
+        self.name = name
+        if mtime:
+            self.mtime = mtime
+        else:
+            self.detect_mtime()
+
+    def detect_mtime(self):
+        try:
+            self.mtime = os.stat(self.name).st_mtime
+        except OSError:
+            self.mtime = 0
+
+    def __str__(self):
+        if isinstance(self.mtime, (int, float)):
+            ts = datetime.datetime.fromtimestamp(self.mtime)
+        else:
+            ts = self.mtime
+        return "{0:50} {1}".format(self.name, ts)
+
+
+def compare_stanzas(a, b, stanza_name):
+    if a == b:
+        return [DiffOp(DIFF_OP_EQUAL, DiffStanza("stanza", stanza_name), a, b) ]
+    elif not b:
+        # A only
+        return [ DiffOp(DIFF_OP_DELETE, DiffStanza("stanza", stanza_name), None, a) ]
+    elif not a:
+        # B only
+        return [ DiffOp(DIFF_OP_INSERT, DiffStanza("stanza", stanza_name), b, None) ]
+    else:
+        return list(_compare_stanzas(a, b, stanza_name))
+
+
+def _compare_stanzas(a, b, stanza_name):
+    kv_a, kv_common, kv_b = _cmp_sets(list(a.keys()), list(b.keys()))
+
+    if not kv_common:
+        # No keys in common, just swap
+        yield DiffOp(DIFF_OP_REPLACE, DiffStanza("stanza", stanza_name), a, b)
+        return
+
+    # Level 2 - Key comparisons
+    for key in kv_a:
+        yield DiffOp(DIFF_OP_DELETE, DiffStzKey("key", stanza_name, key), None, a[key])
+    for key in kv_b:
+        yield DiffOp(DIFF_OP_INSERT, DiffStzKey("key", stanza_name, key), b[key], None)
+    for key in kv_common:
+        a_ = a[key]
+        b_ = b[key]
+        if a_ == b_:
+            yield DiffOp(DIFF_OP_EQUAL, DiffStzKey("key", stanza_name, key), a_, b_)
+        else:
+            yield DiffOp(DIFF_OP_REPLACE, DiffStzKey("key", stanza_name, key), a_, b_)
+
+
 def compare_cfgs(a, b, allow_level0=True):
     '''
     Return list of 5-tuples describing how to turn a into b.
@@ -89,33 +146,13 @@ def compare_cfgs(a, b, allow_level0=True):
     else:
         all_stanzas = list(all_stanzas)
     all_stanzas = sorted(all_stanzas)
-
     for stanza in all_stanzas:
         if stanza in a and stanza in b:
-            a_ = a[stanza]
-            b_ = b[stanza]
-            # Note: make sure that '==' operator continues work with custom conf parsing classes.
-            if a_ == b_:
-                delta.append(DiffOp(DIFF_OP_EQUAL, DiffStanza("stanza", stanza), a_, b_))
-                continue
-            kv_a, kv_common, kv_b = _cmp_sets(list(a_.keys()), list(b_.keys()))
-            if not kv_common:
-                # No keys in common, just swap
-                delta.append(DiffOp(DIFF_OP_REPLACE, DiffStanza("stanza", stanza), a_, b_))
-                continue
-
-            # Level 2 - Key comparisons
-            for key in kv_a:
-                delta.append(DiffOp(DIFF_OP_DELETE, DiffStzKey("key", stanza, key), None, a_[key]))
-            for key in kv_b:
-                delta.append(DiffOp(DIFF_OP_INSERT, DiffStzKey("key", stanza, key), b_[key], None))
-            for key in kv_common:
-                a__ = a_[key]
-                b__ = b_[key]
-                if a__ == b__:
-                    delta.append(DiffOp(DIFF_OP_EQUAL, DiffStzKey("key", stanza, key), a__, b__))
-                else:
-                    delta.append(DiffOp(DIFF_OP_REPLACE, DiffStzKey("key", stanza, key), a__, b__))
+            if a[stanza] == b[stanza]:
+                delta.append(DiffOp(DIFF_OP_EQUAL, DiffStanza("stanza", stanza),
+                                    a[stanza], b[stanza]))
+            else:
+                delta.extend(_compare_stanzas(a[stanza], b[stanza], stanza))
         elif stanza in a:
             # A only
             delta.append(DiffOp(DIFF_OP_DELETE, DiffStanza("stanza", stanza), None, a[stanza]))
@@ -149,6 +186,12 @@ def summarize_cfg_diffs(delta, stream):
         stream.write("\n")
 
 
+def is_equal(delta):
+    """ Is the delta output show that the compared objects are identical """
+    # type: (list(DiffOp)) -> bool
+    return len(delta) == 1 and delta[0].tag == DIFF_OP_EQUAL
+
+
 # Color mapping
 _diff_color_mapping = {
     " ": ANSI_RESET,
@@ -158,24 +201,25 @@ def summarize_cfg_diffs(delta, stream):
 
 
 def _show_diff_header(stream, files, diff_line=None):
-    def header(sign, filename):
-        try:
-            mtime = os.stat(filename).st_mtime
-            ts = datetime.datetime.fromtimestamp(mtime)
-        except OSError:
-            ts = "1970-01-01 00:00:00"
-        stream.write("{0} {1:50} {2}\n".format(sign * 3, filename, ts))
-        tty_color(stream, ANSI_RESET)
+    headers = []
 
-    tty_color(stream, ANSI_YELLOW, ANSI_BOLD)
-    if diff_line:
-        stream.write("diff {} {} {}\n".format(diff_line, files[0], files[1]))
-    tty_color(stream, ANSI_RESET)
-    header("-", files[0])
-    header("+", files[1])
+    for f in files:
+        if isinstance(f, DiffHeader):
+            headers.append(f)
+        else:
+            headers.append(DiffHeader(f))
+
+    with TermColor(stream) as tc:
+        tc.color(ANSI_YELLOW, ANSI_BOLD)
+        if diff_line:
+            stream.write("diff {} {} {}\n".format(diff_line, headers[0].name, headers[1].name))
+        tc.reset()
+        stream.write("--- {0}\n".format(headers[0]))
+        stream.write("+++ {0}\n".format(headers[1]))
 
 
 def show_diff(stream, diffs, headers=None):
+    tc = TermColor(stream)
     def write_key(key, value, prefix_=" "):
         if "\n" in value:
             write_multiline_key(key, value, prefix_)
@@ -187,24 +231,24 @@ def write_key(key, value, prefix_=" "):
             stream.write(template.format(prefix_, key, value))
 
     def write_multiline_key(key, value, prefix_=" "):
-        lines = value.replace("\n", "\\\n").split("\n")
-        tty_color(stream, _diff_color_mapping.get(prefix_))
-        stream.write("{0}{1} = {2}\n".format(prefix_, key, lines.pop(0)))
-        for line in lines:
-            stream.write("{0}{1}\n".format(prefix_, line))
-        tty_color(stream, ANSI_RESET)
+        with tc:
+            lines = value.replace("\n", "\\\n").split("\n")
+            tc.color(_diff_color_mapping.get(prefix_))
+            stream.write("{0}{1} = {2}\n".format(prefix_, key, lines.pop(0)))
+            for line in lines:
+                stream.write("{0}{1}\n".format(prefix_, line))
 
     def show_value(value, stanza_, key, prefix_=""):
-        tty_color(stream, _diff_color_mapping.get(prefix_))
-        if isinstance(value, dict):
-            if stanza_ is not GLOBAL_STANZA:
-                stream.write("{0}[{1}]\n".format(prefix_, stanza_))
-            for x, y in sorted(six.iteritems(value)):
-                write_key(x, y, prefix_)
-            stream.write("\n")
-        else:
-            write_key(key, value, prefix_)
-        tty_color(stream, ANSI_RESET)
+        with tc:
+            tc.color(_diff_color_mapping.get(prefix_))
+            if isinstance(value, dict):
+                if stanza_ is not GLOBAL_STANZA:
+                    stream.write("{0}[{1}]\n".format(prefix_, stanza_))
+                for x, y in sorted(six.iteritems(value)):
+                    write_key(x, y, prefix_)
+                stream.write("\n")
+            else:
+                write_key(key, value, prefix_)
 
     def show_multiline_diff(value_a, value_b, key):
         def f(v):
@@ -215,17 +259,18 @@ def f(v):
         a = f(value_a)
         b = f(value_b)
         differ = difflib.Differ()
-        for d in differ.compare(a, b):
-            # Someday add "?" highlighting.  Trick is this should change color mid-line on the
-            # previous (one or two) lines.  (Google and see if somebody else solved this one already)
-            # https://stackoverflow.com/questions/774316/python-difflib-highlighting-differences-inline
-            tty_color(stream, _diff_color_mapping.get(d[0], 0))
-            # Differences in how difflib returns bytes/unicode?
-            if not isinstance(d, six.text_type):
-                d = d.decode(default_encoding)
-            stream.write(d)
-            tty_color(stream, ANSI_RESET)
-            stream.write("\n")
+        with tc:
+            for d in differ.compare(a, b):
+                # Someday add "?" highlighting.  Trick is this should change color mid-line on the
+                # previous (one or two) lines.  (Google and see if somebody else solved this one already)
+                # https://stackoverflow.com/questions/774316/python-difflib-highlighting-differences-inline
+                tc.color(_diff_color_mapping.get(d[0], 0))
+                # Differences in how difflib returns bytes/unicode?
+                if not isinstance(d, six.text_type):
+                    d = d.decode(default_encoding)
+                stream.write(d)
+                tc.reset()
+                stream.write("\n")
 
     # Global result:  no changes between files or no commonality between files
     if len(diffs) == 1 and isinstance(diffs[0].location, DiffGlobal):
@@ -284,10 +329,21 @@ def show_text_diff(stream, a, b):
     differ = difflib.Differ()
     lines_a = open(a, "r", encoding=default_encoding).readlines()
     lines_b = open(b, "r", encoding=default_encoding).readlines()
-    for d in differ.compare(lines_a, lines_b):
-        # Someday add "?" highlighting.  Trick is this should change color mid-line on the
-        # previous (one or two) lines.  (Google and see if somebody else solved this one already)
-        # https://stackoverflow.com/questions/774316/python-difflib-highlighting-differences-inline
-        tty_color(stream, _diff_color_mapping.get(d[0], 0))
-        stream.write(d)
-        tty_color(stream, ANSI_RESET)
+    with TermColor(stream) as tc:
+        for d in differ.compare(lines_a, lines_b):
+            # Someday add "?" highlighting.  Trick is this should change color mid-line on the
+            # previous (one or two) lines.  (Google and see if somebody else solved this one already)
+            # https://stackoverflow.com/questions/774316/python-difflib-highlighting-differences-inline
+            tc.color(_diff_color_mapping.get(d[0], 0))
+            stream.write(d)
+            tc.reset()
+
+def reduce_stanza(stanza, keep_attrs):
+    """ Pre-process a stanzas so that only a common set of keys will be compared.
+    :param stanza: Stanzas containing attributes and values
+    :type stanza: dict
+    :param keep_attrs: Listing of
+    :type keep_attrs: (list, set, tuple, dict)
+    :return: a reduced copy of ``stanza``.
+    """
+    return {attr: value for attr, value in six.iteritems(stanza) if attr in keep_attrs}

ksconf/conf/parser.py

@@ -402,3 +402,23 @@ def _drop_stanza_comments(stanza):
             continue
         n[key] = value
     return n
+
+
+def conf_attr_boolean(value):
+    if isinstance(value, bool):
+        return value
+    elif isinstance(value, six.string_types):
+        value = value.lower()
+        if value in ("1", "t", "y", "true", "yes"):
+            return True
+        elif value in ("0", "f", "n", "false", "no"):
+            return False
+        else:
+            raise ValueError("Can't convert {!r} to a boolean.".format(value))
+    elif isinstance(value, int):
+        if value == 0:
+            return False
+        else:
+            return True
+    else:
+        raise ValueError("Can't convert type {} to a boolean.".format(type(value)))

ksconf/util/terminal.py

@@ -12,3 +12,32 @@
 def tty_color(stream, *codes):
     if codes and FORCE_TTY_COLOR or hasattr(stream, "isatty") and stream.isatty():
         stream.write("\x1b[{}m".format(";".join([str(i) for i in codes])))
+
+
+class TermColor(object):
+    """
+    Simple color setting helper class that's a context manager wrapper around a stream.
+    This ensure that the color is always reset at the end of a session.
+    """
+    def __init__(self, stream):
+        self.stream = stream
+        if FORCE_TTY_COLOR or hasattr(stream, "isatty") and stream.isatty():
+            self.color_enabled = True
+        else:
+            self.color_enabled = False
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.reset()
+
+    def write(self, content):
+        return self.write(content)
+
+    def color(self, *codes):
+        if codes and self.color_enabled:
+            self.stream.write("\x1b[{}m".format(";".join([str(i) for i in codes])))
+
+    def reset(self):
+        self.color(ANSI_RESET)