Merge pull request #188 from seperman/dev

5.0.0: Delta, Better Numpy support, Pretty view

.gitignore

@@ -63,5 +63,9 @@ target/
 .idea/
 .~lock*
 
-.python-version
+.python-version*
 
+temp*
+
+# env file
+.env

.travis.yml

@@ -12,7 +12,7 @@ matrix:
 install:
   - pip install -r requirements-dev.txt
 
-script: pytest --cov=deepdiff tests/
+script: pytest --cov=deepdiff tests/  --runslow
 
 after_success:
   - coveralls

AUTHORS

@@ -1,8 +1,10 @@
+Developer:
+
 - Seperman
-- Victor Hahn Castell @ Flexoptix
 
-And many thanks to the following people for their contributions to DeepDiff:
+And many thanks to the following people for their contributions to DeepDiff!
 
+- Victor Hahn Castell @ Flexoptix for tree view and major contributions.
 - nfvs for Travis-CI setup script.
 - brbsix for initial Py3 porting.
 - WangFenjin for Unicode support.
@@ -26,3 +28,7 @@ And many thanks to the following people for their contributions to DeepDiff:
 - Hugo (hugovk) for fixes for Python 3.10 and dropping support for EOL Python 3.4
 - Andrey Gavrilin (gaal-dev) for hashing classes.
 - Ivan Piskunov (van-ess0) for deprecation warning enhancement.
+- Nathaniel Brown (nathanielobrown) Adds support for datetime.time
+- Michał Karaś (MKaras93) for the pretty view
+- Christian Kothe (chkothe) for the basic support for diffing numpy arrays
+- Timothy (timson) for truncate_datetime

CHANGELOG.md

@@ -0,0 +1,48 @@
+DeepDiff Change log
+
+- v5-0-0: Introducing the Delta object, Improving Numpy support, Fixing tuples comparison when ignore_order=True, Dramatically improving the results when ignore_order=True by running in passes, Introducing pretty print view, deep_distance, purge, progress logging, cache and truncate_datetime.
+- v4-3-3: Adds support for datetime.time
+- v4-3-2: Deprecation Warning Enhancement
+- v4-3-1: Fixing the issue with exclude_path and hash calculations when dictionaries were inside iterables. https://github.com/seperman/deepdiff/issues/174
+- v4-3-0: adding exclude_obj_callback
+- v4-2-0: .json property is finally removed. Fix for Py3.10. Dropping support for EOL Python 3.4. Ignoring private keys when calculating hashes. For example __init__ is not a part of hash calculation anymore. Fix for #166 Problem with comparing lists, with an boolean as element.
+- v4-0-9: Fixing the bug for hashing custom unhashable objects
+- v4-0-8: Adding ignore_nan_inequality for float('nan')
+- v4-0-7: Hashing of the number 1 vs. True
+- v4-0-6: found a tiny bug in Python formatting of numbers in scientific notation. Added a workaround.
+- v4-0-5: Fixing number diffing. Adding number_format_notation and number_to_string_func.
+- v4-0-4: Adding ignore_string_case and ignore_type_subclasses
+- v4-0-3: Adding versionbump tool for release
+- v4-0-2: Fixing installation issue where rst files are missing.
+- v4-0-1: Fixing installation Tarball missing requirements.txt . DeepDiff v4+ should not show up as pip installable for Py2. Making Murmur3 installation optional.
+- v4-0-0: Ending Python 2 support, Adding more functionalities and documentation for DeepHash. Switching to Pytest for testing. Switching to Murmur3 128bit for hashing. Fixing classes which inherit from classes with slots didn't have all of their slots compared. Renaming ContentHash to DeepHash. Adding exclude by path and regex path to DeepHash. Adding ignore_type_in_groups. Adding match_string to DeepSearch. Adding Timedelta object diffing.
+- v3-5-0: Exclude regex path
+- v3-3-0: Searching for objects and class attributes
+- v3-2-2: Adding help(deepdiff)
+- v3-2-1: Fixing hash of None
+- v3-2-0: Adding grep for search: object | grep(item)
+- v3-1-3: Unicode vs. Bytes default fix
+- v3-1-2: NotPresent Fix when item is added or removed.
+- v3-1-1: Bug fix when item value is None (#58)
+- v3-1-0: Serialization to/from json
+- v3-0-0: Introducing Tree View
+- v2-5-3: Bug fix on logging for content hash.
+- v2-5-2: Bug fixes on content hash.
+- v2-5-0: Adding ContentHash module to fix ignore_order once and for all.
+- v2-1-0: Adding Deep Search. Now you can search for item in an object.
+- v2-0-0: Exclusion patterns better coverage. Updating docs.
+- v1-8-0: Exclusion patterns.
+- v1-7-0: Deep Set comparison.
+- v1-6-0: Unifying key names. i.e newvalue is new_value now. For backward compatibility, newvalue still works.
+- v1-5-0: Fixing ignore order containers with unordered items. Adding significant digits when comparing decimals. Changes property is deprecated.
+- v1-1-0: Changing Set, Dictionary and Object Attribute Add/Removal to be reported as Set instead of List. Adding Pypy compatibility.
+- v1-0-2: Checking for ImmutableMapping type instead of dict
+- v1-0-1: Better ignore order support
+- v1-0-0: Restructuring output to make it more useful. This is NOT backward compatible.
+- v0-6-1: Fixiing iterables with unhashable when order is ignored
+- v0-6-0: Adding unicode support
+- v0-5-9: Adding decimal support
+- v0-5-8: Adding ignore order of unhashables support
+- v0-5-7: Adding ignore order support
+- v0-5-6: Adding slots support
+- v0-5-5: Adding loop detection

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2014 - 2016 Sep Ehr (Seperman) and contributors
+Copyright (c) 2014 - 2020 Sep Dehpour (Seperman) and contributors
 www.zepworks.com
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

MANIFEST.in

@@ -1,5 +1,6 @@
 include LICENSE
 include AUTHORS
+include CHANGELOG
 include *.rst
 include deepdiff/*.rst
 include *.txt

README.md

@@ -1,4 +1,4 @@
-# DeepDiff v 4.3.2
+# DeepDiff v 5.0.0
 
 <!-- ![Downloads](https://img.shields.io/pypi/dm/deepdiff.svg?style=flat) -->
 ![Python Versions](https://img.shields.io/pypi/pyversions/deepdiff.svg?style=flat)
@@ -47,6 +47,7 @@ DeepDiff gets the difference of 2 objects.
 
 > - Please take a look at the [DeepDiff docs](deepdiff/diff_doc.rst)
 > - The full documentation can be found on <https://deepdiff.readthedocs.io>
+> - Tutorials can be found on <https://zepworks.com/tags/deepdiff/>
 
 ## A few Examples
 
@@ -417,50 +418,7 @@ And then running
 
 # ChangeLog
 
-- v4-3-2: Deprecation Warning Enhancement
-- v4-3-1: Fixing the issue with exclude_path and hash calculations when dictionaries were inside iterables. https://github.com/seperman/deepdiff/issues/174
-- v4-3-0: adding exclude_obj_callback
-- v4-2-0: .json property is finally removed. Fix for Py3.10. Dropping support for EOL Python 3.4. Ignoring private keys when calculating hashes. For example __init__ is not a part of hash calculation anymore. Fix for #166 Problem with comparing lists, with an boolean as element.
-- v4-0-9: Fixing the bug for hashing custom unhashable objects
-- v4-0-8: Adding ignore_nan_inequality for float('nan')
-- v4-0-7: Hashing of the number 1 vs. True
-- v4-0-6: found a tiny bug in Python formatting of numbers in scientific notation. Added a workaround.
-- v4-0-5: Fixing number diffing. Adding number_format_notation and number_to_string_func.
-- v4-0-4: Adding ignore_string_case and ignore_type_subclasses
-- v4-0-3: Adding versionbump tool for release
-- v4-0-2: Fixing installation issue where rst files are missing.
-- v4-0-1: Fixing installation Tarball missing requirements.txt . DeepDiff v4+ should not show up as pip installable for Py2. Making Murmur3 installation optional.
-- v4-0-0: Ending Python 2 support, Adding more functionalities and documentation for DeepHash. Switching to Pytest for testing. Switching to Murmur3 128bit for hashing. Fixing classes which inherit from classes with slots didn't have all of their slots compared. Renaming ContentHash to DeepHash. Adding exclude by path and regex path to DeepHash. Adding ignore_type_in_groups. Adding match_string to DeepSearch. Adding Timedelta object diffing.
-- v3-5-0: Exclude regex path
-- v3-3-0: Searching for objects and class attributes
-- v3-2-2: Adding help(deepdiff)
-- v3-2-1: Fixing hash of None
-- v3-2-0: Adding grep for search: object | grep(item)
-- v3-1-3: Unicode vs. Bytes default fix
-- v3-1-2: NotPresent Fix when item is added or removed.
-- v3-1-1: Bug fix when item value is None (#58)
-- v3-1-0: Serialization to/from json
-- v3-0-0: Introducing Tree View
-- v2-5-3: Bug fix on logging for content hash.
-- v2-5-2: Bug fixes on content hash.
-- v2-5-0: Adding ContentHash module to fix ignore_order once and for all.
-- v2-1-0: Adding Deep Search. Now you can search for item in an object.
-- v2-0-0: Exclusion patterns better coverage. Updating docs.
-- v1-8-0: Exclusion patterns.
-- v1-7-0: Deep Set comparison.
-- v1-6-0: Unifying key names. i.e newvalue is new_value now. For backward compatibility, newvalue still works.
-- v1-5-0: Fixing ignore order containers with unordered items. Adding significant digits when comparing decimals. Changes property is deprecated.
-- v1-1-0: Changing Set, Dictionary and Object Attribute Add/Removal to be reported as Set instead of List. Adding Pypy compatibility.
-- v1-0-2: Checking for ImmutableMapping type instead of dict
-- v1-0-1: Better ignore order support
-- v1-0-0: Restructuring output to make it more useful. This is NOT backward compatible.
-- v0-6-1: Fixiing iterables with unhashable when order is ignored
-- v0-6-0: Adding unicode support
-- v0-5-9: Adding decimal support
-- v0-5-8: Adding ignore order of unhashables support
-- v0-5-7: Adding ignore order support
-- v0-5-6: Adding slots support
-- v0-5-5: Adding loop detection
+Please take a look at the [changelog](changelog.md) file.
 
 # Releases
 
@@ -475,21 +433,28 @@ git push && git push --tags
 # Contribute
 
 1. Please make your PR against the dev branch
-2. Please make sure that your PR has tests. Since DeepDiff is used in many sensitive data driven projects, we maintain 100% test coverage on the code. There are occasiannly exceptions to that rule but that is rare.
+2. Please make sure that your PR has tests. Since DeepDiff is used in many sensitive data driven projects, we strive to maintain around 100% test coverage on the code.
+
+Please run `pytest --cov=deepdiff` to see the coverage report.
+
+Or to see a more user friendly version, please run: `pytest --cov=deepdiff --cov-report term-missing`.
 
 Thank you!
 
 # Authors
 
+Developer:
+
 - Seperman (Sep Dehpour)
     - [Github](https://github.com/seperman)
     - [Linkedin](http://www.linkedin.com/in/sepehr)
     - [ZepWorks](http://www.zepworks.com)
 
-- Victor Hahn Castell for major contributions
+And many thanks to the following people for their contributions to DeepDiff!
+
+- Victor Hahn Castell for the tree view and major contributions:
     - [hahncastell.de](http://hahncastell.de)
     - [flexoptix.net](http://www.flexoptix.net)
-
 - nfvs for Travis-CI setup script.
 - brbsix for initial Py3 porting.
 - WangFenjin for unicode support.
@@ -512,3 +477,4 @@ Thank you!
 - Necrophagos for Hashing of the number 1 vs. True
 - gaal-dev for adding exclude_obj_callback
 - Ivan Piskunov (van-ess0) for deprecation warning enhancement.
+- Michał Karaś (MKaras93) for the pretty view

conftest.py

@@ -1,4 +1,64 @@
 import sys
 import os
+import json
+import pytest
 
 sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), 'tests')))
+
+FIXTURES_DIR = os.path.join(os.path.dirname(__file__), 'tests/fixtures/')
+
+
+def pytest_addoption(parser):
+    parser.addoption(
+        "--runslow", action="store_true", default=False, help="run slow tests"
+    )
+
+
+def pytest_configure(config):
+    config.addinivalue_line("markers", "slow: mark test as slow to run")
+
+
+def pytest_collection_modifyitems(config, items):
+    if config.getoption("--runslow"):
+        # --runslow given in cli: do not skip slow tests
+        return
+    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
+    for item in items:
+        if "slow" in item.keywords:
+            item.add_marker(skip_slow)
+
+
+@pytest.fixture(scope='class')
+def nested_a_t1():
+    with open(os.path.join(FIXTURES_DIR, 'nested_a_t1.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def nested_a_t2():
+    with open(os.path.join(FIXTURES_DIR, 'nested_a_t2.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def nested_a_result():
+    with open(os.path.join(FIXTURES_DIR, 'nested_a_result.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def nested_b_t1():
+    with open(os.path.join(FIXTURES_DIR, 'nested_b_t1.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def nested_b_t2():
+    with open(os.path.join(FIXTURES_DIR, 'nested_b_t2.json')) as the_file:
+        return json.load(the_file)
+
+
+@pytest.fixture(scope='class')
+def nested_b_result():
+    with open(os.path.join(FIXTURES_DIR, 'nested_b_result.json')) as the_file:
+        return json.load(the_file)

deepdiff/__init__.py

@@ -1,6 +1,6 @@
-"""This module offers the DeepDiff, DeepSearch, grep and DeepHash classes."""
+"""This module offers the DeepDiff, DeepSearch, grep, Delta and DeepHash classes."""
 # flake8: noqa
-__version__ = '4.3.2'
+__version__ = '5.0.0'
 import logging
 
 if __name__ == '__main__':
@@ -10,3 +10,5 @@
 from .diff import DeepDiff
 from .search import DeepSearch, grep
 from .deephash import DeepHash
+from .delta import Delta
+from .path import extract

deepdiff/anyset.py

@@ -0,0 +1,66 @@
+from ordered_set import OrderedSet
+from deepdiff.deephash import DeepHash
+from deepdiff.helper import dict_
+
+
+class AnySet:
+    """
+    Any object can be in this set whether hashable or not.
+    Note that the current implementation has memory leak and keeps
+    traces of objects in itself even after popping.
+    However one the AnySet object is deleted, all those traces will be gone too.
+    """
+    def __init__(self, items=None):
+        self._set = OrderedSet()
+        self._hashes = dict_()
+        self._hash_to_objects = dict_()
+        if items:
+            for item in items:
+                self.add(item)
+
+    def add(self, item):
+        try:
+            self._set.add(item)
+        except TypeError:
+            hashes_obj = DeepHash(item, hashes=self._hashes)
+            hash_ = hashes_obj[item]
+            if hash_ not in self._hash_to_objects:
+                self._hash_to_objects[hash_] = item
+
+    def __contains__(self, item):
+        try:
+            result = item in self._set
+        except TypeError:
+            hashes_obj = DeepHash(item, hashes=self._hashes)
+            hash_ = hashes_obj[item]
+            result = hash_ in self._hash_to_objects
+        return result
+
+    def pop(self):
+        if self._set:
+            return self._set.pop()
+        else:
+            return self._hash_to_objects.pop(next(iter(self._hash_to_objects)))
+
+    def __eq__(self, other):
+        set_part, hashes_to_objs_part = other
+        return (self._set == set_part and self._hash_to_objects == hashes_to_objs_part)
+
+    __req__ = __eq__
+
+    def __repr__(self):
+        return "< AnySet {}, {} >".format(self._set, self._hash_to_objects)
+
+    __str__ = __repr__
+
+    def __len__(self):
+        return len(self._set) + len(self._hash_to_objects)
+
+    def __iter__(self):
+        for item in self._set:
+            yield item
+        for item in self._hash_to_objects.values():
+            yield item
+
+    def __bool__(self):
+        return bool(self._set or self._hash_to_objects)

deepdiff/base.py

@@ -3,6 +3,7 @@
 
 
 DEFAULT_SIGNIFICANT_DIGITS_WHEN_IGNORE_NUMERIC_TYPES = 12
+TYPE_STABILIZATION_MSG = 'Unable to stabilize the Numpy array {} due to {}. Please set ignore_order=False.'
 
 
 class Base: