Limit discovery cache by size

[golanha]

This change is 

[yehiyam]

Reviewed 5 of 5 files at r1.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @golanha)

---

hkube_python_wrapper/cache/caching.py, line 2 at r1 (raw file):

import datetime
import msgpack

Why do you use msgpack and not the configured encoding?

---

hkube_python_wrapper/cache/caching.py, line 5 at r1 (raw file):

import hkube_python_wrapper.util.type_check as typeCheck

class CustomCache:

Maybe just Cache?

---

hkube_python_wrapper/cache/caching.py, line 20 at r1 (raw file):

            return None
        while (self.sumSize + size) >= self._maxCacheSize * 1000 * 1000:
            if (self._cache.keys()):

shouldn't it be if (!self._cache.keys())? You check if the cache is empty

---

hkube_python_wrapper/config/config.py, line 28 at r1 (raw file):

    "timeout": getIntEnv('DISCOVERY_TIMEOUT', 10000),
    "networkTimeout": getIntEnv('DISCOVERY_NETWORK_TIMEOUT', 1000),
    "maxCacheSize": getIntEnv('DISCOVERY_MAX_CACHE_SIZE_MB', 1500)

Isn't it too large as default?

---

hkube_python_wrapper/config/config.py, line 39 at r1 (raw file):

    "mode": os.environ.get('STORAGE_PROTOCOL', 'v2'),
    "encoding": os.environ.get('STORAGE_ENCODING', 'bson'),
    "maxCacheSize": getIntEnv('STORAGE_MAX_CACHE_SIZE_MB', 1500),

Together with this cache it is 3GB of cache.

---

hkube_python_wrapper/wrapper/data_adapter.py, line 198 at r1 (raw file):

        return self._storageCache.get(path)

    def _setToCache(self, path, data, size):

maybe add @timing to this method to see if the encoding takes too long

[golanha]

Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @timing and @yehiyam)

---

hkube_python_wrapper/cache/caching.py, line 2 at r1 (raw file):

Previously, yehiyam wrote…

Why do you use msgpack and not the configured encoding?

It doesn't really matter which encoding I use, since it is not saved as encoded any ways.

---

hkube_python_wrapper/cache/caching.py, line 5 at r1 (raw file):

Previously, yehiyam wrote…

Maybe just Cache?

refactored

---

hkube_python_wrapper/cache/caching.py, line 20 at r1 (raw file):

Previously, yehiyam wrote…

shouldn't it be if (!self._cache.keys())? You check if the cache is empty

Right, Done.

---

hkube_python_wrapper/config/config.py, line 28 at r1 (raw file):

Previously, yehiyam wrote…

Isn't it too large as default?

changed to 400MB

---

hkube_python_wrapper/config/config.py, line 39 at r1 (raw file):

Previously, yehiyam wrote…

Together with this cache it is 3GB of cache.

changed to 400 MB each cache

---

hkube_python_wrapper/wrapper/data_adapter.py, line 198 at r1 (raw file):

Previously, yehiyam wrote…

maybe add @timing to this method to see if the encoding takes too long

Done.

[yehiyam]

Reviewed 7 of 7 files at r2.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @golanha)

---

hkube_python_wrapper/cache/caching.py, line 19 at r2 (raw file):

                size = len(value)
            else:
                size = asizeof(value)

fix

[yehiyam]

Reviewed 1 of 1 files at r3, 1 of 1 files at r4.
Reviewable status: complete! all files reviewed, all discussions resolved