Cache manager refactor #40

c-w · 2016-02-01T23:49:11Z

@cpeel: I did a bit of refactoring on your metadata cache manager code in order to gain an understanding of it. I'd like to get some feedback on whether I understood everything correctly before I merge.

The gist of this refactor:

Create sqlite/bsddb subclasses of MetadataCache to avoid store-type checking in methods
Create testing MetadataCache to remove testing-only code in the api.
Fall back to sqlite cache when the user doesn't have bsddb installed.

This will allow for easier instantiation of test cache managers.

This lets us make lines shorter!

Really there are only two supported stores: - Sleepycat/BDB - Sqlite

The sqlite-based MetadataCache should work on all systems (given that sqlite is in the Python standard library). Thus we can use this cache as a fall-back in case we don't manage to use Sleepycat (e.g. on Py3/Win).

Also: - remove explicit dependency on bsddb3 under py2 There are some issues installing BSD-DB on Windows (see #26, #31) so this fix should make gutenberg work on Windows Python 2 again. On Windows Python 3, we can't use the standard-library bsddb module and we can't fall-back to bsddb3 so we use the slow SQLite cache instead.

cpeel · 2016-02-02T01:06:02Z

gutenberg/acquire/metadata.py


-def set_metadata_cache_manager(cache_manager):


The set_metadata_cache() function is not just for testing, but rather an integral part of my branch. Without a way to explicitly set a cache, you can't control where the cache is stored on the filesystem, or in the new code, control which of the two backend types you want to use. This level of control is vital when using this as part of a larger service.

For an example of what I'm trying to achieve with this module, see:
https://github.com/cpeel/openlibrary/blob/095df4b3be522eef218813a0106ee2ce7934f120/openlibrary/plugins/gutenberg/pgebook.py
and
https://github.com/cpeel/openlibrary/blob/73533a3e011074e2ffbfe416f241a6aa14fb5b6e/scripts/dev-instance/populate_gutenberg_cache.py

Fixed in 8df3e24.

cpeel · 2016-02-02T01:18:24Z

Overall I like your refactor -- very clean. My one big concern is the removal of set_metadata_cache[_manager]() as that function enables one of the core requirements of what I was trying to achieve -- the ability to control where the cache ends up on the filesystem and what type of cache it is.

cpeel · 2016-02-02T22:13:26Z

gutenberg/_domain_model/exceptions.py

@@ -23,9 +23,5 @@ class CacheAlreadyExists(Error):
    pass


-class CacheNotRemovable(Error):


Built-in python errors (almost) all end in Exception, Error, or Warning (https://docs.python.org/2/library/exceptions.html). See also PEP8: https://www.python.org/dev/peps/pep-0008/#exception-names

(I should have raised this comment against the commit when the exceptions.py file went in, as they're all this way.)

Fixed in 8921ee1.

This reverts commit 7c8f7c1 and adds some more documentation on how to use the method.

c-w · 2016-02-05T20:59:22Z

Added exception suffixes in 8921ee1.
Froze dependency on sqlalchemy-rdflib in d887023.
Brought back set_metadata_cache in 8df3e24.

cpeel · 2016-02-06T01:19:29Z

Ship it! :)

c-w · 2016-02-06T02:14:39Z

           (`-,-,
           ('(_,( )
            _   `_'
         __|_|__|_|_
       _|___________|__
      |o o o o o o o o/   
     ~'`~'`~'`~'`~'`~'`~

Cache manager refactor

c-w added 24 commits January 31, 2016 22:58

Turn _iter_metadata_triples into a class-method

9013206

This will allow for easier instantiation of test cache managers.

Shorten imports

5242188

Remove testing argument to cache manager populate

05bbf9d

Prefer single quotes over doubles

05afe8d

Remove unnecessary call to re.search

4432234

Prefer from contextlib import X over contextlib.X

af27b63

This lets us make lines shorter!

Make number of metadata stores explicit

8bcc43f

Really there are only two supported stores: - Sleepycat/BDB - Sqlite

Remove unused removable=False code path

a9c6cb2

Static import set_metadata_cache_manager

8f27be4

Extract SqliteMetadataCacheManager subclass

4b80ce1

Extract SleepycatMetadataCacheManager subclass

be62f1a

Turn exists into a property

3872cdb

Turn _get_local_storage_path into a property

23233b8

Rename MetadataCacheManager to MetadataCache

3156b0d

Remove direct reference to _METADATA_CACHE

79a19f7

Rename cache_open to is_open

b4dc4a5

Remove early return

d4c8e92

Extract set_text_cache function

170386c

Remove testing-only set_metadata_cache function

7c8f7c1

Remove duplicate calls to format catalog_source

556f189

Use path for MetadataCache constructors

0d48f3d

Take an explicit dependency on rdflib-sqlite

9feaeae

The sqlite-based MetadataCache should work on all systems (given that sqlite is in the Python standard library). Thus we can use this cache as a fall-back in case we don't manage to use Sleepycat (e.g. on Py3/Win).

Rename test cache to encode bsddb dependency

c65da7f

cpeel reviewed Feb 2, 2016
View reviewed changes

c-w added 3 commits February 5, 2016 12:28

Freeze rdflib-sqlalchemy dependency

d887023

Add Exception suffix to custom errors

8921ee1

Bring back set_metadata_cache

8df3e24

This reverts commit 7c8f7c1 and adds some more documentation on how to use the method.

Fix typo

1932aa7

c-w closed this Feb 6, 2016

c-w reopened this Feb 6, 2016

c-w added a commit that referenced this pull request Feb 6, 2016

Merge pull request #40 from c-w/cache-manager-refactor

c7343e9

Cache manager refactor

c-w merged commit c7343e9 into master Feb 6, 2016

c-w deleted the cache-manager-refactor branch February 6, 2016 02:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache manager refactor #40

Cache manager refactor #40

c-w commented Feb 1, 2016

cpeel Feb 2, 2016

c-w Feb 5, 2016

cpeel commented Feb 2, 2016

cpeel Feb 2, 2016

c-w Feb 5, 2016

c-w commented Feb 5, 2016

cpeel commented Feb 6, 2016

c-w commented Feb 6, 2016

		@@ -23,9 +23,5 @@ class CacheAlreadyExists(Error):
		pass


		class CacheNotRemovable(Error):


		def set_metadata_cache_manager(cache_manager):

Cache manager refactor #40

Cache manager refactor #40

Conversation

c-w commented Feb 1, 2016

cpeel Feb 2, 2016

Choose a reason for hiding this comment

c-w Feb 5, 2016

Choose a reason for hiding this comment

cpeel commented Feb 2, 2016

cpeel Feb 2, 2016

Choose a reason for hiding this comment

c-w Feb 5, 2016

Choose a reason for hiding this comment

c-w commented Feb 5, 2016

cpeel commented Feb 6, 2016

c-w commented Feb 6, 2016