New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Final SQLite3 data store backend #75

Merged
merged 6 commits into from Oct 28, 2018

Conversation

Projects
None yet
2 participants
@jarvisms
Contributor

jarvisms commented Oct 27, 2018

Tested on Python 3.7.0, 3.6.5, 3.5.3 and 2.7.13 (where 3.5.3 and 2.7.13 are native to Raspbian).

jarvisms added some commits Oct 9, 2018

SQLite3 datastore module and transfer utility
First release candidate for an SQLite3 backend and general data store transfer script.

Signed-off-by: Mark Jarvis <jarvism@thisaddressdoesnotexist.co.uk>
Empty iterable to update method should not fail
If update() is fed an empty iterable, it should not cause an exception to be raised.

If update() or __setitem__() genuinely fail, sqlite3 will throw its own exception, so the rowcount==0 check is entirely redundant so has been removed.

Signed-off-by: Mark Jarvis <jarvism@thisaddressdoesnotexist.co.uk>
Modification to process.py to resolve Issue #1
Wrapped the main processing loops into nested generator functions allowing the use of .update() so data writes are bulk committed with less overhead and less write ahead log payload on sqlite3.

Signed-off-by: Mark Jarvis <jarvism@thisaddressdoesnotexist.co.uk>
Rewrite for PEP8 compliance and Py2.7 compatibility
Restyled for PEP8 compliance (issue #2). Rewrote f-strings, timestamp manipulations and dictionary merging to be backward compatible to Py2.7 (Issue #3).  A few other simplificiations and tweaks along the way.

Signed-off-by: Mark Jarvis <jarvism@thisaddressdoesnotexist.co.uk>

@jim-easterbrook jim-easterbrook merged commit 33e10ef into jim-easterbrook:master Oct 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment