DB API2 support in SmartHome.py used in SQLite plugin

[ohinckel]

This pull request implements three things:

• make it possible to use different database backends in core or plugins
• make use of database backend accessor in SQLite plugin
• implement new plugin dblog which logs item changes to a database and use different implementation (currently only tested with SQLite, but should also work with any MySQL DB API2 implementation)

I implemented this in one pull request, since the dblog plugin depends on the DB API enhancement. The documentation is also updated and can be provided in a new pull request when this is merged.

Using database backends

I thought about how to use different database implementation / backends with SmartHome.py and came to this implementation.

It adds a new configuration setting to smarthome.conf called db. This configures the database backend implementation (need to implement the DB API 2 specifications to have a common interface available) and defaults to sqlite:sqlite3, which means the SQLite API is available using the alias sqlite and the Python module sqlite3.

Since SQLite is bundled with Python directly you do not need to explicitely configure this - it's always added as default if not registered explicitely. It's also possible to register multiple database backends using |, e.g.
db = sqlite:sqlite3 | mysql:mysqldb.

Additionally to this configuration the SmartHome.py core have a new function called dbapi(alias) which returns the module for the given alias (e.g. in the case above sh.dbapi('sqlite') will return the sqlite3 module and you can do something like this:

db = sh.dbapi('sqlite')
db.connect(...)
db.execute(...)

For convenience a new class lib.db.Database was added which can be constructed by using the following code

 lib.db.Database("NameIdentifier", sh.dbapi("sqlite"), {"database" : "/path/to/sqlite.db"})

The first identifier argument is just a name which is used when something is written to the log and one want to identifier which database connection is related to a log message. Additionally to this, it is also used internally (e.g. to name the version table) - so just us characters like [a-zA-Z0-9_] and do not start with numbers.

The second DB API parameter is the module of the database backend implementation. In this example it will use the SQLite backend implementation which is registered as default.

The third connect parameter is a dict containing all connection parameters required to invoke the module's connect() function. Alternatively this can also be a simple string using a map-like syntax of parameters, e.g. host:localhost | port:1234 | ... - see also below.

Use DB API in SQLite plugin

This pull request will make use of this new feature in the SQLite plugin already to be able to use other database backends later (not part of this pull request and may be implemented in a new pull request and maybe a new plugin which could obsolete the SQLite plugin; suggestions are welcome!).

Since the SQLite module already implements the DB API2 specification it was not really hard to change it. The plugin implementation was already using the specification and the patch is very simple (just use the sh.dbapi('sqlite') - everything else seems to be working well.

The 'sqlite' backend is always automatically registered and available.

New Plugin dblog

The new plugin is logging item changes to the database. It was tested using a SQLite database to log changes to. It will not do any pack mechanism which is the SQLite plugin doing. It's something like a log and stores all changes to the database.

The amount of data could be large when SmartHome.py is running long time. That's why it also support other database backends (e.g. MySQL or PostgreSQL).

The configuration of this plugin is simple:

• db - defines which database backend should be used (e.g. sqlite - which is registered as default)
• connect - the list of parameters to pass to database backend's connect() function (e.g. for SQLite something like connect = database:/path/to/log.db | check_same_thread:0) - the parameters depends on the database implementation

[dblog]
  class_name = DbLog
  class_path = plugins.dblog
  db = sqlite
  connect = database:/home/service/apps/smarthome/var/db/log.db | check_same_thread:0

This plugin is already using the new lib.db.Database class to do something with the database backend.

[ohinckel]

Trying to be more generic we also need to use generic/standard SQL queries. Some features are not available using standard SQL (e.g. CREATE TABLE IF EXISTS is not available in MySQL but in SQLite).

The dblog plugin tries to create tables / indexes and log errors but continue with the plugin. I think this behaviour is not very clean, but works at the moment. Maybe one of you have another good idea how to solve this (maybe trying to select data from a table to check if it exist).

Further I start testing the dblog plugin with another DB API: PyMySQL. It working well with it and using this I was able to log the data into a MySQL database.

[ohinckel]

Added a new class lib.db.Database which can be used to talk to different database backend implementations using DB API2. It have the following functions:

• __init__(): will get a identifier name, backend implementation module and connect parameter (see introduction)
• connect(): will connect to the database
• close(): will close the connection
• setup(): can be used to run multiple database setup queries (e.g. create tables / indexes)
• execute(): execute the given query with given parameters (use correct quoting style of backend module)
• fetchone(): execute given query and return one row
• fetchall(): execute given query and return all rows
• cursor(): create a cursor, which can be passed to the execute() and fetch*() functions
• lock(): create a lock
• release(): release the lock
• verify(): verify if the connection is still available and try to reconnect if required

It can be used to execute simple queries, query for data and return one or all of them, make use of cursors explicitely by using cursor() function and create/release locks - if required. It also checks the database backend module's parameter style in SQL queries and replaces the placeholders accordingly. The queries passed to the query function of the Database class should always use ? (qmark style)!

[mknx]

Hi Oliver,

I have to think about it...

so long

Marcus

[ohinckel]

I just commited a solution for:

The dblog plugin tries to create tables / indexes and log errors but continue with the plugin. I think this behaviour is not very clean, but works at the moment. Maybe one of you have another good idea how to solve this (maybe trying to select data from a table to check if it exist).

The setup() method will not accept not only a list of queries to setup a database, instead it will accept a dict where the keys are just incrementing version numbers and the values are the queries to execute. The lib.db.Database class will automatically create a (internal) table called <name>_version to track the current status of the database (where name is the name given in the Database constructor). This table is used to find out which queries need to be executed to reach the newest version of the database.

F.e. using the setup() method with the following dictionary

  db.setup({
    '1' : 'CREATE TABLE item(name varchar)',
    '2' : 'CREATE INDEX item_name on test (name)',
    '3' : 'CREATE TABLE log(name varchar, date datetime, data varchar)',

    // These lines were added in a later release
    '4' : 'ALTER TABLE item add id integer autoincrement',
    '5' : 'ALTER TABLE log add item_id integer',
    '6' : 'UPDATE log set item_id=id FROM item, log where item.name= log.name'
  });

... will automatically migrate the table structure and data to the new layout (don't know if all of the queries are working, it should only show what should be possible with this). The layout was changed to introduce a new ID column (in item and log table) and the data was migrated (based on the name in log table the ID in item table will be updated).

[ohinckel]

Created new PR smarthomeNG/smarthome#131 in new project and will close this issue now. Thanks for your great application!