/
DBIStoreContrib.txt
186 lines (142 loc) · 9.24 KB
/
DBIStoreContrib.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
---+!! !DBIStoreContrib
<!--
* Set SHORTDESCRIPTION = %$SHORTDESCRIPTION%
-->
%SHORTDESCRIPTION%
%TOC%
This extension caches wiki data and implements search operations (query and text search) using the Perl DBI interface to SQL databases. It has been tested with [[http://www.sqlite.org/][SQLite]], the popular lightweight implementation of an SQL relational database, [[http://www.mysql.com/][MySQL]], and [[http://www.postgresql.org/][Postgresql]].
---++ Database Schema
The database schema reflects 1:1 the schema of Foswiki topics. The following tables exist by default:
=topic= %BR%
Contains all wiki topics.
* =tid= - this column contains a numeric identifier that uniquely identifies the topic
* =web= - the web name
* =name= - the topic name
* =text= - the full text of the topic, without embedded meta-data
* =raw= - the full text of the topic including embedded meta-data
=metatypes= %BR%
Contains the names of all meta-tables.
* =name= - table name, e.g. =TOPICINFO=, =FORM= etc
At a minimum, =metatypes= will include the names of all the
tables listed below. The columns of these tables have identical semantics
(and syntax) to their corresponding =%META:= statements
=TOPICINFO= %BR%
* =TEXT author=
* =TEXT version=
* =TEXT date=
* =TEXT format=
* =TEXT reprev=
* =TEXT rev=
* =TEXT comment=
* =TEXT encoding=
=TOPICMOVED= %BR%
* =TEXT from=
* =TEXT to=
* =TEXT by=
* =TEXT date=
=TOPICPARENT= %BR%
* =TEXT name=
=FILEATTACHMENT= %BR%
* =TEXT name=
* =TEXT version=
* =TEXT path=
* =TEXT size=
* =TEXT date=
* =TEXT user=
* =TEXT comment=
* =TEXT attr=
=FORM= %BR%
* =TEXT name=
=FIELD= %BR%
* =TEXT name=
* =TEXT value=
* =TEXT title=
=PREFERENCE= %BR%
* =TEXT name=
* =TEXT value=
* =TEXT type=
Note that file attachments are *not* stored in the database at this time.
---++ Limitations
---+++ Regular Expressions
Regular expression searches are mapped to whatever regular expression
support exists in the database. Most SQL databases support sophisticated
regular expression matching; however there are features of the default
Perl syntax supported by Foswiki that cannot be mapped to the databases.
Regular expressions written using this extended syntax may fail.
---+++ Numeric comparison
Foswiki "knows" when two values in the database can be compared using numeric, as against lexical, comparison (the =, <, >, <= and >= operators). SQL doesn't have this kind of support, and has to be explicitly *told* whether a the values being compared are numeric or lexical.
If one of the things being compared is explicitly a number, then numeric comparison will be used by default.
If the query expression isn't explicit about the type to be used, you can use the =is_number()= operator to indicate when one side represents a number or =is_string()= when it is a string. You only need to use =number= / =string= on one side of an expression. For example,
* =%<nop>SEARCH{"info.version<'1.1'"}%= will always use lexical comparison
* =%<nop>SEARCH{"info.version<1.1"}%= will always use numeric comparison
* =%<nop>SEARCH{"info.version<is_number('1.1')"}%= will use numeric comparison
* =%<nop>SEARCH{"info.version<is_string(1.1)"}%= will use lexical comparison
---+++ Date comparison
Date conversion using the =d2n= operator is not supported.
---+++ Row indexes
Integer indexes are not supported. Use queries instead.
---++ Installation Instructions
%$INSTALL_INSTRUCTIONS%
* Go to =configure= and:
1 Set a DSN etc. for the contrib in the 'Extensions' section (the default is for sqlite3),
1 Select =Foswiki::Store::QueryAlgorithms::DBIStoreContrib= for the ={Store}{QueryAlgorithm}= *EXPERT* setting
1 *For Foswiki > 1.1* only
* add =Foswiki::Contrib::DBIStoreContrib::DBIStore= to the ={Store}{ImplementationClasses}= *EXPERT* setting in the 'Store' section
* select =Foswiki::Store::SearchAlgorithms::DBIStoreContrib= for the ={Store}{SearchAlgorithm}= setting
1 *For Foswiki 1.1 only*
* Select Foswiki::Store::QueryAlgorithms::DBIStoreContrib for the ={Store}{QueryAlgorithm}= configuration setting
* enable the integrated =DBIStorePlugin=
---+++ Testing
Basic tests for queries can be found in the DBIStoreTest topic.
---+++ Reloading the database
Should it become necessary to reload the DBI database - for example, because
changes to topics have been made outside of Foswiki - then you can click on the following link to re-load the database. *WARNING* on a large site this may take a very long time. You may have to run it from the command-line,
using a command like this:
<verbatim>
./view topic=System.DBIStoreTest skin=text dbistore_reset=1
</verbatim>
---+++ !MySQL Notes
The !MySQL database user needs at least the following privileges:
SELECT, INSERT, CREATE, and DROP.
---+++ Postgresql Notes
---+++ Microsoft SQL Server Notes
If you are using Windows authentication for users on SQL Server, then the simplest
thing to do is to use the ODBC driver with DBIStoreContrib and create a data
source for SQL Server in the ODBC Administrator which uses Windows authentication.
Then set an empty username and password for !DBIStoreContrib.
SQL Server does not come equipped with regular expression matching, which is
required for Foswiki, so you wil need to install a regular expression library.
The default personality module included in this module requires the user function =dbo.fn_RegExIsMatch= to stub the =.NET= !RegEx class. Instructions for building and installing this user function can be found at
http://www.codeproject.com/Articles/19502/A-T-SQL-Regular-Expression-Library-for-SQL-Server
---+++ SQLite Notes
SQLite requires the =pcre= module to be installed to support regular expression searches. The path to this module is set up in =configure=.
---++ How it works
---+++ Searches and Queries
Foswiki has two internal interfaces, "search algorithm" and "query algorithm", that are selected from =configure=. These two interfaces are implemented in a variety of ways in the Foswiki core, but the typical solution is to implement the query interface in terms of the search interface i.e. map _queries_ to regular expression _searches_. This is done by "hoisting" those parts of a query that can be mapped to regular expressions and using those hoisted expressions as a filter to reduce the set of matching topics. The "unhoistable" parts of the query are then applied to the remaining topics using "brute force" to give a final set of matching topics.
The DBIStoreContrib turns this process on its head by mapping _searches_ to SQL _queries_. Most modern RDBMS support regular expression searches, so a text search can be expressed as a regular expression match on a "raw text" field in the DB. Structured queries, on the other hand, are hoisted to extract SQL from the Foswiki query statement.
Note that you don't *have* to use the mapping search algorithm for text searches - if you have a caching full text search implementation (such as Foswiki:Extensions.SolrPlugin) you should be able to continue using that.
---+++ The Database
The database is used simply as a cache, to accelerate searches. The contrib is designed to work with the standard Foswiki RCS-based store, but can also work with *any* post 1.2.0 store implementation. It can also hook into the standard plugin handlers for a slightly reduced capability.
The database is interfaced to via the standard Perl DBI interface, so any RDBMS that has an adapter can be used for the cache. This includes most standard SQL RDBMS.
The schema used to represent Foswiki topics is (currently) a 1:1 mapping of the schema described in System.QuerySearch. The same schema could be used to store Foswiki topics in the actual store, and this is one of the longer term goals. Among other things, this would allow us to search topic histories.
---+++ The Code
Here's an overview of the important bits of the contrib:
* =lib/Foswiki/Contrib/DBIStoreContrib/HoistSQL.pm= - code that hoists SQL statements from Foswiki queries
* =lib/Foswiki/Contrib/DBIStoreContrib/DBIStore.pm= - a partial =Foswiki::Store= implementation that eveavesdrops on =Store::recordChange= events (Foswiki >1.1 only)
* =lib/Foswiki/Plugins/DBIStorePlugin.pm= - plugin handlers (Foswiki <=1.1 only)
* =lib/Foswiki/Store/QueryAlgorithms/DBIStoreContrib.pm= - the query algorithm
* =lib/Foswiki/Store/SearchAlgorithms/DBIStoreContrib.pm= - the search algorithm
The subversion repository (but not the official release) contains a fixed version of DBI::Shell from CPAN. This
can be useful when debugging SQL server, or for simply understanding the tables. Run it using:
=perl -I <path to DBIStoreContrib checkout>/lib -MDBI::Shell <dsn from configure> <username> <password>=
---++ Info
| Author(s): | Crawford Currie http://c-dot.co.uk |
| Copyright: | © 2010 C-Dot Consultants |
| License: | [[http://www.gnu.org/licenses/gpl.html][GPL (Gnu General Public License)]] |
| Release: | %$RELEASE% |
| Version: | %$VERSION% |
| Change History: | <!-- versions below in reverse order --> |
| Dependencies: | %$DEPENDENCIES% |
| Home page: | http://foswiki.org/bin/view/Extensions/DBIStoreContrib |
| Support: | http://foswiki.org/bin/view/Support/DBIStoreContrib |
<!-- Do _not_ attempt to edit this topic; it is auto-generated. -->