Item9808: automatically create new tables and columns as and when the…

…y are encountered in data. Plus various fixes to protect against bad data (for example, manually hacked %META)

git-svn-id: http://svn.foswiki.org/trunk/DBIStoreContrib@17299 0b4bb1d4-4e5a-0410-9cc4-b2b747904278

data/System/DBIStoreContrib.txt

@@ -116,12 +116,32 @@ Basic tests for queries can be found in the DBIStoreTest topic.
 
 ---+++ Reloading the database
 
-Should it become necessary to reload the DBI database - for example, because
-changes to topics have been made outside of Foswiki - then you can click on the following link to re-load the database. *WARNING* on a large site this may take a very long time. You may have to run it from the command-line,
+It may become necessary to reload the DBI database - for example, because
+changes to topics have been made outside of Foswiki.
+
+---++++ Foswiki <1.20
+Click on the following link to clear down and re-load the entire database. *WARNING* on a large site this may take a very long time. You may prefer to run it from the command-line,
 using a command like this:
 <verbatim>
 ./view topic=System.DBIStoreTest skin=text dbistore_reset=1
 </verbatim>
+You can also update an individual topic by passing =dbistore_update= to a
+view of the topic. For example,
+<verbatim>
+./view topic=System.DBIStoreTest skin=text dbistore_update=1
+</verbatim>
+will update the DB for System.DBIStoreTest *only*.
+
+---++++ Foswiki 1.2 and later
+To be completed
+
+---+++ Unknown meta-data *EXPERT*
+
+Normally the module will only record recognised meta-data in the database,
+and so make it accessible for searching. "Recognised" meta-data is meta-data
+created by the core and by plugins that use the =Foswiki::Func::registerMETA=
+method to announce meta-data to the system. In exceptional circumstances you
+can override this behaviour by setting the ={Extensions}{DBIStoreContrib}{AutoloadUnknownMETA}= control in =configure=. This will cause *all* meta-data being recorded in the database, even if the plugin which is supposed to register it is missing, broken, or simply too old to do the right thing.
 
 ---+++ !MySQL Notes
 The !MySQL database user needs at least the following privileges:

data/System/DBIStoreTest.txt

@@ -1,50 +1,56 @@
+%META:TOPICINFO{author="SimianApe" comment="save topic" date="1393348518" format="1.1" reprev="6" version="6"}%
 <!--
 <verbatim>
    * Set SOP = topic="DBIStoreTest*" nonoise="on" format="" footer="$ntopics" separator=""
    * Set QOP = type="query" %SOP%
-   * Set TA = %IF{"
-   * Set TB = =0%SEARCH{
-   * Set TC = %QOP%}%" then="OK" else=" $percntRED$percnt *FAIL* $percntENDCOLOR$percnt "}%
+   * Set XD = }%" then="OK" else=" $percntRED$percnt *FAIL* $percntENDCOLOR$percnt "}%
+   * Set XA = %IF{"
+   * Set XB = =0%SEARCH{
+   * Set XC = %QOP%%XD%
 </verbatim>
 -->
 This topic is used for testing the DBIStoreContrib.
 Note that this topic, and the DBIStoreTestForm topic,
 have to be present in the database cache for the tests to work.
 
-To reset the cache, click on %SCRIPTURL{"view"}%/%WEB%/%WOPIC%?dbistore_init=1 (slow)
+To reset the cache, click on %SCRIPTURL{"view"}%/%WEB%/%TOPIC%?dbistore_init=1 (slow)
+
+To reload this topic, click on %SCRIPTURL{"view"}%/%WEB%/%TOPIC%?dbistore_update=1
+
+To reload the companion form topic, click on %SCRIPTURL{"view"}%/%WEB%/DBIStoreTestForm?dbistore_update=1
 
 <!-- TA number of hist TB search query TC -->
-   * Non-query: %IF{"2=0%SEARCH{"1" %SOP%}%" then="OK" else="$percntRED$percnt *FAIL* $percntENDCOLOR$percnt"}%
-   * False: %TA% 0 %TB% "0" %TC%
-   * True: %TA% 2 %TB% "1" %TC%
-   * Table.selector: %TA% 2 %TB% "form.name='DBIStoreTestForm' AND length('x')=1" %TC%
-   * Array: %TA% 1 %TB% "fields[name='string'].value='String'" %TC%
-   * Field: %TA% 2 %TB% "number" %TC%
-   * Boolean field: %TA% 1 %TB% "boolean" %TC%
-   * Field cmp: %TA% 1 %TB% "number=99" %TC%
-   * Simple regex: %TA% 2 %TB% "'AA'=~'A'" %TC%
-   * Simple LIKE: %TA% 1 %TB% "name~'DBIStoreT*orm'" %TC%
-   * Simple AND: %TA% 1 %TB% "number=99 AND string='String'" %TC%
-   * Simple OR: %TA% 1 %TB% "number=99 OR string='String'" %TC%
-   * Const-table cmp: %TA% 1 %TB% "666=fields[name='number'].value" %TC%
-   * Table-const cmp: %TA% 1 %TB% "fields[name='number'].value=666" %TC%
-   * Simple ref true: %TA% 2 %TB% "'DBIStoreTestForm'/number=666" %TC%
-   * Simple ref false: %TA% 0 %TB% "'DBIStoreTest'/number=666" %TC%
-   * Complex re: %TA% 1 %TB% "fields[name="string" AND value=~'^St.(i|n).*'].name!=''" %TC%
-   * Constant op field: %TA% 1 %TB% "fields[99=value].name" %TC%
-   * Complex conjunction: %TA% 2 %TB% "number=99 OR ( string='test' AND not boolean )" %TC%
-   * name=undefined: %TA% 0 %TB% "name=undefined" %TC%
-   * undefined=name: %TA% 0 %TB% "undefined=name" %TC%
-   * undefined=undefined: %TA% 2 %TB% "undefined=undefined" %TC%
-   * Length(num): %TA% 2 %TB% "length(99)" %TC%
-   * Length(str): %TA% 2 %TB% "length('99')>0" %TC%
-   * Deep where: %TA% 2 %TB% "fields[value=fields[name='number'].value].name" %TC%
-   * No such field: %TA% 0 %TB% "fields['x'].name='x'" %TC%
+   * Non-query: %XA 2 %XB "1" %SOP%%XD%
+   * False: %XA% 0 %XB% "0" %XC%
+   * True: %XA% 2 %XB% "1" %XC%
+   * Table.selector: %XA% 2 %XB% "form.name='DBIStoreTestForm' AND length('x')=1" %XC%
+   * Array: %XA% 1 %XB% "fields[name='string'].value='String'" %XC%
+   * Field: %XA% 2 %XB% "number" %XC%
+   * Boolean field: %XA% 1 %XB% "boolean" %XC%
+   * Field cmp: %XA% 1 %XB% "number=99" %XC%
+   * Simple regex: %XA% 2 %XB% "'AA'=~'A'" %XC%
+   * Simple LIKE: %XA% 1 %XB% "name~'DBIStoreT*orm'" %XC%
+   * Simple AND: %XA% 1 %XB% "number=99 AND string='String'" %XC%
+   * Simple OR: %XA% 1 %XB% "number=99 OR string='String'" %XC%
+   * Const-table cmp: %XA% 1 %XB% "666=fields[name='number'].value" %XC%
+   * Table-const cmp: %XA% 1 %XB% "fields[name='number'].value=666" %XC%
+   * Simple ref true: %XA% 2 %XB% "'DBIStoreTestForm'/number=666" %XC%
+   * Simple ref false: %XA% 0 %XB% "'DBIStoreTest'/number=666" %XC%
+   * Complex re: %XA% 1 %XB% "fields[name="string" AND value=~'^St.(i|n).*'].name!=''" %XC%
+   * Constant op field: %XA% 1 %XB% "fields[99=value].name" %XC%
+   * Complex conjunction: %XA% 2 %XB% "number=99 OR ( string='test' AND not boolean )" %XC%
+   * name=undefined: %XA% 0 %XB% "name=undefined" %XC%
+   * undefined=name: %XA% 0 %XB% "undefined=name" %XC%
+   * undefined=undefined: %XA% 2 %XB% "undefined=undefined" %XC%
+   * Length(num): %XA% 2 %XB% "length(99)" %XC%
+   * Length(str): %XA% 2 %XB% "length('99')>0" %XC%
+   * Deep where: %XA% 2 %XB% "fields[value=fields[name='number'].value].name" %XC%
+   * No such field: %XA% 0 %XB% "fields['x'].name='x'" %XC%
    * Numeric index   %#TA% 1 %#TB% "fields[0].name='number'" %#TC%
-   * Table Ref      %TA% 2 %TB% "'DBIStoreTest'/META:FORM.name='DBIStoreTestForm'" %TC%
-   * Complex ref     %TA% 2 %TB% "(fields[name='Other'].value)/META:FORM.name='DBIStoreTestForm'" %TC%
-   * Escapes  %TA% 0 %TB% "name =~ '\\\' OR name ~ '\\\' OR name = '\\\'" %TC%
-   * Escapes  %TA% 0 %TB% "name =~ '\\.x.y\\\'" %TC%
+   * Table Ref      %XA% 2 %XB% "'DBIStoreTest'/META:FORM.name='DBIStoreTestForm'" %XC%
+   * Complex ref     %XA% 2 %XB% "(fields[name='Other'].value)/META:FORM.name='DBIStoreTestForm'" %XC%
+   * Escapes  %XA% 0 %XB% "name =~ '\\\' OR name ~ '\\\' OR name = '\\\'" %XC%
+   * Escapes  %XA% 0 %XB% "name =~ '\\.x.y\\\'" %XC%
    * Table=Table     %#TA% 0 %#TB% "fields=attachments" %#TC%
 
 %META:FORM{name="DBIStoreTestForm"}%

lib/Foswiki/Contrib/DBIStoreContrib.pm

@@ -4,10 +4,11 @@ package Foswiki::Contrib::DBIStoreContrib;
 use strict;
 use Foswiki ();
 
-our $VERSION = '1.1';           # version of *this file*.
-our $RELEASE = '12 Dec 2013';
+our $VERSION = '1.1';           # plugin version is also locked to this
+our $RELEASE = '25 Feb 2014';
 
-use constant MONITOR => 1;
+# Very verbose debugging. Used by all modules in the suite.
+use constant MONITOR => 0;
 
 our $SHORTDESCRIPTION =
 'Use DBI to implement searching using an SQL database. Supports SQL queries over Form data.';
@@ -40,3 +41,25 @@ use constant {
     PSEUDO_BOOL => 14,
 };
 
+1;
+__DATA__
+
+Author: Crawford Currie http://c-dot.co.uk
+
+Module of Foswiki - The Free and Open Source Wiki, http://foswiki.org/, http://Foswiki.org/
+
+Copyright (C) 2013-2014 Foswiki Contributors. All Rights Reserved.
+Foswiki Contributors are listed in the AUTHORS file in the root
+of this distribution. NOTE: Please extend that file, not this notice.
+
+This program is free software; you can redistribute it and/or
+modify it under the terms of the GNU General Public License
+as published by the Free Software Foundation; either version 2
+of the License, or (at your option) any later version. For
+more details read LICENSE in the root of this distribution.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+As per the GPL, removal of this notice is prohibited.