Item13378: bulk_copy.pl now handles encodings correctly. Deleted redu…

…ndant (and dangerous) change_store.pl script.

PlainFileStoreContrib/data/System/PlainFileStoreContrib.txt

@@ -1,4 +1,4 @@
-%META:TOPICINFO{author="ProjectContributor" date="1426439233" version="1"}%
+%META:TOPICINFO{author="ProjectContributor" date="1431855011" version="1"}%
 ---+!! PlainFileStoreContrib
 <!--
    * Set SHORTDESCRIPTION = %$SHORTDESCRIPTION%
@@ -42,27 +42,7 @@ Developers note: this contrib is tested using the standard store unit tests.
 
 ---+++ Foswiki < 1.2
 
-The =tools/change_store = script included in this module allows you to move
-between different store implementations. The script uses internal, unpublished
-store APIs.
-
-To support source and destination store implementations that both use files on
-disc pointed at by ={DataDir}= and ={PubDir}=, the the script supports
-redirecting the output of the conversion to a different directory.
-
-Run the script without parameters to get instructions:
-
-<verbatim>
-$ tools/change_store.pl -I lib
-</verbatim>
-
-%X% *it is dangerous* to run two different store implementations over the
-same data, so once you have completed the transfer you should *remove* all ,v files from the store. *All* of them.
-
----+++ Foswiki 1.2 and later
-
-The =tools/bulk_copy.pl= script should be used to move topics between two
-Foswiki instances running different store implementations. *Do not* share data areas between store implementations.
+The =tools/bulk_copy.pl= script (part of the Foswiki 1.2.0 release package, or available from [[https://raw.githubusercontent.com/foswiki/distro/master/core/tools/bulk_copy.pl][github]] or ) allows you to move between different store implementations. *Do not* share data areas between different store implementations.
 
 ---++ Developer Notes
 Testing is performed using the standard system tests. Because some
@@ -85,14 +65,8 @@ Many thanks to the following sponsors for supporting this work
    * [[http://c-dot.co.uk][C-Dot Consulting]]
    * [[http://www.taxonomy.org.au][TRIN]], The Taxonomy Research & Information Network
 
-|  Dependencies: | %$DEPENDENCIES% |
-|  Change History: | <!-- versions below in reverse order -->&nbsp; |
-|  1.3 (28 Feb 2013) | Foswiki:Tasks/Item12395: fix taint problems |
-|  1.2 (21 Jun 2012) | Added change_store.pl script; Corrected TOPICNFO handling |
-|  1.1 (30 Mar 2012) | Removed dependency on VC |
-|  1.0 (23 Mar 2012) | Initial version |
-
-
+---++ Dependencies
+%$DEPENDENCIES%
 
 %META:FORM{name="PackageForm"}%
 %META:FIELD{name="Author" title="Author" value="CrawfordCurrie http://c-dot.co.uk"}%

core/data/System/ReleaseNotes01x02.txt

@@ -1,4 +1,4 @@
-%META:TOPICINFO{author="ProjectContributor" date="1431220264" format="1.1"  version="1"}%
+%META:TOPICINFO{author="ProjectContributor" date="1431855011" format="1.1"  version="1"}%
 %META:TOPICPARENT{name="ReleaseHistory"}%
 ---+!! Foswiki Release 1.2.0 Beta 1
 
@@ -325,9 +325,9 @@ The Foswiki Store has been restructured into separate pluggable store implementa
    * RCSStoreContrib: This implements the RCSWrap and RCSLite "Revision Control System" based stores, compatible with prior versions of Foswiki and TWiki.
    * PlainFileStoreContrib: A new plain file store that saves topic and attachment as time-stamped copies instead of the "diff" based RCS store. This implementation can use more disk space but is expected to be _much higher performing_ than the RCS store. Existing topics can be migrated to the new store. *Store formats may not be mixed*. One store must be selected.
 
-The utility to migrate between store implementations is =tools/change_store.pl=. Be sure to backup your system before running the conversion. For information on how to run the tool, run:
+The utility to migrate between stores is =tools/bulk_copy.pl=. Be sure to backup everything before running the conversion. For information on how to run the tool, run:
 <verbatim>cd foswiki/tools
-perl -I ../lib change_store.pl --help</verbatim>
+perl -I ../lib bulk_copy.pl --help</verbatim>
 
 ---+++ New "Natural Editor", !NatEditPlugin replaces the old !WikiText editor
 The !NatEditPlugin, a component of the optional !NatSkin, is now the default

core/data/System/UpgradeGuide.txt

@@ -1,4 +1,4 @@
-%META:TOPICINFO{author="ProjectContributor" comment="reprev" date="1430795834" format="1.1" reprev="2" version="1"}%
+%META:TOPICINFO{author="ProjectContributor" comment="reprev" date="1431855011" format="1.1" reprev="2" version="1"}%
 %META:TOPICPARENT{name="AdminDocumentationCategory"}%
 %STARTINCLUDE%
 ---+ Foswiki Upgrade Guide
@@ -93,8 +93,7 @@ If these steps are not done, users will have to clear their cache, or "shift-rel
 
 *Choose your desired Store.* Foswiki ships with two native stores.
    * !RcsStoreContrib is compatible with topics created in prior versions of Foswiki.
-   * !PlainFileStoreContrib requires that topic history be converted to a new history format.  =perl -I lib tools/change_store.pl -help= for more information on conversion.
-   * %X% If you intend to change the topic character encoding, this *must* be done prior to conversion to the !PlainFileStore.
+   * !PlainFileStoreContrib requires that topic histories be converted to a new history format.  =perl -I lib tools/bulk_copy.pl --help= for more information on conversion.
 
 *If you are using authentication, prepare a test plan to verify that your authentication mechanism is working correctly.* Make sure you are able to test logins by a sufficient sample of users to cover all categories of users of your site. For example, users of various groups may need to be tested. In particular, ensure you test that non-admin users cannot access topics restricted to admins.
 
@@ -205,16 +204,17 @@ By default, empty =DENYTOPIC= rules will be ignored by Foswiki 1.2.  You must ch
 
 When =convertTopicSettings= saves the modified topics, they will be saved by user _UnknownUser_.
 
----+++ Convert to Plain File Store (optional)
+---+++ (Optional) Convert RCS store to !PlainFile
 
-The convertStore.pl utility will copy the =data= and =pub= directories into a new structure, with new revision data.  It also will _validate_ the contents of topics and verify that the revision history is consistent and complete.  *Subdirectories below topic directories within the =pub= directory are not copied!*  Because of this, it is not possible to completely convert the %SYSTEMWEB% directory.  We recommend using the new %SYSTEMWEB% shipped with Foswiki 1.2.  Normally the %SYSTEMWEB% does not contain any topic history, however if %SYSTEMWEB% topics have been edited, then conversion may be required. If you attempt to convert the %SYSTEMWEB%, you will need to copy critical =pub= subdirectories, found under =pub/%SYSTEMWEB%/JQueryPlugin/..=,  =pub/%SYSTEMWEB%/TinyMCEPlugin/..=, and other key components.
+By default Foswiki 1.2.0 uses a new store implementation, called 'PlainFile'.
+This store implementation offers significantly better performance when you
+have topics with long histories, at the cost of fractionally more disk space.
+If you have an existing RCS-based Wiki, you can continue using RCS, but new
+installations will use !PlainFile.
 
-   $ Validate the store in-place: =perl -I lib tools/change_store.pl -q -v !RcsLite !RcsLite=
-      * Any topics with inconsistent revisions are reported
-      * Any pub files without Attachment metadata in the topic are reported.
-   $ Convert the store to PlainFile, creating directories under =./newstore=: =perl -I lib tools/change_store.pl -s . -t newstore -q !RcsLite !PlainFile=
-      * Any "unattached" attachments will be listed, and can be piped to a command to copy separately
-      * The %SYSTEMWEB% will be copied but *will be incomplete / unusable*.
+Changing from RCS to !PlainFile is accomplished using the =tools/bulk_copy.pl=
+script to copy existing content from your current install to the new install.
+See =perl tools/bulk_copy.pl --help= for more information.
 
 ---+++ Apply preferences from old installation
 

core/lib/Foswiki.pm

@@ -149,6 +149,11 @@ sub _getLibDir {
 
 BEGIN {
 
+    # First thing we do; make sure we can read and print unicode
+    binmode( STDIN,  ":utf8" );
+    binmode( STDOUT, ":utf8" );
+    binmode( STDERR, ":utf8" );
+
     #Monitor::MARK("Start of BEGIN block in Foswiki.pm");
     if (DEBUG) {
         if ( not $Assert::soft ) {

core/lib/Foswiki/Contrib/core/MANIFEST

@@ -955,7 +955,7 @@ templates/view.xml.tmpl 0444
 templates/viewplain.tmpl 0444
 templates/viewprint.tmpl 0444
 templates/viewtopicactionbuttons.tmpl 0444
-tools/change_store.pl 0755 Store format conversion script
+tools/bulk_copy.pl 0755 Data transfer script
 tools/configure 0755
 tools/convertTopicSettings.pl 0755
 tools/dependencies 0755