Adjusted the configs and several functionality, add git importer and exporter #12

danny0838 · 2013-10-28T04:55:02Z

I made a rather big change for this plugin, here's a simple list of features:

Configs change: add autoCommit, importMetaMsg, backupSuffix, gitPath, repoBase, dataBase, gitBranch; removed repoWorkDir
Changed the default path of git repo and git working dir. All git manipulation are now under an independent directory that will never bother the real dokuwiki data.
Fixed an issue that non-ASCII characters in the commit message are stripped.
Add git importer and exporter (command line interface). Supports history hiding and external commit.
Replaced the lib/Git.php with a more customized one helper/git.php.

I've done most tests on my private server and it worked fine. Though further investigation and experiments would never be too much. :)

Below is a simple documentation about the importer and the exporter:

The importer imports all page and media revision into the git repo.

Importing must be atomic. You should lock the wiki and the git repo from writing during the import process in case of data inconsistency.
Only changes later than the top commit (of the specified branch) in the git repo are imported (unless you specified --full-history). You can reset your repo top to a previous time and re-import those dokuwiki changes later than that.
Hidden pages, media, or meta (history entry) are supported. Move the item to the corresponding folder with a suffix (eg. attic/blah.123456789.txt.gz => attic.bak/blah.123456789.txt.gz) and they will be recognized during the importing process.
Meta files other than *.changes, *.indexed, *.meta can be imported for backup (if --no-meta is not set), but their timestamp will be reset in the repo and be changed on export.
Importing process is chronological, non-time-ordered history in *.changes will be resorted.
Plugins recording data to extra non-meta files are not imported. You have to backup them separately.

The exporter exports history records in the git repo into the DokuWiki data.

Exporting must be atomic. You should lock the wiki and the git repo from writing during the export process in case of data inconsistency.
All wiki pages, media, and meta files (or limited to *.changes, *.indexed, and *.meta if --no-meta is specified) are removed before exporting, making a backup is recommanded in case something go wrong.
After exporting, index (and meta data) rebuilding is required to get all things work right.
Exporting process follows git revision order, non-time-ordered history is available, but they'll be messed up if you re-import them to git.
External commiting (commits other than wiki import) and history rewriting are supported. You can import datain to git, rewrite git history, and them export it back to DokuWiki. For example, to rename a page, media, or a directory (namespace), simply git mv and then commit.
To hide the content of a history entry, add "cmd: hide data" to the last line below ~~dokuwiki~~ in the commit message. Commits with this command will have attics saved to the corresponding backup folder (attic.bak or media_attic.bak).
To hide an entire history entry, add "cmd: hide change" to the last line below ~~dokuwiki~~ in the commit message. Commits with this command will have entry in *.changes saved to the corresponding backup folder (meta.bak or media_meta.bak).
Do not hide an entry that is exactly the current revision, or you'll get some problem after next import. Instead, delete the page in wiki and then hide the old histories.
Only files in pages/, meta/, attics/, media/, media_meta/, and media_attic/ (may change by workDir) will be recognized and exported to DokuWiki. Other files are git-only.

…rdized data structure

…and then write to the history list

…removed at end.

…ulling.

…anch name)

1.periodic pull and edit commits now uses independent timestamped directories as the working dir, and locks the git on processing; 2.use repoBase and dataBase instead of repoWorkDir that improves the flexibility of directory structure; 3.import meta now uses current time as author date 4.exporter now prints the current manipulation

woolfg · 2013-10-28T16:26:51Z

Wow, thanks, great work. Are your changes compatible to the current version?

danny0838 · 2013-10-28T17:36:10Z

There may be problems due to changes of default configs.

To transfer the configs from the previous version (those came to my mind, may not be completed yet):

turn autoCommit on
set all dokuwiki data paths (those under data/) to the default path
if used default repoPath, move the git repo to the new default folder (./data.git)
to import pages and media to the same path as old default, set repoBase to blank and dataBase to 'data'

However, instead of 3 and 4, It's more recommended to use new dafult configs, discard the old repo, and import data to the new default repo data.git (importer.php --run is enough) if the wiki still have complete data.

…tedly in large operation

… directly. Also fixed a potential fatal error with very large content.

woolfg · 2013-11-26T09:38:22Z

At the moment we have to major pull requests #12 and #14 which adapt the git library. Did you see #14 and what is your opinion about it?

danny0838 · 2013-11-26T12:20:14Z

lib/Git.php was totally discarded by this work. Though other improvements could be merged in.

danny0838 · 2014-11-09T14:09:45Z

This is become messy and needs to much rework, so I closed it. The importing (to git) feature can now be better done via dokuwiki2git (original project and my fork with lots of improvements). The exporting feature is pending since it's not in such a hurry, though currently I think it would probably be done better via extending dokuwiki2git instead of gitbacked.

There are still other features that could be contained in gitbacked. Here are some I'm thinking about:

Get rid of the lib/Git.php and develop a framework on our own (it should be not too complicated according to my evaluation). Therefore we can largely optimize the codes.
Switch to a new technique that directly writes files to the git index, so that a working directory is needless, importing to bare repos can be supported, and the git repo is totally independent with the dokuwiki data directory so any potential interference could be avoided.
Accordingly, repoWorkDir config is canceled, and pages and media are always saved in the 'pages' and 'media' directory in the git repo no matter where the .git is or where dokuwiki configs of savedir, mediadir, etc are. This should be more flexible and portable for users.
This technique probably requires a framework redesign, i.e. the point 1.
There would be a data inconsistency if there is a data change not monitored by gitbacked, such as a revert manipulation via the revert plugin in the admin backend, or other admin plugins.
It seems that we can hardly solve this. Should we put a notice and maintain a incompatible list (with other plugins) for this?
Add customization of the commit author and email, and more placeholders support, such as %user%, %ip%, %userip% (returns ip if the user name is empty), %email%, %item% (fullname of a page or media), %itemns% (namespace), %itembase% (basename), %nl% (newline), some of which has been introduced in Improve message templates & Git environment support (easier pushing) #14.

How do you about them?

woolfg · 2014-12-29T09:15:25Z

sry for my late repsone: great ideas, contributions are always welcome

low priority I would say, but it makes sense
so you mean to call "git add" on every file/change?
if you know such plugins a list would be great. fell free to change the wiki or send a pull request
would be great!

danny0838 · 2014-12-29T13:39:49Z

No. Say we have a dokuwiki data directory at /home/www/dokuwiki/data and the git repo at /home/www/dokuwiki/repo with a /home/www/dokuwiki/repo/.git. Traditionally we have to copy /home/www/dokuwiki/data/* to /home/www/dokuwiki/repo/data/* and then git add and git commit, or we have to limit the .git/.. directory to contain the /home/www/dokuwiki/data.

We now can use a new technique that directly writes /home/www/dokuwiki/data/* files into the git object database, which will resides in the data/* as the git internal path.

This technique can be found at http://stackoverflow.com/questions/19616758/commit-to-git-with-a-different-path or some otherwhere. The main idea is a combination of git hash-object and git update-index for adding and git rm --cached for removing.

woolfg · 2014-12-29T14:43:35Z

is there any advantage over links?

danny0838 · 2014-12-29T15:29:12Z

What do you mean by 'links'?

If symbolic links: git do not support adding files beyond a symbolic link.
If hard links: hard links do not support folders, and creating folders and creating all mapping files probably don't have optimized performance. Besides, hard links cannot cross a device.
If mount: a sudo auth of the system is required.

Additionally, a bare repo cannot hold any link.

woolfg · 2014-12-29T22:47:11Z

sounds very convincing

danny0838 added 30 commits October 24, 2013 20:39

Add git importer

31d9fd5

Move git class used by importer.php to helper.php

aba2d81

Move helper.php to helper/git.php

d36bed2

Take care of --git-dir option

88c9065

Auto create .htaccess for the git repo

e16c568

support git branch config

3041e34

Change algorithm, merge page-ids and media-ids first

b2c50dd

move $temp_dir to $importer->temp_dir

ea5b9fd

Optimize the code and add some notes

44e6249

Add detection of external deletes

1a4f410

Do not make a log file for external edits

2bf24af

Combine the input streams of $historylist and related sub-functions

4ccf65d

Add a detection for missing data

f671506

Add histmeta helper and moved edit log data to commit message

309dea1

Switch id and data_type in the historylist

1a7c5d2

Make variable names more comprehensive

ffee728

Add packHistoryLine and unpackHistoryLine functions for better standa…

7922723

…rdized data structure

Support 'hide data'

2960c04

Support 'hide change'

2d490df

Change algorithm: write both meta and meta.bak to a temp file, sort, …

7d7942c

…and then write to the history list

Add --quiet option and adjust output messages

4475920

Simplify the code for linefeed seeking

d986de3

Rename cmd_git to git, which is simple enough

88f65de

Add $extra_param parameter for setGitRepo()

1cdacc5

Add getGitRepo() method

f400190

Changed the lock mechanism to flock() in case of abnormal termination

c61fbb0

Record logline again for external edits

8dc39ef

Change lock file path and fixed a problem that lock file doesn't get …

2a13883

…removed at end.

Removes all files from the staging area after switching to a new branch

0fcc22e

Fix a bug that old history line is not fully removed

b679f6f

danny0838 added 3 commits October 28, 2013 12:31

Replace lib/Git.php with helper/git.php. Use an empty directory for p…

67ab3a0

…ulling.

Fix an issue that non-ASCII characters are stripped (edit summary, br…

8df425a

…anch name)

This was referenced Oct 28, 2013

config param path of git bin #8

Closed

Add old revisions? #6

Open

danny0838 added 11 commits November 1, 2013 20:48

Fixed a bug that --no-meta doesn't take effect in exporter.php

56aea14

Wraps the help messages in importer.php and exporter.php

21f903d

Add --keep-meta option to exporter.php

368189f

Supports bare repo and defaults to create a bare repo

1edef50

Fixed an issue that commits sometimes contains working tree deletions

5316c88

Use a listfile to handle files import in case of large list

880ac9f

Fixed an issue that importer.php and exporter.php may shut up unexpec…

cd41506

…tedly in large operation

Removed unused --work-dir option from exporter.php

6d43bf0

Set up an empty working directory to prevent problems in bare repos

3a6a7fc

Prevent unnecessary I/O by sending loaded contents to the target file…

96097da

… directly. Also fixed a potential fatal error with very large content.

Code style compliance: removes unnecessary spaces

64a44c0

woolfg mentioned this pull request Nov 26, 2013

Improve message templates & Git environment support (easier pushing) #14

Closed

danny0838 mentioned this pull request Nov 3, 2014

Several simple fixes #18

Merged

danny0838 closed this Nov 9, 2014

danny0838 deleted the devel branch November 11, 2014 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjusted the configs and several functionality, add git importer and exporter #12

Adjusted the configs and several functionality, add git importer and exporter #12

danny0838 commented Oct 28, 2013

woolfg commented Oct 28, 2013

danny0838 commented Oct 28, 2013

woolfg commented Nov 26, 2013

danny0838 commented Nov 26, 2013

danny0838 commented Nov 9, 2014

woolfg commented Dec 29, 2014

danny0838 commented Dec 29, 2014

woolfg commented Dec 29, 2014

danny0838 commented Dec 29, 2014

woolfg commented Dec 29, 2014

Adjusted the configs and several functionality, add git importer and exporter #12

Adjusted the configs and several functionality, add git importer and exporter #12

Conversation

danny0838 commented Oct 28, 2013

Below is a simple documentation about the importer and the exporter:

woolfg commented Oct 28, 2013

danny0838 commented Oct 28, 2013

woolfg commented Nov 26, 2013

danny0838 commented Nov 26, 2013

danny0838 commented Nov 9, 2014

woolfg commented Dec 29, 2014

danny0838 commented Dec 29, 2014

woolfg commented Dec 29, 2014

danny0838 commented Dec 29, 2014

woolfg commented Dec 29, 2014