sqlite3

[pztrn]

Patched blogtng for proper sqlite3 support

Credits to hatred - http://htrd.su/

For converting you need:

1. Convert database: sqlite blogtng.sqlite .dump | sqlite3 blogtng.sqlite3
2. Add co conf/local.php: $conf['plugin']['blogtng']['sqlite_version'] = 'SQLite3';

[andreashaerter]

Cool, thank you!. I'll have a look on it ASAP.

[Klap-in]

The sqlite plugin (http://www.dokuwiki.org/plugin:sqlite) has now a PDO branch that gives support to sqlite3 by using PHP PDO sqlite.

Maybe a good idea to join forces and use here also the sqlite plugin?
(when someone really needs PHP sqlite3 extension, that can also be added to sqlite plugin... But is has now quite good PDO support..)

[michitux]

By reloading pages (which belong to a blog/include such pages) quickly with purging caches I can reproduce error messages like "database is locked - SELECT val FROM opts WHERE opt = 'dbversion'" or "Warning: SQLite3::query(): Unable to prepare statement: 5, database is locked in /home/michitux/pub/projekte/dokuwiki-plugins/plugin-blogtng/helper/sqlite3.php on line 167". I've also got these errors by accident, not just by trying to get them (and I suspect that the corrupted CSS cache file which I've just had which couldn't be parsed by the browser anymore was because of the same problem). Any idea how this could be fixed? Maybe we need some additional locking for the database file or react differently on locking errors and retry the queries?

[michitux]

Any progress here? I think moving to the sqlite plugin is a good idea, unfortunately I don't think I'll find the time to do it. I've merged this pull request into the sqlite3 branch but because of the problems I've already mentioned I don't think this is ready to be merged in the master branch yet. If somebody of you wants commit access in order to continue the development of the sqlite3 branch or implement/fix other things in blogtng feel free to ask for it, the development of blogtng has been discontinued so we really need more people who work on it, fix bugs and implement new/missing features.

[dploeger]

I agree with @michitux on this one. Shouldn't we then close this pull request?

[dkriesel]

Any news on this? Especially, is it safe to apply the above patch in production use? THX so far for this great plugin :)

[dploeger]

No, sorry. I guess, this pull request won't make it into the code. I don't know, if the patch works. I would recommend, that you follow issue #74, where we embed PDO support or download the current PDO-branch if you need sqlite3-support right about now.

[lainme]

This patch does work and I am now using it. But as mentioned by @michitux , the error "database is locked" appears occasionally, especially when opening multiple pages in a short time

[maaaks]

Hmmm... Is the SQLite3 support in the official version now? For me, it doesn't work and says "Call to undefined function sqlite_open()" running on PHP 5.4. At the same time, patched version by @pztrn works for me.

[dploeger]

I'm sorry. I haven't had any time to develop the new PDO-branch and I think, won't have in the next couple of months. If @pztrn 's version works for you, you're good to go.

[dkriesel]

As BlogTNG seems not to be maintained any more, I guess the way of SQLite3 support presented by @pztrn in here is the only way to somehow use BlogTNG with recent versions. I use this branch on my blog, dkriesel.com, and it works fine for me except for one thing: The error messages @michitux mentioned (confirmed by @lainme). Those do get nasty beyond a certain page im pression niveau, as probability for simultaneous page impressions increases. So I suggest that instead for waiting until a new official BlogTNG release appears, it would be the easiest way to get a working BlogTNG by just patching this little bug. What do you think? Unfortunately, I'm not a php coder. Anyone able to do it?

--David

[hArpanet]

@dkriesel I'm currently doing some updating of BlogTNG and a number of pull requests have been merged recently. I haven't come across this locking issue myself, but I don't 'include' content and I have very few visitors. I'm currently consolidating the Master and Sqlite3 branches in order to remove the need for a separate Sqlite3 branch and then plan to consolidate the PDO branch too, leaving only a single Master branch. If/when this is achieved, I can look into using the SQLITE plugin within BlogTNG.

[lpefferkorn]

@dkriesel @hArpanet Thanks for the effort !

[dkriesel]

@hArpanet Nice to see some maintenance's going on on this IMHO most important plugin :-) If you want to see the locking issue, apply the patch of @pztrn and reload pages with blogTNGcontent frequently. Or reload http://www.dkriesel.com/start frequently, my page seems to be a nice example. The issue occurs with only a few hundred hits a day and gets worse if they are more. Last month I had a story on my site that got like 400k hits within a few days, and I was forced to unplug the blogpost from blogtng due to the bug. Do you have a time schedule for what you are doing? I really appreciate that you're on it and look forward to having a more stable SQlite3 branch.

[hArpanet]

NOTE: THE PATCH IS NO LONGER NECESSARY FOR NEW INSTALLS as it is now integrated in the GitHub Master and Sqlite3 branches.
@dkriesel I agree, if this plugin didn't exist it would have made my website work much harder, so I'm happy to pay back some of the effort already done. No update timescales I'm afraid. I'm fairly new to DokuWiki and newer to BlogTNG so still finding my way around. It'll also depend on what other work comes in, but I am actively updating things at the moment (apart from a few distractions). I got fed up submitting 3 pull requests, hence the consolidation :-)

[dkriesel]

@hArpanet As for the bug, I can't code PHP but if you need some testing, I can contribute by testing it for you, if this helps ...

[hArpanet]

@dkriesel No, at present the 3 branches remain separate. When I've finished the consolidation I would expect the Sqlite3 branch to disappear from GitHub leaving just Master and PDO. Until that happens, you still need to install the Sqlite3 version if that's what you want.

After Sqlite3 has been consolidated I will do the same with the PDO branch, the result of which will be that the PDO branch disappears from GitHub, leaving only the Master branch to handle all database types.

After that I will either look at the Sqlite plugin or more likely find out why there is a 'security warning' notice on the plugin page and see what I can do about that.

@dkriesel I'll definitely get back to you at some point to do some testing of the db errors, thanks.

[dkriesel]

@hArpanet Thanks a lot for this very helpful overview. I then stay with the patched SQlite3 branch I have and wait until further notice from you :-)

[Klap-in]

@hArpanet i have here fork of the blogtng, which is refactored so it uses now the sqlite helper plugin.

i will publish it on short term. Either is not yet tested.

[Klap-in]

@hArpanet

My work sofar is published at: https://github.com/Klap-in/plugin-blogtng/tree/usesqliteplugin

This code is NOT YET TESTED, due to missing localhost test environment. Next week i will setup that again and test this.

When you create a separated branch at the blogtng so i can do a pull request or if someone can give me contributor permissions on this repo, then i can create it my self and share this code with you. @michitux can you give me access?

Part of my code that needs to improve:

• the pdo adapter of the sqlite helper plugin hasn't a normal count function, so it fetch the whole result and stores it. e.g. in the admin.php the rowcount is used, so it will always fetch the whole result. The code needs to be improved here, when has bad performance. Also i'm not sure if relative rows can be requested e.g. res2row($res,5) This can be seen in the unit tests, but i not sure whether all tests succeed.
• the adapter of the helper_plugin_sqlite has not implemented a rollback function, that is still needed (inclusive unit tests)

[Klap-in]

I have tested a bit. So the general functions are working.
(i guess that in the particular situation that someone use the blog without PHP sqlite extension installed, someone can catch server errors...most such cases are already fixed)

I guess that the rollback via a query('ROLLBACK TRANSACTION') like done in sqlite2 will also work in sqlite3 pdo.

[michitux]

@Klap-in I've just added you to the team that should give you access to this repository. As I've already said I think using the sqlite plugin is the way to go so thank you very much for your work. How does the migration for existing installations work? If the migration is easy enough I would vote for getting rid of all the branches as soon as possible (i.e. as soon as the sqlite plugin code is working) and to concentrate on having one master branch which uses the sqlite plugin.

[hArpanet]

@Klap-in Cheers, I'll look at this more closely when I get closer to a single branch version of BlogTNG. That way it'll ensure that all the various patches/commits have been applied consistently and all the files are in sync, which isn't the case at the moment.

[Klap-in]

There is a update available for the BlogTNG that will use the sqlite plugin (please install separately) as sqlite backend. At the moment it is still in the branch usesqlitehelperplugin , but i think it is ready for integration in the master branch.

When you have the possibility to test, please try it and report whether it is successful!

Some info is available at the wiki page:
https://www.dokuwiki.org/plugin:blogtng#download_and_installation

[dkriesel]

I would like to install the plugin for testing purposes as described in https://www.dokuwiki.org/plugin:blogtng#download_and_installation. However, I have used the sqlite3-patched version so far. Just to be completely sure: Will the master branch accept my existing sqlite3 database and probably just work if I

• rename the existing database to [dokuwiki]/data/meta/blogtng.sqlite3
• deinstall the old blogTNG plugin
• install the sqlite plugin
• and install the blogTNG master branch?

(Of course appropriate backup measures will be taken. I however ask because I want to test it on the live website.)

[Klap-in]

The first 3 steps are correct, but the last one should be:

• install BlogTNG from usesqlitehelperplugin branch

This update is now available in the usesqlitehelperplugin branch, and is not yet merged into the master branch. When the feedback is positive i will merge it as soon as possible.

[dkriesel]

Ah, my bad, sorry, this was even the one I actually meant ;) I will try to get it installed today, however within the weekend at latest, and report back.

Last Question: What is the state of the locking issue? Do you think it will disappear?

[Klap-in]

Sofar we have not notices it happens in cases where the sqlite helper plugin is applied.
Further the write actions of the BlogTNG are reduced by only storing stuff for blog entries.

(I have added a changelog to the message at the wiki page too)

[Klap-in]

@dkriesel, is the fileextension of the database file used by sqlite3 part of the sqlite3 branch not already .sqlite3? So i guess renaming is not needed.. (or you have it modified again for yourself?)

[dkriesel]

It already was like this, yep. I installed the plugin like suggested above, and it seems to work: http://www.dkriesel.com/

[dkriesel]

After some days: No problems so far.

[Klap-in]

The branch is merged with master. Updated version is now available from the download url at the wiki page.