Command line tool to convert current database to others, except sqlite

[bartv2]

I think it needs some more help/description what it does, but can't think of what to write.

@karlitschek @butonic @DeepDiver1975 @bantu @ringmaster

[ringmaster]

Any chance this could be made generic to move between any database engines?

[ringmaster]

Could this use $schemaFrom->getDbStructure($stream); instead to get the actual schema of the source database, and then use the migration methods to create/update the tables in the destination database? Might save those database comparisons later on.

[bartv2]

Tried this, the sqlite export misses information. For example converting to postgresql still needs to follow changes:

ALTER TABLE "oc_filecache" ALTER size TYPE BIGINT;
ALTER TABLE "oc_filecache" ALTER unencrypted_size TYPE BIGINT;
ALTER TABLE "oc_locks" ALTER created TYPE BIGINT;
ALTER TABLE "oc_locks" ALTER scope TYPE SMALLINT;
ALTER TABLE "oc_locks" ALTER depth TYPE SMALLINT;
ALTER TABLE "oc_share" ALTER share_type TYPE SMALLINT;
ALTER TABLE "oc_share" ALTER permissions TYPE SMALLINT;
ALTER TABLE "oc_share" ALTER stime TYPE BIGINT;
ALTER TABLE "oc_share" ALTER accepted TYPE SMALLINT;
ALTER TABLE "oc_share" ALTER mail_send TYPE SMALLINT;

And this is only for the core tables, the app tables also have the wrong type in places

[karlitschek]

Great. @bartv2 Do you think this could be extended to move between any databases?

[bantu]

Inject the config object via the constructor instead

[bantu]

Please make use of dependency injection. The quality of the code in master needs to improve.

[kprovost]

I've been trying to use this code to convert from SQLite to postgreSQL and I've run into a couple of issues.

The biggest one is that the owncloud db uses the 'user' field all over the place and that's a reserved keyword. As a result it needs to be escaped during insert calls, which this code doesn't do. I've hacked around it by adding the escapes in the DBAL layer, but that's probably not the cleanest solution.

I've also had trouble with the size of the mime field in oc_files_trash and the value field in oc_contacts_cards_properties.

I added the following hacks just before the copyTable() foreach:

$toDB->exec("SET CONSTRAINTS ALL DEFERRED;");
$toDB->exec("ALTER TABLE oc_files_trash ALTER mime TYPE character varying(128);");
$toDB->exec("ALTER TABLE oc_contacts_cards_properties ALTER value TYPE text;");
$toDB->exec("DROP INDEX cp_value_index");

Again, these are hacks, but at least they show where the problems lie.

[karlitschek]

@bartv2 So what is the status here?

[bartv2]

@kprovost thanks for testing this, i only checked sqlite->mysql, where this apparently wasn't a problem.
@karlitschek still working on this, just been distracted the last couple of weeks

[karlitschek]

Cool. Thanks.

[kprovost]

Just ran into another problem: the sequences need to be altered to start from max(id) + 1.

For those unfamiliar with postgres, sequences generate the next ID for index (or any other column you like) columns, much like the MySQL auto_increment. Apparently the conversion left me in a situation where the sequence generated IDs which already existed.

I (manually) fixed it by just doing select max(id) from ...; and then alter sequence ... restart with [max(id) + 1];

[bartv2]

It still only converts from sqlite, next is making it work to also convert from other database types

[bantu]

Is this something you want to try in this PR?

[bartv2]

Is this something you want to try in this PR?

No lets do that in another PR. That should also make this easier to test.

[bartv2]

Any more comments on this? Or can we merge this?

[DeepDiver1975]

No lets do that in another PR. That should also make this easier to test.

I'd prefer to do it all at once and right - partial solution would help that much from my point of view.
In addition there is no rush in get this merged now (besides the fact that the branch was kind of rotting the past weeks 😉 )

[scrutinizer-notifier]

The inspection completed: 8 new issues, 22 updated code elements

[ghost]

🚀 Test Passed. 🚀
Refer to this link for build results: https://ci.owncloud.org/job/pull-request-analyser/4735/

[karlitschek]

@Nowaker Can you help to review that?

[Nowaker]

@karlitschek I don't think so. I use version 6.

[bantu]

@karlitschek Consider merging without additional review.

[karlitschek]

@schiesbn @PVince81 @DeepDiver1975 Can anyone have a quick look?

[PVince81]

The code looks good, didn't test.

Would it make sense to automatically enable maintenance mode before the conversion starts and disable it afterwards ?

[DeepDiver1975]

I did review this PR last week already - without testing.

I'm fine with the code as well

[bantu]

Would it make sense to automatically enable maintenance mode before the conversion starts and disable it afterwards ?

This is being done here. https://github.com/owncloud/core/pull/6457/files#diff-edd78618a1329a6cad8cfffd51901e60R259

[PVince81]

@bantu ok got it. Seems I used the search on the wrong page.
👍

[bantu]

@karlitschek Let's have this then?

[karlitschek]

👍

[karlitschek]

done :-)

[bantu]

@karlitschek Merci

[cdamken]

@karlitschek

It seems this is completed? Has it been tested? I have a need for this script at the moment. If it is ready for release, are there instructions on how to use?

[karlitschek]

@cdamken It is now merged and should work. Please not that this is not a simple conversion script but requires complex changes in core. So this works only in ownCloud 7 and is not available in older versions.

[Nowaker]

Will there be owncloud 6 to 7 migration script, so I could then convert my SQLite db to PostgreSQL with your converter?

[holmboe]

I am also in need of this. I have a server on it's knees. :)

My current setup is an OC v6 with sqlite3 so an update to OC v7, when it is released, before migrating mysql is totally fine.

[Bugsbane]

"It is now merged and should work. Please not that this is not a simple conversion script but requires complex changes in core. So this works only in ownCloud 7 and is not available in older versions."

@karlitschek - So for someone trying to migrate a v7.0.1 database, how do we go about it? I'm moving my sqlite3 / v7.0.1 instance to a server that only supports Mysql currently, and am keen to not accidentally use one of those DB breaking proceedures that seemed popular previously.

[HeisSpiter]

@Bugsbane Basically, you've to use OwnCloud Console (occ). It's in root directory for your owncloud installation. If you installed it in /var/ww/owncloud, it's /var/www/owncloud/occ. So, basically, you need to:

/var/www/owncloud/occ db:convert-type

For usage, see:

/var/www/owncloud/occ help db:convert-type

[Bugsbane]

Great. Thanks HeisSpiter. That's exactly what I needed!