occ db:convert-type doesn't work with migration

[PVince81]

Steps

1. Setup ownCloud with sqlite
2. Use occ db:convert-type to convert to MySQL
3. Refresh the page

Expected

All tables migrated

Actual

Some tables were omitted: oc_external_config, etc. These tables are created not with MDB2 schemas but with migrations.

Versions

master.

We need to adjust the command to also run the migrations on the new database to initially create the tables. I'm not sure if this will work correctly because a migration isn't always only creating tables...

± % occ db:convert-type --all-apps --password owncloud42 mysql owncloud localhost owncloudmb4
Creating schema in new database
The following tables will not be converted:
oc_external_applicable
oc_external_config
oc_external_mounts
oc_external_options
oc_migrations
Continue with the conversion (y/n)? [n] y
oc_activity
    0 [>---------------------------]oc_activity_mq
    0 [>---------------------------]oc_addressbookchanges
    0 [>---------------------------]oc_addressbooks
 1/1 [============================] 100%oc_appconfig
 66/66 [============================] 100%oc_authtoken
 1/1 [============================] 100%oc_calendarchanges
    0 [>---------------------------]oc_calendarobjects
    0 [>---------------------------]oc_calendars
 1/1 [============================] 100%oc_calendarsubscriptions
    0 [>---------------------------]oc_cards
    0 [>---------------------------]oc_cards_properties
    0 [>---------------------------]oc_comments
    0 [>---------------------------]oc_comments_read_markers
    0 [>---------------------------]oc_credentials
    0 [>---------------------------]oc_dav_shares
    0 [>---------------------------]oc_federated_reshares
    0 [>---------------------------]oc_file_locks
 3/3 [============================] 100%oc_filecache
 14/14 [============================] 100%oc_files_trash
    0 [>---------------------------]oc_group_admin
    0 [>---------------------------]oc_group_user
 1/1 [============================] 100%oc_groups
 1/1 [============================] 100%oc_jobs
 14/14 [============================] 100%oc_mimetypes
 8/8 [============================] 100%oc_mounts
 1/1 [============================] 100%oc_preferences
 2/2 [============================] 100%oc_privatedata
    0 [>---------------------------]oc_properties
    0 [>---------------------------]oc_schedulingobjects
    0 [>---------------------------]oc_share
    0 [>---------------------------]oc_share_external
    0 [>---------------------------]oc_storages
 2/2 [============================] 100%oc_systemtag
    0 [>---------------------------]oc_systemtag_group
    0 [>---------------------------]oc_systemtag_object_mapping
    0 [>---------------------------]oc_trusted_servers
    0 [>---------------------------]oc_users
 1/1 [============================] 100%oc_vcategory
    0 [>---------------------------]oc_vcategory_to_object
    0 [>---------------------------]

@DeepDiver1975

[PVince81]

I'm afraid that there will be no solution for this.
Migrations cannot provide a list of tables to migrate

Maybe convert-type needs to work differently: instead of using the schemas, just migrate whatever data is there regardless... in which case the command itself becomes generic and could be written as a completely separate tool (not part of ownCloud).

I'm already hearing @DeepDiver1975 from far away saying "kill convert-type" 😉

[PVince81]

Wait a minute... The option --all-apps is supposed to also migrate extra tables.

But the code seems to just through away the extra tables instead of doing so.
When the option is not set, a hint tells that you can set it.

But then it doesn't do anything. WTF

[PVince81]

Ok, found the code.

Actually it finds the extra tables only for the warnings.
When you specify --all-apps it uses getAllApps() instead of getEnabledApps() and reads database.xml to find out how to create the tables...

So no luck there.

[DeepDiver1975]

I'm already hearing @DeepDiver1975 from far away saying "kill convert-type" 😉

😬

[DeepDiver1975]

We could clear the migrations table on the target schema and run all migration steps for all apps.

somthing like:

./occ migrations:migrate --all --restart

[PVince81]

You don't even need to clear the migration table, the convert script doesn't copy it over.

My worry about running the migration scripts is that some might want to operate on existing data. But the data hasn't been copied over yet. On the other hand, the data in the source tables is already converted to whatever format is required by the migrations once they all ran. So it might work.

I guess there can still be some possible migration scenarios where this wouldn't work, if for example a migration expects data to exist and fails if it doesn't (ex: some kind of orphaned share situation where the missing data wasn't copied yet to the target tables)

[PVince81]

We need to sort this out before final. So far I don't see any good solution apart from deprecating the tool... Or extracting it to be a general DB conversion tool not part of OC releases.

[PVince81]

Assigning to @DeepDiver1975 to make a decision

[PVince81]

We'll need a decision here, this feature will be completely broken in 10.0 and I don't see any possible solution because DB migrations work completely differently than what this feature here expects.

I'd suggest removing this feature. Maybe a separate generic tool could be provided to convert a SQLite DB to MySQL but nothing bundled or OC specific.

@pmaier1