No way to import data in ownCloud Instance Migration

[RandolfCarter]

When enabling the app "ownCloud Instance Migration", there is an export facility in the "Admin" section for admin users. However, there is no matching import functionality to be found anywhere! See also this forum thread: http://forum.owncloud.org/viewtopic.php?f=3&t=4538.

[RandolfCarter]

Why is this considered an "enhancement"? It should be a bug in my opinion - the instance migration app is useless without import...

[MTRichards]

Hi - mostly this app provides a single way to get what you need to backup your ownCloud. Without the upload, yes, not right - completely agree. BUT, the data inside the archive can be extracted, downloaded and used to restore a system - though not being automatic is an issue, and we will fix it.

Since nothing is actually broken, just not that easy to use, made it an enhancement. And, it is in the backlog and will get addressed, being an enhancement is not a bad thing...

What we really need to do is remove the "suitable for import" list. That is a bug.
#63

[tomneedham]

Hi All,

As Matt said, the issue currently is more that the label in incorrect.

There are a couple of reasons why the import functionality is currently not available. Firstly, due to a bug upstream with MDB2 we are not able to re import the database. I have made several attempts to fix this myself, and contacted the MDB2 developers to no avail. However there has been talk of moving to the doctrine database abstraction layer which may fix this problem.

The second reason is more of a contraint on using PHP to do all this. It is likely that the size of your whole ownCloud install is larger than the memory given to PHP to run. Because of this it makes it a great pain to create a zip file containing your whole ownCloud install without coming across memory problems. I think the best solution here is to export the database into a file that is easy for the system administrator to re import into the relevant database system. For example generate a .sql file if ownCloud is setup with mySQL. We should then provide documentation on how to backup and migrate the data folder that ownCloud uses.

Cheers,
Tom

[tomneedham]

Following from my previous message...

This brings us to the import function, and trying to work out what is actually feasible. As I mentioned previously, for larger installs the size of the theoretical export zip will be relatively large and is an impractical method for exporting a whole instance. This is also a problem for import, it is highly impractical to have to upload a whole instance zip file and then let PHP extract it.

In my opinion we should improve the documentation on what is required to move an ownCloud instance between servers.

[RandolfCarter]

In my opinion we should improve the documentation on what is required to move an ownCloud instance between servers.

That sounds like a good plan. When making tons of data accessible via owncloud (e.g. multiple GB or even TB) exporting everything in one zip file isn't an option anymore anyway....

[legalstuff]

Now, we have 2 month later.. and what result?
Of corse.. TB in a zip-File is not suitable. But where is documanted which tables are the OwnCloud own?

Think about some WEB-Space with enough storage, but only one DB.. and parallel to owncloud they have Typo3, Wordpress and some other WEB-Apps.. who is able to find the right Table from OC?

I dont find any OC_table.

[RandolfCarter]

I dont find any OC_table.
Maybe you have chosen sqlite?

In any case: Naming an App "Instance Migration" and then only providing a convenient export functionality but no import (and not even a documentation on how to do the import) is actively misleading and confusing the users...

Like it is at the moment, I would recommend that the app be removed to avoid confusing administrators with it, and instead to add a guide to the documentation on how to do export/import manually (I don't think administrators will save that much time by using the export functionality it has - if it even works, especially when many/big files are involved).

[chadbernick]

I'm on the same page as Tom. Pull out the relevant config bits and provide documentation on moving the data.
Once this is done and an import routine is written, it might be better to qualify the "Instance Migration" as a shallow copy procedure ("shallow clone" maybe?)

[tomneedham]

Docs on migrating an install are available here: http://doc.owncloud.org/server/5.0/admin_manual/maintenance/migrating.html

I have removed the text from the list that suggested the export was suitable for import again. I will also change the name of the app to 'ownCloud Instance Export' instead of 'ownCloud Instance Migration'

[esaule]

The documentation that is pointed by tomneedham is pretty useless. In particular, the archive that is generated by the ownCloud Instance Export pretty much contains data/ but not config/. It also contains the database in some XML format. How are we suppose to import that XML database into a mysql database or a sqlite database?

[tomneedham]

It's pretty hard to reliably export each of the different db backends we support, therefore I figured for most admins it would just be easier for them to export using their own tools.

As for the config, I'll add this to the export that is generated.

[surak]

While it's understandable that it is hard to export reliably to all the different backends, the presence of this tool implies it is possible to migrate data to another install. In this case, this is completely misleading. Either the tool should let the admin know about it, or the tool should not be available at all.

[tomneedham]

@jancborchardt I noticed this app got renamed again to Admin Import / Export. For the reasons mentioned above this is misleading. Perhaps we should tailor this to more of a 'Admin Settings Export' app. We could just export the settings file and link to docs on how to move your install (exporting and moving a database). Or by this point is there any point in the app and we should just improve the docs? Lets face it if you can export a database you can also read the settings file from the file system. Is there any advantage of having this in an app format?

[jancborchardt]

Ok, then I'd say we should not have the app, remove it and improve the docs. @karlitschek @DeepDiver1975

[karlitschek]

@tomneedham Do you see a way for a quickfix? Only export the settings and add a link to a documentation page.
Otherwise we should remove it indeed from ownCloud 6.

[DeepDiver1975]

Otherwise we should remove it indeed from ownCloud 6.

We have some core regarding migation which need love as well - in case we touch it we shall take care of this as well owncloud/core#5388

[tomneedham]

@DeepDiver1975 is that critical for it to work in ownCloud 6?

[DeepDiver1975]

The OC_Migration code in core still used MDB2 stuff - which should use the doctrine stuff like any thing else.
Looks like this has been forgotten during the migration from mdb2 to doctrine

[tomneedham]

Yup. But is this essential before oc6? Are the methods not backwards compatible?

[DeepDiver1975]

In addition we still have one conceptual issue to solve regarding the user data import/export.
Creating one big zip for the user data can cause issues - 20GB in one file - doesn't really work :-(

[jancborchardt]

So is this already removed by the way?

[sysfu]

Thanks OwnCloud devs, I just got f***ed by this misleading "feature".

I made what I had every reason to believe was a complete "backup" of my OwnCloud 5.0.11 instance by going to the admin menu => Export this ownCloud instance => select "ownCloud instance (user data and database)" radio button => click Export.

Now I can't even import this useless "backup" without being a SysAdmin, if at all?

This is why people opt for data silos, they get tired of losing their data because of crap interfaces.

This is also not the first time I've been burned by trusting my data to OwnCloud either.

POd.

[DeepDiver1975]

@srf21c I understand your frustration - sorry for the mess.

@karlitschek @tomneedham I definitly vote for dropping the current implementation of the import/export tool.

• conceptually still broken (one zip containing all files)
• database schema export got finally migrated to Doctrine - but AFAIK it's still not working properly
• using sqlite as container seems wrong to me - this dependency will not be recognized until the export/import fails, which can be after YEARS of running ownCloud
• finally: we have been waiting for a solution for month and there is no progress - sorry -> drop it!

[karlitschek]

It's an important feature but it seems that we can't make it working :-(
@tomneedham What do you suggest?

[jancborchardt]

Wow – open for one year and no fix. I suggest we remove it because it apparently does not work and only add it back when it works properly. See #1542

[jancborchardt]

And yeah, I also think export-import is a really important feature, especially for an open source platform. But it needs to work, otherwise it’s just misleading.

[DeepDiver1975]

@jancborchardt there is user_migrate as well - which is broken for the same reason

[jancborchardt]

@DeepDiver1975 updated the pull request.

[tomneedham]

As I've mentioned multiple times now the admin migrate app is no longer possible due to the migration to doctrine from MDB2. MDB2 had functions (which didn't even work, and no response from dev's upstream) to export a database in xml format, which could be imported into different types of databases, allowing for migration across db types.

So I would recommend we remove this feature. Instead documentation should be provided. Unless someone see's a way of moving our data between database types (does doctrine let us do this easily?)

As for user_migrate, imo this is an import application for users as it allows users to retain their freedom from the service they are using (in the case of service providers). It has been fully functional until the migration to doctrine.

Several improvement could be made to improve the experience of the app (other than reviewing/fixing the move to doctrine db methods in the code):

• Add some detection for the sqlite dependancy
• Stream the files to be downloaded as the zip file can become very large (perhaps export user app data in zip and use existing methods for streaming the large user files)

[karlitschek]

@tomneedham Does it make sense to provide at least the export of the config as an admin? No everyone has access to the config.php file.

I agree for the user part. We could also say that we only export the DB content and not the files and advice the users to migrate the files manually? Perhaps this solves the problem with the huge zip files.

What do you think?

[tomneedham]

@tomneedham Does it make sense to provide at least the export of the config as an admin? No everyone has access to the config.php file.

This is true, I had missed this use case.

We could also say that we only export the DB content and not the files and advice the users to migrate the files manually?

Yes this would be an easier way of doing things, and saves replicating features.

[DeepDiver1975]

okay - nice discussion - where is the code to fix these issues?

Please don't get me wrong but we want to release OC6 - NOW! THX

[tomneedham]

#1546

[tomneedham]

Am waiting on @bartv2 to look at owncloud/core#6005 before I can change what is included in exports

[tomneedham]

Closing due to inactivity and age.