[WP] Data import and export

[cellear]

For years, my biggest gripe with Drupal has been how hard it is to get data in and out. When I started (mid D6) the best solution was the Node Import module. It worked ok, but the interface struck me as large, involved, and focused on covering edge cases. Two years later, when I moved to D7, the maintainers of Node Import didn't create a D7 version, and instead guided us users to Feeds. Feeds is more flexible than Node Import, but even less friendly -- you can't even read the headers from a CSV file directly, you have to type in the header names. Case matters! Only a mother could love that interface.

D8, of course, is making it even harder. Migrate. You have to write an object-oriented module. Easy to do, for Donald Knuth.

Meanwhile, install Wordpress and you get this in your main menu:

Can't we do something to help non-gurus get their data into (and out of) Backdrop? It's incredibly important. More than the code, more than the design, more than anything, the data is what people care about. Every single Drupal project I've been involved with -- without exception! -- has struggled with this. We should do better.

I took at look at what would be involved in porting Feeds to Backdrop myself, but it seems daunting. It's big, and depends on Ctools, which is also big, and probably already at least partially included in Backdrop. Eventually I could give it another shot, but rather than struggle and curse alone, I wanted to see what other people thought. Especially since I'm concerned that low-level architecture decisions being made now (or possibly already made) might easily make this easier or harder.

My strawman blue-sky suggestion: A facility that would spit out all of the nodes in a Backdrop system (and obviously other entity types, but nodes would be a good start) as, probably, JSON files. Probably as big aggregate files, though I'd be interested in one file per node, which a beginner could construct by hand. The important thing is it should be able to import whatever it exports. If you have that, then an end user can re-create those export files in whatever way is comfortable to them.

I realize there are lots of edge-cases; custom or contrib modules that store data in all sorts of wacky places. But following our 80/20 rule, if we support data import and export for sites built with non-extended Backdrop, we can worry about other cases later -- but we would already start out miles ahead of any version of Drupal in a feature that is extremely important to our core audience.

Thoughts?

[2020 edit - adding some more screenshots from recent WP v5.4]

[docwilmot]

+1
Feeds, migrate, excellent if you're building healthcare.gov, but overkill for mysecondsite.small.
Node export/import was good enough.

[quicksketch]

But following our 80/20 rule, if we support data import and export for sites built with non-extended Backdrop, we can worry about other cases later -- but we would already start out miles ahead of any version of Drupal in a feature that is extremely important to our core audience.

Considering the number of attempts we've had at this problem in the past, I'm not confident that we could build the right solution directly into core on our next attempt. Considering how separated this functionality is from everything else in core, I think we should try and build this in contrib first before we attempt to put it into core.

I took at look at what would be involved in porting Feeds to Backdrop myself, but it seems daunting. It's big, and depends on Ctools, which is also big, and probably already at least partially included in Backdrop.

Almost all of CTools is indeed included in Backdrop core, with the exception of the CTools plugin architecture (which is probably what Feeds utilized). De-pluginifying modules isn't impossible but it is indeed a messy operation (we de-pluginified both Views and Panels for Backdrop core).

[kreynen]

@quicksketch Is there an issue where the reasons for excluding the CTools plugin architecture was discussed?

UPDATE: Found #41

Node export/import was good enough.

It wasn't good enough for me. Feeds was a huge improvement over Feed API. While it can still be difficult to work with, I don't build many sites without it and use have used it for both importing initial data as well as ongoing imports.

IMHO the biggest issues related to Feeds is really same as any other large, feature rich project in Drupal where the original architects have moved on. Getting even minor improvements rolled into a release of Feeds has become impossible.

I was one of > 20 developers who contributed to getting a working version of entity support committed in https://www.drupal.org/node/1033202#comment-8359807. More than 3 years after than the issue started and > 1 year after it was "fixed", it is still only available in the dev snapshot. I really hope that moving contrib projects to GitHub helps to reduce the number of popular module projects that get stuck in perpetual alpha on Drupal.org. If maintainers stop responding to PRs and a fork becomes more popular, Backdrop should have a well defined process to change the repository considered the primary version of the project.

Community Media groups require importing for both date based schedules from different video playback servers and video on demand from providers like YouTube and Archive.org, so an import architecture that is more than Node Import this is something I am very interested in helping with.

While I agree that Feeds is overkill for basic imports, I think the MVP for Backdrop imports should include features beyond what the D6 Node Import offers like:

• import internal/uploaded or external sources
• import multiple formats (CSV, RSS, XML, JSON, etc)
• import an exportable format (ie. https://www.drupal.org/project/node_export)
• UI to map fields from source to target based on selected entity type
• hooks to extend the fields defined in a source (ie. http://drupal.org/project/feeds_mediarss)
• hooks to manipulate data from source on import (ie. https://www.drupal.org/project/feeds_media_internet_files)

A solid, basic import started in contrib that eventually gets added to core and the option of having a number of additional contrib modules for more complicated use cases aren't mutually exclusive, but to have something in core that can be extended will require more thought and planning than simply attacking the need for a WordPress-like import/export. I would hate to see effort go into an import that couldn't be extended to eventually be a feature match for Feeds. The team at Development Seed who started the Feeds project had the ambitious goal replace the Aggregator module in core with something developers could extend and yet http://cgit.drupalcode.org/drupal/tree/core/modules/aggregator.

Has anyone even seen a module that extends or alters Aggregator? I was happy to see #199, but obviously there is a need for imports.

[biolithic]

If anyone is interested in this problem, I ported the Feeds and Job Scheduler modules to Backdrop except for the CTools code, which someone will probably have to explain how it works in Backdrop. Or, if anyone is familiar with the where CTools stuff is Backdrop, you are welcome to fork and go from there. Thanks!

I actually have time tonight, so I think I will start porting:
https://www.drupal.org/project/migrate
https://www.drupal.org/project/feed_import

Relevant links:
https://github.com/biolithic/backdrop_feeds
#1073

[sentaidigital]

While I agree that Feeds is overkill for basic imports, I think the MVP for Backdrop imports should include features beyond what the D6 Node Import offers like:

• import internal/uploaded or external sources
• import multiple formats (CSV, RSS, XML, JSON, etc)
• import an exportable format (ie. https://www.drupal.org/project/node_export)
• UI to map fields from source to target based on selected entity type
• hooks to extend the fields defined in a source (ie. http://drupal.org/project/feeds_mediarss)
• hooks to manipulate data from source on import (ie. https://www.drupal.org/project/feeds_media_internet_files)

I have exactly this problem on a site I'm building for a local LUG. I would like importing the data exported in CSV to be as easy as uploading the CSV and mapping it to an existing content type.

To that end, I've created a module to do so. https://github.com/backdrop-contrib/simple_import

I've only started it, it's barely alpha-grade code. In particular:

• It only accepts CSV files. The other import methods are not written yet.
• It only accepts uploaded files, even though there are fields for remote and server-local files.
• It can't handle dates, files, or any kind of references (node, term, or user)
• All nodes are owned by the user uploading the CSV file.
• No checks are made for required fields.
• The confirmation page has no text. (really, barely alpha!)
• It only imports nodes, the Users tab isn't working.

[docwilmot]

Incidentally I ported Feeds somewhere I think.

[sentaidigital]

Yes, I've tried it. This is simpler.

[Graham-72]

We definitely need this. I imported Feed Import in order to find some way of importing a CSV file but was only partially successful. We need something that is as good as in import into PhpMyAdmin. Bonne chance!

[JAyenGreen]

Unless I'm missing something (which I very well could be!), D6 node export will not create a csv. I used the drush command and specified --format=csv and ended up with an object dump.

I don't see a source format in backdrop feeds for what D6 node export gave me. And were one of them to match, I also don't see the bulk operation link/tab to get it to import multiple nodes.

Finally, I'm confused as to how it determines the source fields to offer for mapping before it's seen the csv file.

[klonos]

Such a feature in core would be a great addition to #2461 as in allowing users to export a backup of their data before deleting it. It could work as a bulk operation as well for general use in the content, comments, files and users listings.

[klonos]

This came up in Zulip today. As I've mentioned there:

There's also https://www.drupal.org/project/views_data_export ...but not able to find a D7 module that imports that data 😞 ...only D8: https://www.drupal.org/project/csv_importer and https://www.drupal.org/project/contentimport

[domaingood]

I think don't need to do that Just port Druplal 7 wordpress migrate module to backdrop port
https://www.drupal.org/project/wordpress_migrate

[klonos]

Thanks @domaingood for chiming in, but this issue is not about allowing content from WP to be migrated over to a Backdrop site. This is about allowing people to export their Backdrop data (into a csv, or other format), for example if they want to rebuild their site with another CMS.