Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Import & Export Updates. Content Migration and Import Overhaul for 5.6 & 5.7. #1991
Content Import, Export & Migration
Concrete5 versions 6 and 7 employ a similar system for content population: the concrete5 Content Import Format (CIF). This is an XML description of a site’s content, including pages, areas, attributes, blocks, files and more. This XML language is primarily used when installing Concrete5’s sample content, but it can also be used to add items like attributes, single pages, block types and more during a package install.
Example: Creating a page, and specifying a content block inside that page:
This simple snippet of CIF XML specifies a page named Portfolio, found at /portfolio in the sitemap. Its page type is one with the handle portfolio, and its page template is one with the handle “full.” It will have its “exclude_subpages_from_nav” checkbox attribute set to true, and will add one content block to its Main area.
The Content Import Format was designed with the following goals in mind:
Portability: you should be able to take the XML generated by one Concrete5 site and easily import it into another site without worrying about the state of a database or a site environment.
Currently there is a single tool for Concrete5 CIF generation: the Sample Content Generator add-on. This add-on can be found in the 5.6 marketplace here:
and in a 5.7 version on GitHub:
This add-on is in the Concrete5 package. When a developer installs this package in their Concrete5 site, it adds a Dashboard page that gives an administrator the ability to export the entire site in the CIF format. This includes everything: the Dashboard, all attributes, all pages and stacks and all files.
The paragraph above that describes the main features of the Sample Content Generator also sums up its glaring weakness: there are no tools to filter any of the content exported by the Sample Content Generator. The entire Dashboard is exported, even if 99% of the time the developer building this CIF XML has no need for the Dashboard. All attribute keys are exported – not just the ones added after installation. This means that any time content is exported from a Concrete5 site, a significant amount of pruning and cleanup must take place before that content can be used with a package or as the starting point for a new Concrete5 install.
Import and Export Update
To achieve these goals, we will change the Sample Content Generator into the following:
Content Exporter Add-On
Content Importer Add-On
While CIF XML files will still be used to power Starting Point packages as they are today (and thereby still used as before to create Concrete5 sample content), we will create a new add-on that can import this XML as well, with much more flexibility and reliability. The usage of the Content Importer add-on will typically be as follows:
Goals for the Content Importer Add-On and Content Import Architectural Update
Must: Update the \Concrete\Core\Backup\ContentImporter and \Concrete\Core\Backup\ContentExporter classes to use PortableContentItem objects rather than directly work with XML.
Must: Add additional OutputFormatter objects that can take PortableContentItem objects and translate them into CIF XML.
Must: Add adapters for content formats. CIF XML is just one type of adapter. These handle the creation of PortableContentItem objects from different formats.
Must: The CIF XML format must still be used for Concrete5 sample content.
Must: The Content Exporter add-on must work with 5.6 sites.
Must: The Content Exporter add-on must work with 5.6 sites.
Must: The Content Importer add-on must work with 5.7 sites.
Must: The 5.7 Content Importer add-on must import content generated from both the 5.6 Content Exporter add-on and the 5.7 Content Exporter add-on.
Must: The Content Importer add-on must provide a way to import content over an existing site or into an Imports area of the Sitemap.
Must: Both Content Exporter add-ons should provide filter controls for choosing what types of content to output.
Should: A Wordpress Content Import Adapter should be available that can translate Wordpress’s content export format (XML?) into PortableContentItem objects.
Should: The Content Exporter add-on and the Content Importer add-on should be usable as solutions for building out an entire sub-section of a single site. For example, on a development server, creating a product catalog. Export the content using the content exporter add-on, and import it on the live site in one go, and voila – you’ve just created content.
Could: An ImportBatch may allow administrators to upload more than one content file within it,
Could: Provide a mechanism to map certain block types to other block types during import. For example, take a custom content block and map to the core Content block.
Could: Content import could be scriptable via command line or a deployment tool like rocketeer.
Could: The Content Importer add-on could provide a way to rollback a recently applied import.
Won’t: Import any Permissions.
Won’t: Import any Users.
Won’t: There will not be a 5.6 version of the Content Importer add-on.
Totally this sounds nice. I'm very interested in this project.
I already made an another importer add-on works with current ContentImporter class.
And here is a WordPress plugin that exports contents as concrete5 cif format.
Very cool! Yeah, I saw your add-on in the marketplace. Everything will ideally continue to work with the same output format from Wordpress, we're just going to make getting the content into your concrete5 site be a lot less error-prone, help you actually step through the process, give you options if something doesn't map 100% perfectly, etc...
What's the reason for not exporting/importing users? I thought the passwords are compatible.. We have quite a few sites with more than 100 users and one with more than 100k. At some point we must have a way to migrate them.
Settings the permissions again is okay for most of our projects. Takes quite some time, but cleaning them up should improve things too.
It just seems like a separate problem that can be solved any number of ways. Many times when migrating content you're not going to want to bring over permissions exactly as they are, and I'm not sure you want XML files hanging around with password salts in them.
Not sure I see why this should be a different problem, but I don't care whether it's an additional add-on or not, but there has to be a way to migrate users and groups. The point with password salts is definitely valid though, but no matter if it's a second add-on or not, I'll have those things in a file.. Adding a warning should imho be good enough - who ever exports / imports content should be able to handle things like that.
I also need a user migration tool. I can live with manual content migration, but with several thousand users, I absolutely cannot ask them to re-signup. Can I just match up the previous password salt and import users with a script? I don't want to go through the trouble of doing this if there is a gotcha.
We're currently building importers for Razor Commerce right now and it's our goal to have a systematic way to structure both import and export. Couple of ideas to pass along that have come from what we've built so far:
It would be great to see a system where you can export your data, then turn right around and import it or do the reverse. Full in/out capacity. That is what we are striving for with Razor Commerce. Of course to achieve it starts with the focus on testing. It might also require some options also, as in how to handle updating existing records on import.
Regarding User imports, we plan to build a Customer Importer shortly it's on our list of importers and it will use the Migrate system described here within Razor Commerce. Customers are essentially just C5 users so I think it will be possible to install Razor, use the Customer install and then uninstall. Somebody inclined to pull our Migrate section out and create a stand-alone User importer out of it would be welcome to that as well.
How would you change this xml to make the following content populate an existing composer form content block named Description? Simply adding the name attribute on doesn't do it.
Happy Independence Day for the U.S. citizens here -
While I applaud the efforts toward these high end goals, I guess I would recommend taking the project goals down a notch and focus on developing a map that enables the translation of the C5.6 exported file to the C5.7.3 import file. With a simple, yet complete Excel map for this scenario, we can then create a script to translate the import file into the expected format for input with hissy's addon_cif_importer. Does anyone have the variable specifications for each of these? I suppose there must be a way for me to figure this out, but if someone's already got that information, it would greatly speed things up for me.
Currently I am of the mindset that I NEED to upgrade my concrete5 to 7.4 as soon as possible. I HAVE developed a process to do so, but I need to create this map first. I am definitely open to suggestions.
Thanks to all for the consideration,