[RFC] Mod sample data

[Bakual]

Issue

Currently we handle our sample data with plain SQL queries that are applied.
Managing those queries became quite complex over time. Due to the way they are set up (they replace the tables, not just add to it), it also became complex to add new extensions to core since you need to add them in the sample data as well.
Also the sampledata currently can't handle extensions which we decoupled from core (eg Weblinks). We had to remove the sampledata associated to it.

Proposed Solution

This PR adds a new module and some plugins which lets you install the sampledata after the installation is done. It will show up in the administrator control panel.

The sampledata itself is handled by a plugin. Which means in theory they could also be provided by 3rd parties.

When the user activates one of the links in the module, it will fire some AJAX request which will install the chosen sampledata set. It does this in multiple steps and shows the progress while doing so.

Background

• The plugin will install the data using the respective models/tables from the extension.
• It will skip the installation step if the respective extension isn't installed and enabled.
• All relevant strings are coming from language strings, making it simple for translators to provide translated sampledata should they wish to do so. The alias is created automatically from the title.
• I had to do a change in how the menuitem model includes its helper. It used a require_once which failed when the model was called outside of com_menus. I changed the call to JLoader so the class gets autoloaded if needed. In the plugin is also register the same class with the correct path, and thus JLoader will find it. That's the only needed change outside the sampledata extensions.

Testing

You can test this by applying this PR and then discover and install the module and plugins.
You can also download my branch and install the CMS fresh, then the module and plugins should already be installed and visible.
Make sure you try on a system with no sampledata already installed, otherwise it may very well clash with the existing data.

Known Limitations

• Currently, only the "Testing Sampledata" works. The other sets don't install anything at the moment.
• Installing the sampledata twice doesn't work since the used alias already exists.
• The sampledata itself is quite outdated - especially if you look at the templates stuff 😆
• The CMS installation still shows the sampledata step. This can obviously be removed if this gets accepted.

Request For Comments

I would be very happy if someone could look over this implementation. I'm quite sure there are ways to improve it.

[Fedik]

👍 very very good idea 😉

[infograf768]

Tested.
A bit confused when installing branch as I did not know what to choose. Chose no sample data as if I was creating a basic multilang site.
Used the module link to install Testing sample data. All went well. Alias for Home was changed to home-1.
And there I did what I should not have done...:
I clicked again on the module link and it is still running today 😃

Maybe should be considered :

1. Prevent installing another sample data if one has been installed (or the site has already some data. Imagine the mess if one kills his site this way...)
2. Add somewhere a Delete all data except the basic usual home page. This would let install a new data if desired.
3. Make all this fully safe to avoid messing with the site: Double Warnings, etc.

[Bakual]

I clicked again on the module link and it is still running today 😃

Yeah, I noticed that already soemtime ago. I need to fix that JavaScript there so it properly terminates and doesn't try to load the next step 😄

Alias for Home was changed to home-1.

I was thinking about best approach with the home. Currently I create a new "main menu" and set the the existing "home" to a regular menuitem and create a new home from sampledata. This is something that could be improved I guess.

Prevent installing another sample data if one has been installed (or the site has already some data. Imagine the mess if one kills his site this way...)

I had that thought as well, but haven't figured out a sane way to do it. Also I could see that an extension could potentially provide its own sampledata set which can be installed with this.
I was also thinking about assigning all sampledata content to the currently active language, which then actually would allow to install the sampledata twice if you change the language. Just some idea for future 😄

Add somewhere a Delete all data except the basic usual home page. This would let install a new data if desired.

Here it gets very tricky. Currently I store the inserted IDs in the session so I can use it later when I need to link eg an article to a menu item. After you close the browser, Joomla doesn't know anymore which content comes from the sampledata. Thus uninstalling it is not possible easily.

Make all this fully safe to avoid messing with the site: Double Warnings, etc.

Any ideas on implementation? 👍

[infograf768]

Any ideas on implementation?

Sorry, none. I see how useful it can be for a 3rd party extension once the extension is installed, but I see also all the possible failures.
Making it easy for custom full Joomla distribs to provide sample datas through an ini file looks promessing "at Joomla install time", but when already in administrator, it looks very dangerous.

[Bakual]

it looks very dangerous.

It isn't really dangerous as it doesn't delete anything. It just adds data and modules. The only thing that is changed is the default "home" menuitem. Which is simple to switch back and we could also remove that from the sampledata if we want.
All modules can be disabled again easy.
All content is also added to own categories, so it's simple to filter for them and get rid again.

[Fedik]

couple notes before I forget 😄

use SampledataApplyStepXXX could have unexpected effect when there enabled a couple plugins that has the same method or I wrong?
Beacuse this, maybe better use <PluginName>ApplyStep($step), so it wil be adressed exactly to the <PluginName> and plugin internaly call which step is required ... means each plugin has "one enter point". (of course it not solwe the problem if another plugin has same method, but more safe, from my point of view)

And maybe we do not need $data->steps = 7; at all, we can just look on the request result, something like inprogress/done/fail

Currently I store the inserted IDs in the session so I can use it later when I need to link eg an article to a menu item

maybe can save this ids as parameters in related plugin, somehowe ... then it solwe reinstall problem, if user click install again we show warning, something: All previous data will be removed! you sure?

will try test this evening,
and hope I will find a time to help with JavaScript 😄

[brianteeman]

I like the idea - the only problem is when someone installs one set of sample data on top of another set or even worse on top of their own content.

If you want to do it post-install like this then I would suggest the only way to do it properly is to remove all existing content before installing the sample content - obviously with plenty of warnings

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/7680.}

[Bakual]

use SampledataApplyStepXXX could have unexpected effect when there enabled a couple plugins that has the same method or I wrong?

I made this intential this way. So multiple plugins could run on the same step if wanted. Again thinking about 3rd party plugins which "hook" into eg the testing sampledata.
I haven't tested it specifially yet but in theory it should be possible to have a weblinks plugin which adds its own sampledata on the first step of the testing sampledata.
In the current plugins, I have a check included which returns empty if the request is not meant for this plugin.
I think it's also more in line with how plugins should work. The caller should not care how many plugins actually do something with the request.

Beacuse this, maybe better use ApplyStep($step), so it wil be adressed exactly to the and plugin internaly call which step is required ... means each plugin has "one enter point". (of course it not solwe the problem if another plugin has same method, but more safe, from my point of view)

The requests go through com_ajax. So using ApplyStep($step) will not work anyway. I would have to request the step from JInput and dispatch afterwards. In the end it would just add more duplicated code to the plugin I think.

And maybe we do not need $data->steps = 7; at all, we can just look on the request result, something like inprogress/done/fail

I need that mainly for the progress bar, so I know how big steps to draw. The other things could indeed be done without having that step property.

maybe can save this ids as parameters in related plugin, somehowe ... then it solwe reinstall problem, if user click install again we show warning, something: All previous data will be removed! you sure?

That would be a (quite hacky) idea. You would have to make sure they don't get overwritten when one saves the plugin. And it would need code again to delete the items using the models/tables. I wouldn't want to use direct queries there so we don't have to deal with nested sets, asset tracking and the like.
I'm not sure if it's worth the effort. Currently there is no way to delete the sampledata either 😄

and hope I will find a time to help with JavaScript 😄

That would be awesome. It's definitively not my strongest area. I was glad it actually worked 😆

[Bakual]

I like the idea - the only problem is when someone installs one set of sample data on top of another set or even worse on top of their own content.

Usually, installing two sets will fail due to already existing alias. I can solve that by automatically generate a new alias till it sticks, but I haven't seen the need yet.
But then, I don't see it as a big issue. Worst case would be someone presses the button on an existing site. This will generate and publish about six modules which will be visible on frontpage instantely. And it will change the home menuitem.
As said, the modules are easy to disable again. The home menu is the one which probably has most impact and we could change the sampledata so it doesn't change the home. If this is seen as an issue. Would make the code actually simpler 😄

If you want to do it post-install like this then I would suggest the only way to do it properly is to remove all existing content before installing the sample content - obviously with plenty of warnings

I would never remove existing content. Even after 15 confirmation boxes I'm sure some users would accidentally delete the content. YCFS and I don't want to confirm that 😄

[Fedik]

So multiple plugins could run on the same step if wanted. Again thinking about 3rd party plugins which "hook" into eg the testing sampledata.

Sorry, then seems I wrong understand this part of the idea.

I thought that each extension (component) can have separated plugin, that can be installed separately, and each this plugin can provide own data collection (sorry for my English 😄 )

and <PluginName>ApplyStep still allow to "hook", even with biger precision 😄 ,
example we have next plugins:

• BlogData add:
  • Menus
  • Content
  • Categories
• BroshurData add:
  • Menus
  • Content
• TestinData add:
  • Menu
  • Content data
  • Category
  • .... all other core component
• SomeExtension2Data
  • Add data only for this extension
• ExtensionToExtendBlogData
  • Add data only for BlogData

So you can install BroshurData and then add SomeExtension2Data.
Or
If you run BlogData and ExtensionToExtendBlogData will "hook" in to BlogDataApplyStep() and add adittional data for the Blog

That would be a (quite hacky) idea. You would have to make sure they don't get overwritten when one saves the plugin.

true, but each plugin can have hidden fields that keep the list of ids:
field step1ids: 1,2,3....n
field step2ids: 1,2,3.....n
....

so if you need get access to these values you just do $this->params->get('step2ids'),
also this will help esianly to check which Data alredy installed and show the proper message if user want to insatll some incompatible SampleData ... in theory 😄

yes, it still hacky ... but also I think that it not so bad 😄

[Bakual]

Sorry, then seems I wrong understand this part of the idea.
I thought that each extension (component) can have separated plugin, that can be installed separately, and each this plugin can provide own data collection (sorry for my English 😄 )

You didn't understand wrong. It's both variants which should work (in theory).
I think renaming the methods is a good idea and makes it a bit simpler as we can get rid of those lines in each method: https://github.com/Bakual/joomla-cms/blob/ModSampleData/plugins/sampledata/testing/testing.php#L83-L86
You're right that it still allows the same flexibility.

true, but each plugin can have hidden fields that keep the list of ids:
field step1ids : 1,2,3....n
field step2ids : 1,2,3.....n
....
so if you need get access to these values you just do $this->params->get('step2ids') ,
also this will help esianly to check which Data alredy installed and show the proper message if user want to insatll some incompatible SampleData ... in theory 😄

I think I like that idea.

[wilsonge]

I presume these plugins are going to be core-supported (we can package them up into the full install packs) - but we once people are done with them then they need to be uninstallable and not reappearing on updates?

[Bakual]

once people are done with them then they need to be uninstallable and not reappearing on updates?

Good point.
If we handle it like weblinks, but ship it with new installs then we need to figure out how we want to handle the installation SQL that installs/enables the module and plugins. I guess it gets interesting if the extensions are in the SQL but those using the Git repo to install the CMS are missing the files.

We could of course make them installable as a package without it being included in core, but that kind of is counterproductive for the sampledata. Doesn't make sense I guess.

Not sure what the best approach is here and how it can be handled.

[arrowthemes]

I'm hoping this module will make it in the final 3.5 release. This will really (like really) come in handy for third party vendors. The question that I hear over and over again from my customers is if they can install sample data on an existing Joomla site with data and for me this will be a relief to know now it's possible. Kudos Bakual :)

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/7680.}

[Bakual]

It will certainly not be in 3.5.
It may go into 3.6, but then there should be webservices available with 3.6 which means the whole thing could be written using those webservices. Thus it's unlikely that the module will go in as it is now. Most likely it will be rewritten to use the webservices we are supposed to have by then 😄

[andrepereiradasilva]

@Bakual i din't know about this PR until you pointed. Thanks.
Is this waiting for 3.7 webservices?

[Bakual]

Kind of, yes.

[andrepereiradasilva]

ok then. will wait.

[Luchen6]

I have tested this item 🔴 unsuccessfully on 9e2d6ac

Applied patch and discovered module and 4 plugins. Activated plugins and added a module in cPanel position.

Deactivating one of the plugins changed the possible sample data to install.
Choosing 'Testing sampledata' gave a modal window with untranslated text MOD_SAMPLEDATA_CONFIRM_TASK.

Could not revered the patch
Applying OK resulted in 'Step 1 Failed: Another Tag has the same alias (remember it may be a trashed item).'

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/7680.}

[arrowthemes]

Is this still scheduled for 3.7.0?

[Bakual]

I haven't heard about webservices for some time. Maybe @rdeutz and @chrisdavenport know the current state and what should be done here.

[Bakual]

I have now updated this PR with latest staging and it should work again. It needs still a lot of work but since Webservices doesn't seem to gain much traction lately I thought it may be worth to revisit this approach.

[Bakual]

I've now fixed the asset issue you mentioned and I changed the access level so it's only visible to the super users.

I think I need to add the CSRF tokens to the module and plugins. If you merge it before I get to do that, I can do it also in a separate PR.
I'm not sure if we have to do any ACL checks in the plugins. The respective models should imho take care of that, but of course we could add some component.create checks as well into it.

@matrikular raised the question if it should live in a different place (eg installer) instead of a module in the cPanel. However I doubt I have the time to change that. So that would have to be done by someone else if you guys also think the cPanel is the wrong place.

I haven't done anything for people updating. Not sure what we should do there. Add the module/plugins to the #__extensions table but unpublished? The module certainly shouldn't show up for existing sites.

[mbabker]

I haven't done anything for people updating. Not sure what we should do there. Add the module/plugins to the #__extensions table but unpublished?

That'd probably be a good idea. We shouldn't put an extension onto a site that isn't properly installed, so without that it'd be sitting in abyss and people hitting discover would stumble upon it that way.

[mbabker]

You didn't do the access level change for the non-MySQL databases.

[Bakual]

Aw right. I did the lft/rgt stuff but forgot the viewlevel. Will do this evening.

[Bakual]

Done

[Bakual]

Will add the update SQLs this evening so the plugins and module are installed but unpublished.

[Bakual]

Added the update SQL files which will install the module and the two plugins as unpublished extensions.

[ciar4n]

Nice work @Bakual 👍 👍

[infograf768]

versions and date, including copyright, need correcting

[mbabker]

The only thing that needs to be manually corrected is version tags and dates in the new extension manifests. The version bump script will take care of version placeholders and copyright headers.

[infograf768]

Tested and commented #17415

[sevenstudio]

I've just read about this feature in the article about beta of 3.8 at the joomla.org and it looks like a nice addition. I've looked quickly at the code and I think that I missed something obvious - how do I build own sample data for the whole site with this approach?

Do I need to build the sample data plugin by hand?

At the moment I can prepare a QuickStart installation package with own sample data by exporting and adjusting SQL dump file in a matter of minutes. If option to install sample data will be removed from installation, with this approach I will need weeks? to create that kind of plugin for scratch.

It is a nice addition to install some sample data for a single extension after installation, but not for the whole site. In this case some kind of feature to import/export content would be great. I haven't looked closely at the whole code yet, so I'm sorry, but I must missed something obvious, how do I build own sample data for the whole site with this approach if installation of SQL dump will be removed from installation?

[Bakual]

Do I need to build the sample data plugin by hand?

You need to write your own plugin if you want to have a different sample data set.
Now I'm not exactly sure what you want to do exactly. If you just want to adjust the text, you could copy the existing plugin and alter the strings in the INI files. If you need a different structure, then you indeed need to write new code.
However depending on your needs, the code can be much simpler than what I wrote for the core ones.
Your plugin needs the "onSampleDatagetOverview()" method (eg https://github.com/joomla/joomla-cms/blob/staging/plugins/sampledata/blog/blog.php#L62-L72). That method just has to return an object with the name, title, description, icon and the count of steps.
And then you need the respective amount of steps as functions like "onAjaxSampledataApplyStep1()".
If you want, you can do it all in one step and just apply your SQL commands directly there and be done.

For core, the intention was to get rid of those SQL files because we had to always do three sets of them for the various databases we support. Also there was always errors in them (eg wrong assets and the like). That's why I use the respective models and tables so i don't have to deal with the stuff Joomla can handle itself.

[sevenstudio]

@Bakual, Thank you for your reply.

Now I'm not exactly sure what you want to do exactly.

What I want to do - I want to build a QuickStart package, this is an additional, optional installation type of Joomla! provided by most, if not by all template providers.

The name varies by different providers, for example Gavick call it a QuickStart package, RocketTheme call this a RocketLauncher ...

Basically, it is a modified, complete installation package of Joomla!, that contains files of third party extensions and own SQL sample data, that are used during installation.

After that kind of installation, client get's an exact copy of the demo site, that already has all used extensions, content and settings as in the demo site of the template.

Basic advantage of this type of installation was, that client didn't had to install everything by hand, and tries to recreate content of demo site, or guess what settings were used.

Of course this was intended only for new installs, because there was no offical way to import content in existing site.

Currently it was quite easy to build that kind of package, as you just needed to have a modified package with all additional files + own modified SQL sample data file.

Option to install sample data after installation is great (and I do understand that is a better way to handle that part of installation for the core) and it opens a new ways for developers, but if this possibility will disapear in 3.8, this may cause a lot of problems for the template providers.

I didn't have time to review the code yet but thanks for the tips!

Does decision to remove import of sample data during installation in 3.8 was alrady made? In beta I still see it in place? Once again thank you for your reply.

[mbabker]

Does decision to remove import of sample data during installation in 3.8 was alrady made? In beta I still see it in place? Once again thank you for your reply.

It will stay in place through the rest of the 3.x releases. We are looking at removing the sample data SQL files for 4.0 though.

Even if those do go away, you'd still have a couple of options for using Joomla's native web installer to create a quick start package:

1. Modify the core joomla.sql file with your additional data
2. Core supports a localise.sql file which some language distributions use to create localized data at install, you could use that to your advantage to plug in your additional data

Another thing to keep in mind is with the plugin approach, you wouldn't have to build and distribute a customized Joomla distribution. It might take some extra steps for end users, but as a distributor/packager it may be easier to use the plugin approach and advise users to install your sample data plugin which creates the same end result as your fully packaged quick start (instead of having to synchronize the entire package with each release, your plugin has the instructions in it to install the extra extensions, load the data, etc.).

[sevenstudio]

It will stay in place through the rest of the 3.x releases. We are looking at removing the sample data SQL files for 4.0 though.

Even if those do go away, you'd still have a couple of options for using Joomla's native web installer to create a quick start package:

Great to hear that! I was worried that 3.8 will bring some biger changes without any prior notice.
Also, it's nice to hear that the same/similar approach will still be an option, even in 4.0.

Yes, I see many possibilities and advantages here, so I'm happy to see such a feature available in the core. Overall process could be simplified for both sides and what is more important not limited only to the new installs.

Thanks guys for your replies and for the contribution!

[Pikkolo]

Hi :3 Thanks everybody for your just amazing effort and results.

I just installed Joomla 3.8.0 by the cpanel softacolous script. The only choice I happen to receive as I login the admin panel, is "Blog sample data". I checked the plugins, and the blog sample plugin is the only one installed.

What could I do to try and set other sample data plugins ? I would love to install the testing one.

Sorry for bothering you here :3

[brianteeman]

The testing sample data is not distributed with Joomla - it is only for testing during the development process

[Pikkolo]

Well, I probably misdefined. I talk about the "install all modules and menus" sample data set.
Thanks :3 Last question then I won't bother anymore in the wrong place :) Is there a way to find more sample plugins and install them?

[Bakual]

There is the testing sampledata plugin which can be found here: https://github.com/joomla-extensions/testing-sample-data/releases

Other than that, there are no other plugins from core. But I'd expect more such plugins to come, especially from template provides as part of their bundles.

[Pikkolo]

Thank you everybody for your very kind help :3