Provide templates for migrators

[st-anton]

During a migration, the BS9 templates are overwritten by the BS8 templates. This is not bad, the user has it's settings like format of date, filter settings, comments entry and so on. On the other side, the layout settings are worthless, table rows and columns are no more useful.
Idea: We provide one or two templates as 'reference' to show the user how to set up a layout. With the template -> export we can easily generate this files and put them in /media/com_biblestudy/templates.
@tomfuller2 your template has less 'table' entries, mine has more 'table' entries: You and I provide a template each?

[bcordis]

@tomfuller2 are we going to do this? Not sure this will be helpful if they have migrated on 9.0.1?

[bcordis]

@tomfuller2 Not sure we can do this. More so this will be documentation more than code works.

[bcordis]

@tomfuller2 we could ask people to export their templates but this would be something if they used custom template_fils

[tomfuller2]

I feel like this is already possible. I’ll check it out.

[bcordis]

Yes I think it is possible now also. We could create multiple versions of a template that they could use. We also need to document CSS usage system. And a template repository for them to pull from. Like the idea.

On Sat, Mar 6, 2021 at 4:46 PM Tom Fuller ***@***.***> wrote: I feel like this is already possible. I’ll check it out. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#540 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHSXU33IDID2Z77AI2MDH3TCKWD3ANCNFSM4BANECZA> .

-- Sent from Gmail Mobile

[bcordis]

Picking this up eleven years later, because the answer changed: the idea is still good, but the mechanism proposed here is now the reason to build it differently.

First, the 2015 plan no longer has a working part

The proposal was "with template → export we can easily generate this files". That export is broken on every current install. CwmtemplatesController::templateExport() selects #__bsms_templates.text, a column that existed in the 10.0.0 baseline but is not in the shipped schema any more and is never re-added by an update file. An unknown column in a SELECT is fatal, so the request dies before it produces anything. Confirmed against a clean 10.5.x install and against j5-dev — neither has text or pdf. Filed separately as #1918.

Two more reasons not to build on that pair:

• templateImport() executes any statement from an uploaded file whose text merely contains the substring INSERT. Making that a supported distribution channel would promote a liability into a feature.
• getExportSetting() emits MySQL-only INSERT INTO ... SET. With Support PostgreSQL install, uninstall and update SQL #1857 (PostgreSQL) on 11.0.0, SQL-file presets would be born broken on the driver we are about to support.

What a preset actually is now

A template today is one row whose params is a large JSON registry driven by admin/forms/template.xml — filters, media/player settings, list and detail display options, and the Layout Editor's element ordering. So a preset is just a params blob. No new table, no schema change, no PHP.

That also settles a question from further up the thread: presets should carry zero #__bsms_templatecode rows and zero #__bsms_styles rows. Shipping PHP-in-DB isn't something to institutionalise, and custom_css is already an editor field on the template — which covers @bcordis's "we also need to document CSS usage system" without any new machinery.

Proposal

Storage — admin/presets/*.json, shipped with the component. No build step.

{
  "name": "Classic Table",
  "description": "Dense table listing — closest to a BS8/BS9 table layout.",
  "author": "CWM Team",
  "version": "1.0",
  "minProclaim": "10.5.10",
  "thumbnail": "classic-table.png",
  "params": { }
}

Apply — a New from preset button on the Templates list opens a picker and creates a new template row, reusing the shape of CwmtemplateModel::copy(). New row only: a preset never overwrites an existing template, so there is no destructive path to undo.

Whitelist, two gates. Bind the preset through $form->filter() against template.xml — Joomla already drops anything not declared there — and additionally strip the site-specific keys before binding:

• row columns: id, asset_id, location_id, created*, modified*, checked_out*
• content filters: lteacher_id, lseries_id, lbooknumber, ltopic_id, lmessagetype, llocations, lyears
• style FK: css (points into #__bsms_styles.filename)
• override refs: sermonstemplate, sermontemplate, teacherstemplate, teachertemplate, seriesdisplaystemplate, seriesdisplaytemplate
• cross-template refs, visible in the seeded Default params: studieslisttemplateid, detailstemplateid, teachertemplateid, serieslisttemplateid, seriesdetailtemplateid

Any of those carried in a preset either breaks on the target site or leaks the source site's structure.

Versioning — presets carry a version, and CwmtemplatemigrationHelper already knows how to backfill params introduced in later releases, so a preset authored at 10.5 stays usable at 11.x without re-authoring.

Export as preset (JSON) — this is what makes @st-anton's original offer work. "You and I provide a template each" becomes: configure a template, export JSON, open a PR adding it to admin/presets/. It also gives us a reason to retire the broken SQL exporter rather than repair it.

The test that has to be written

The obvious check — "preset applied, row created, no errors" — passes while params silently vanish. $form->filter() drops any param lacking a type="hidden" declaration in template.xml, and every Layout Editor per-element setting depends on one being present. A preset carrying a setting whose hidden field was never declared loses it with no error anywhere.

So acceptance is a round-trip params diff: fully configure a template, export it as a preset, apply it as a new template, diff the two registries. Any key present before and absent after is a missing declaration. Anything weaker cannot tell success from silent loss.

Staging

Phase 1 is the above — bundled presets, apply-as-new, JSON export. Self-contained.

Phase 2 is @bcordis's 2021 "template repository for them to pull from". That one is remote fetch, so it carries new SSRF and supply-chain surface and wants its own review. Deliberately kept separate.

One thing this does not solve

@st-anton's actual complaint was that migration overwrote an existing template — one that menu items already point at. New from preset gives a good starting point but leaves those users repointing menus by hand. Going with new-row-only for now; if that pain turns out to be real once presets are in use, an apply-to-existing action with a confirm and a snapshot of the prior params can follow as its own change.

[bcordis]

Addendum after reading #1179, which turns out to be load-bearing here in a way that changes two things above.

#1179 removes the main risk I flagged

I said the dangerous failure mode is silent param loss: $form->filter() drops any param lacking a type="hidden" declaration in template.xml, so a preset can apply "successfully" while quietly losing settings.

#1179's proposed storage — a single layout_elements JSON array stored in one hidden field — makes that failure mode structurally impossible for layout settings. One declaration, one blob, nothing per-element to forget. That is a much better foundation for presets than the current ~2,500 lines of flat fields, and it is worth saying out loud that presets are a second, independent argument for #1179's Option 1.

The round-trip params diff is still the right acceptance test, but afterwards it guards a much smaller surface.

So the sequencing is a real decision

A preset is a serialised copy of the params format. #1179 changes that format, and is scoped as a v11.0 breaking change. That gives two options:

• Presets after Dynamic template layout elements — repeatable spacers and custom fields #1179. The preset file format is born in the shape we intend to keep. Nothing to migrate. Costs: no presets in 10.x at all.
• Presets now, versioned. The version / minProclaim fields already in the proposed preset JSON exist precisely for this. Dynamic template layout elements — repeatable spacers and custom fields #1179 already plans a converter — read legacy flat params, write layout_elements — and presets can ride the exact same converter rather than needing their own. A 10.5-era preset then keeps working at 11.x.

The second is only cheap because #1179 commits to that back-compat converter. If #1179 ever drops the legacy-read path, 10.x-era presets become dead files, so whichever way this goes the two should be decided together rather than separately.

My inclination is presets-after-#1179 — the whole value of a preset is that it is a stable artefact people trade and contribute, and minting a format we already know we are about to break is the one thing that undermines that. But it does mean migrators get nothing until v11.

Unchanged by this

The rest stands: params-only, no templatecode or styles rows, the explicit strip-list for site-specific keys, apply-as-new-row only, and JSON rather than SQL.

The broken SQL exporter is now filed as #1918. It turned out to be more interesting than a dead reference — text/pdf were removed from the install SQL but no update file ever drops them, so export is fatal on clean installs and works fine on sites upgraded from the 10.0.0 baseline. Verified across three local sites. That lineage split is exactly the kind of thing that makes a bug report read as "works for me", and it is another argument for not building the distribution story on that code path.

[bcordis]

Correction to my previous comment.

I said #1918's export bug was fatal on clean installs but "works fine on sites upgraded from the 10.0.0 baseline", inferring a lineage split from the fact that no update SQL file drops #__bsms_templates.text.

That inference was wrong. Proclaim drops those columns from PHP, not SQL: proclaim.script.php::dropLegacyTemplateColumns() runs in postflight() and removes text and pdf with an idempotent existence-checked ALTER TABLE. It has been there since 10.1.0, and its docblock explains why — the columns were never rendered on the frontend. Same pattern accounts for the scripture flat columns (CwmscriptureMigration::LEGACY_COLUMNS) and #__bsms_studies.teacher_id (CwmmigrationHelper). DROP COLUMN isn't idempotent in MySQL, so the carefully-retired columns are precisely the ones whose DROP can't live in a replayable migration file.

So there is no lineage split. The column is gone everywhere, the removal was deliberate and correct, and only the exporter is stale — which makes the bug simpler and more uniform than I described, not less severe. #1918 has been rewritten accordingly, and my "add idempotent DROPs so upgraded sites converge" suggestion is withdrawn; that work already exists.

Nothing in the preset design depends on this. The reasons not to build distribution on the SQL export path — the arbitrary-SQL import sink, and MySQL-only INSERT ... SET against #1857 — are unaffected.

[bcordis]

Decision recorded: the importer is replaced, not repaired — and no former format is accepted

Following on from the comments above and #1925 / #1926.

The direction is a clean break. templateImport() gets rebuilt on a new data format, and legacy .sql template files are rejected outright with a clear message rather than parsed best-effort. The exporter has been fatal on every current install since the text/pdf columns were retired, so no file produced by a supported version exists; anything surviving is 9.x-era and MySQL-only. There is no population to preserve compatibility for.

The hard requirement on the replacement is that it never executes uploaded content. The current importer splits an upload on ; and runs any statement whose text merely contains the substring INSERT. That is an arbitrary-SQL-execution primitive reachable from a file the admin did not author — which is precisely the sharing use case this discussion has wanted since 2015. The full requirement list is on #1926; the short version is that the format carries data, not statements, keys are validated against template.xml before anything is written, and writes go through the model/Table layer rather than raw SQL.

Two findings that make this materially easier than it looked

#__bsms_templatecode is vestigial. Its templatecode column holds PHP source — the seeded row still has Joomla 3-era JHtml::addIncludePath() calls — and nothing executes it. No eval, no include; Cwmpagebuilder never reads the table. Its filename is the only live part: it flags the page builder on, and in mod_proclaim maps to a layout via layout = 'default_' . $filename. Verified across four dev sites — the only published row anywhere is easy, matching site/tmpl/cwmsermons/default_easy.php.

So a shared template needs to carry a layout selector, never code. That removes most of the reason the old format was dangerous.

The 2015 goal does not need the exporter at all. A Proclaim template is one row whose params JSON is the entire design. Shipping reference designs — the actual ask in this thread — means shipping preset JSON files in media/ plus an "apply preset" action, and the on-disk layouts already exist in useful variety:

cwmsermons:  main, simple, simple2, easy, fluidsermonlist
cwmsermon:   main, simple, fluidsermondetail
cwmteacher:  main, cards, list
cwmseriesdisplay(s): main, custom

That is a small, shippable feature and it does not block on the importer rebuild. Backup/restore was checked as an alternative delivery route and ruled out: it is all-or-nothing across every Proclaim table, so it cannot move one template without flattening messages and teachers too.

Suggested split: presets first (closes the original 2015 ask), importer rebuild second (#1926), and it should not be gated behind #1179 — if the params storage format later changes, presets need a data migration, not a rebuild.

[bcordis]

Correction to my comment above: the templatecode PHP is executed

I said #__bsms_templatecode was vestigial and that nothing executes its PHP column. That is wrong.

CwmtemplatecodeTable::store() writes the column to components/com_proclaim/tmpl/Cwm<View>/default_<filename>.php via File::write(), and the front end includes that file through loadTemplate(). It is a live code-generation feature — an admin authors PHP in the database and it becomes a front-end template. I missed it because I searched for eval/include and there is neither; the execution is ordinary template inclusion of a file written from the database.

site/tmpl/cwmsermons/default_easy.php in the repo is literally that mechanism's output — it carries the same header as the seeded templatecode row.

Why it matters for this thread specifically. The idea here has always been to distribute reference templates for people to import. With the current mechanism, importing a template file authored by someone else means letting that file's author put PHP in your web root — templateImport() executes the uploaded INSERTs and then calls store() on the resulting templatecode row. The REST API already refused to expose this resource for exactly that reason (plugins/webservices/proclaim/src/Extension/Proclaim.php — "arbitrary PHP in the web root — remote code execution, not a content problem, and no ACL makes that acceptable").

So the 2015 plan was not merely using a broken exporter; the sharing model it implied is unsafe as long as a shared file can carry template code.

The recommendation is unchanged and now better founded. Ship reference designs as params presets — a preset selects a layout that Proclaim already ships and tunes the ~17KB of params around it. A preset cannot introduce code, so the sharing use case this discussion has wanted since 2015 becomes safe by construction rather than by validation. Any future import format must likewise be unable to create or modify a #__bsms_templatecode row, directly or as a side effect.