dev/core#4130 add template support for imports

[eileenmcnaughton]

Overview

dev/core#4130 add template support for imports

This creates UserJob Templates in tandem wtih the use of saved civicrm_mapping records for imports.

This provides a nice-to-have functionality for non-Civi-Import imports - ie the import can store details from the initial DataSource screen - anyone who always has to remember to save dates will appreciate that. In addition for Civi-import imports it can store more nuanced defaults for Civi-Import (which works on the Contribution Import)

The templates are loaded when

1. the url holds a valid template_id - eg. civicrm/contribute/import?reset=1&template_id=2. A link to that url appears on the Summary screen -
   

or

2. a Saved Mapping is selected on the DataSource screen. In this case the defaults for the DataSource screen are not re-loaded but the defaults for MapField is. In practice this is invisible when civi-import is not enabled

E.g - DataSource with contact type default loaded

E.g - The sort of field you can set defaults for with Civi-import enabled

https://lab.civicrm.org/dev/core/-/issues/4130

Before

MappingFields are too blunt to save the options available in Contribution Import with CiviImport enabled or the general options on the DataSource screen.

The goal is to work towards replacing the old civicrm_mapping - which is also inadequate for some soft credit details - with UserJob template records.

After

Records in civicrm_user_job with is_template = ` used to provide templates for full import configurration.

Both the UserJob Template and the SavedMapping are managed in tandem with a soft link based on name - ie
if civicrm_mapping.name = 'saved_map' then civicrm_user_job.name = 'import_saved_map' for the relevant template.

If the UserJob does not exist it will be created when the DataSource form is submitted with savedMapping not empty or when the MapField form is submitted with saveMapping not empty. Since there are not yet any template UserJobs in the database the reverse will not occur. We could instead create on upgrade. At this stage I decided it would be easier to create as needed in the first iteration.

When the Mapping is updated on the MapField form (either by SaveMapping or UpdateMapping being selected the UserJob template will also be updated.

On the Summary screen a link to use the template is included.

Technical Details

*Flow - do an import, saving a saved mapping. *

• This will import an associated UserJob Template when the Mapping is saved - ie
  
• a link to it will be present on the Summary screen allowing
  Flow 2 Use the template from the url

Add template_id =x to the url & process the import....

Flow 3 Update a template loaded by url

Make a change to an import loaded from the url - note even if the change is on the DataSource page it will only be updated if the Update or Save option is chosen on the MapField page

Flow 3 Update a template loaded by SavedMapping field on the DataSource page

In this case no change will be visible with Civi-Import disabled

Comments

[civibot]

(Standard links)

• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

test this please

[aydun]

A few things:

1. There is something odd with the results table after importing:
   
   The blue looks like clickable links and clicking on them underlines the whole row. The HTML confirms they are not links.

2. The contributions seem to import correctly. Hurray!

3. If I click the link to reuse the template, I get the same options showing up, including the Saved Field Mapping. However, everything is changeable except that Saved Field Mapping:
   
   For consistency, maybe select the saved field as the default, but still changeable.

[eileenmcnaughton]

@aydun thanks for looking - I'm gonna check out the links - but re the mapping - the reason it is locked is the 2 are tied together - ie there is a one-to-one between the import template and the mapping (with a view to maybe one day getting rid of the mapping)

[eileenmcnaughton]

@aydun that's odd - I just tried to see the psuedo urls with Shoreditch & greenwich & didn't see them ...

I did make the numbers into real urls with Civi-Import enabled though - cos it's handy

[aydun]

@aydun thanks for looking - I'm gonna check out the links - but re the mapping - the reason it is locked is the 2 are tied together - ie there is a one-to-one between the import template and the mapping (with a view to maybe one day getting rid of the mapping)

OK

@aydun that's odd - I just tried to see the psuedo urls with Shoreditch & greenwich & didn't see them ...

Strange - I'm still seeing them on the test site http://core-25808-1xum.test-3.civicrm.org:8001

[eileenmcnaughton]

@aydun yeah - I could see it on that demo site too - I think I fixed it now

[aydun]

Those sneaky little '/' !! Obviously you meant '< / a>' not '< a >' embedded in another '< a >' - duh

All seems ok without Civi-Import enabled.

With Civi-Import enabled, I get:

Deprecated function: json_decode(): Passing null to parameter #1 ($json) of type string is deprecated in Civi\Api4\Event\Subscriber\ImportSubscriber->getImportTableFromEvent() (line 125 of /home/jenkins/bknix-dfl/build/core-25808-3gzg/web/sites/all/modules/civicrm/ext/civiimport/Civi/Api4/Event/Subscriber/ImportSubscriber.php).

on the queue runner page.

After importing, I clicked the row count to get to the SearchKit page. From there, I wasn't sure what I was supposed to be able to do with it ... but I found the fields were editable so I tried changing the currency from GBP to USD. That didn't affect the imported contributions (makes sense), so I tried importing again and got: An error occurred while attempting to import 1 row/s. but no more info.

[eileenmcnaughton]

I just pushed up a fix to avoid that notice - it would be php version specific so I hadn't hit it

Regarding the landing page - yeah the idea isn't that you would re-import succeeded rows - it's the same SearchKit as for other errors. @colemanw I do want to check in with you on how to make the actions conditional per row - then we could suppress them is the status is not appropriate. I think that is out of scope though. It would be great if edit-in-place could also be conditional. I think we could merge this based on the review & follow up on that

@aydun the main advantage of being able to view the rows after import is you can see the entity it linked through & click through to it, if you wish

[colemanw]

Per conversation this is mergeable and can do with a followup.