SearchKit - Add ability to join on multi-select ContactRef fields #20764

colemanw · 2021-07-04T18:22:02Z

Overview

Improves support for custom contactRef fields in APIv4, SearchKit, and Afform. Makes it possible to save, search for, and filter by serialized (multi-select) custom ContactRef fields.

Before

Bug prevents saving custom contactRef data in APIv4
Serialized custom contactRef outputs as numbers rather than display names in SearchKit
Not possible to create links for serialized contactRef fields in SearchKit
Bug gave incorrect results when filtering with a multiselect afform contactRef field

After

Saving custom contactRef data in APIv4 fixed, with test
Serialized custom contactRef outputs display names in SearchKit
Possible to create links for serialized contactRef fields in SearchKit
Fixed filtering with a multiselect afform contactRef field, added test
Also made it possible to use a text field instead of an autocomplete-select as a contactRef search filter in Afform

Comments

This also reverts #20156 as the limitation has been lifted.

This allows the API to return e.g. an array of display names from a serialized contact reference field.

civibot · 2021-07-04T18:22:05Z

(Standard links)

If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

eileenmcnaughton · 2021-07-05T01:58:36Z

@kurund @michaelmcandrew is this something you are able to test?

kurund · 2021-07-05T08:40:30Z

@kurund @michaelmcandrew is this something you are able to test?

Yup, will check it later today.

eileenmcnaughton · 2021-07-05T22:19:56Z

thanks @kurund - it's good to merge once you OK it

kurund · 2021-07-06T00:10:59Z

@colemanw

I am getting this weird behavior after applying the patch on nightly build.

I had to manually apply the patch as auto apply failed :(

I will recheck again if I missed something.

colemanw · 2021-07-06T01:12:19Z

What nightly build did you use?
Have you cleared caches and run upgrade scripts?

kurund · 2021-07-06T09:52:01Z

What nightly build did you use?

civicrm-NIGHTLY-drupal.tar.gz ( 5.40.alpha1-202107051700 )

Have you cleared caches and run upgrade scripts?

Yes, cache cleared and upgrade went well.

kurund · 2021-07-06T10:05:07Z

I did a fresh setup and above errors are gone. Let me test the functionality and get back to you.

kurund · 2021-07-06T11:10:37Z

@colemanw

Overall patch is working as expected. I found few minor glitches.

a. Now we can add contact reference autocomplete as filter and is working fine. Only issue is that it's not respecting the configuration settings of autocomplete. eg: if the contact field has advanced filter 'action=get&contact_type=Organization' it still shows all the contacts.

b. I was trying to replicate activity tab and 'With Contact' does have individual clickable links but contact id is not evaluated. for example http://localhost/civicrm/contact/view?reset=1&cid=[Activity_ActivityContact_Contact_01.id]

c. For one of the search I am using contact sub type as display column. It is showing correct result in Compose Search however in the table display it just shows blank.

I will do few more tests later update here.

colemanw · 2021-07-06T13:42:50Z

@kurund thanks for the testing

a. I think this is a related issue but should be a separate PR. It's a little tricky because traditional contact reference fields don't use the same callback as EntityRef fields, but afform uses EntityRef.

b. I couldn't reproduce this. Can you create a new search to demonstrate the problem and paste the url fragment here? For example when I tested, my search was
civicrm/admin/search#/create/Activity?params=%7B"version":4,"select":%5B"id","subject","Activity_ActivityContact_Contact_01.display_name"%5D,"orderBy":%7B%7D,"where":%5B%5D,"groupBy":%5B%5D,"join":%5B%5B"Contact%20AS%20Activity_ActivityContact_Contact_01","LEFT","ActivityContact",%5B"id","%3D","Activity_ActivityContact_Contact_01.activity_id"%5D,%5B"Activity_ActivityContact_Contact_01.record_type_id:name","%3D","%5C"Activity%20Targets%5C""%5D%5D%5D,"having":%5B%5D%7D

c. I couldn't reproduce this either. Can you paste a url fragment of a search to reproduce?

…links This handles multivalued fields by looping through each item and formatting it as a link

colemanw · 2021-07-06T16:19:41Z

FYI I just noticed a problem where the admin search screen was formatting every field as a link instead of just label fields like display_name. Fixed and rebased.

kurund · 2021-07-07T13:29:26Z

c. I have created a contact list with related contacts and trying to display contact sub type of related contact.

Compose Search (where it is shown correctly)

{"version":4,"select":["row_count"],"orderBy":[],"where":[["contact_type:name","=","Organization"]],"groupBy":[],"join":[["Contact+AS+Contact_RelationshipCache_Contact_01","LEFT","RelationshipCache",["id","=","Contact_RelationshipCache_Contact_01.far_contact_id"],["Contact_RelationshipCache_Contact_01.near_relation:name","IN",["Manufactured+by","Supplied+by"]]]],"having":[],"limit":100,"offset":0}

Table view (not showing sub type)
{"return":"row_count","savedSearch":{"id":9,"name":"Contacts","label":"Contacts+Search","form_values":null,"mapping_id":null,"search_custom_id":null,"api_entity":"Contact","api_params":{"version":4,"select":["id","display_name","Contact_RelationshipCache_Contact_01.far_relation:label","Contact_RelationshipCache_Contact_01.display_name","Contact_RelationshipCache_Contact_01.contact_sub_type:label"],"orderBy":[],"where":[["contact_type:name","=","Organization"]],"groupBy":[],"join":[["Contact+AS+Contact_RelationshipCache_Contact_01","LEFT","RelationshipCache",["id","=","Contact_RelationshipCache_Contact_01.far_contact_id"],["Contact_RelationshipCache_Contact_01.near_relation:name","IN",["Manufactured+by","Supplied+by"]]]],"having":[]},"created_id":18742,"modified_id":18742,"expires_date":null,"created_date":"2021-07-07+14:14:15","modified_date":"2021-07-07+14:24:58","description":null,"groups":[],"displays":[{"id":11,"name":"Contact_Search","label":"Contact+Search","saved_search_id":9,"type":"table","settings":{"limit":50,"pager":true,"columns":[{"key":"id","label":"Contact+ID","dataType":"Integer","type":"field"},{"key":"display_name","label":"Display+Name","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.far_relation:label","label":"Contact+Related+Contacts:+Relationship+from+contact","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.display_name","label":"Contact+Related+Contacts:+Display+Name","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.contact_sub_type:label","label":"Contact+Related+Contacts:+Contact+Subtype","dataType":"String","type":"field"}]},"acl_bypass":false}]},"display":{"id":11,"name":"Contact_Search","label":"Contact+Search","saved_search_id":9,"type":"table","settings":{"limit":50,"pager":true,"columns":[{"key":"id","label":"Contact+ID","dataType":"Integer","type":"field"},{"key":"display_name","label":"Display+Name","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.far_relation:label","label":"Contact+Related+Contacts:+Relationship+from+contact","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.display_name","label":"Contact+Related+Contacts:+Display+Name","dataType":"String","type":"field"},{"key":"Contact_RelationshipCache_Contact_01.contact_sub_type:label","label":"Contact+Related+Contacts:+Contact+Subtype","dataType":"String","type":"field"}]},"acl_bypass":false},"sort":[],"filters":{},"afform":null}

kurund · 2021-07-07T13:41:49Z

b. I couldn't reproduce this. Can you create a new search to demonstrate the problem and paste the url fragment here?

Table view

{"return":"row_count","savedSearch":{"id":10,"name":"Transfer_Search","label":"Transfer+Search","form_values":null,"mapping_id":null,"search_custom_id":null,"api_entity":"Activity","api_params":{"version":4,"select":["id","subject","GROUP_CONCAT(DISTINCT+Activity_ActivityContact_Contact_01.display_name)+AS+GROUP_CONCAT_DISTINCT_Activity_ActivityContact_Contact_01_display_name"],"orderBy":[],"where":[["activity_type_id:name","=","Transfer"]],"groupBy":["id"],"join":[["Contact+AS+Activity_ActivityContact_Contact_01","INNER","ActivityContact",["id","=","Activity_ActivityContact_Contact_01.activity_id"],["Activity_ActivityContact_Contact_01.record_type_id:name","=","\"Activity+Targets\""]]],"having":[]},"created_id":18742,"modified_id":18742,"expires_date":null,"created_date":"2021-07-07+14:34:15","modified_date":"2021-07-07+14:37:06","description":null,"groups":[],"displays":[{"id":12,"name":"Transfer_Search","label":"Transfer+Search","saved_search_id":10,"type":"table","settings":{"limit":50,"pager":true,"columns":[{"key":"id","label":"Activity+ID","dataType":"Integer","type":"field"},{"key":"subject","label":"Subject","dataType":"String","type":"field"},{"key":"GROUP_CONCAT_DISTINCT_Activity_ActivityContact_Contact_01_display_name","label":"Activity+Contacts:+(List)+Display+Name","dataType":"String","type":"field","link":{"path":"civicrm/contact/view?reset=1&cid=[Activity_ActivityContact_Contact_01.id]"},"title":"View+Activity+Contacts"}]},"acl_bypass":false}]},"display":{"id":12,"name":"Transfer_Search","label":"Transfer+Search","saved_search_id":10,"type":"table","settings":{"limit":50,"pager":true,"columns":[{"key":"id","label":"Activity+ID","dataType":"Integer","type":"field"},{"key":"subject","label":"Subject","dataType":"String","type":"field"},{"key":"GROUP_CONCAT_DISTINCT_Activity_ActivityContact_Contact_01_display_name","label":"Activity+Contacts:+(List)+Display+Name","dataType":"String","type":"field","link":{"path":"civicrm/contact/view?reset=1&cid=[Activity_ActivityContact_Contact_01.id]"},"title":"View+Activity+Contacts"}]},"acl_bypass":false},"sort":[],"filters":{},"afform":null}

Each activity contacts have links like: http://localhost/civicrm/contact/view?reset=1&cid=[Activity_ActivityContact_Contact_01.id]

kurund · 2021-07-07T13:43:37Z

a. I think this is a related issue but should be a separate PR. It's a little tricky because traditional contact reference fields don't use the same callback as EntityRef fields, but afform uses EntityRef.

Thanks, that make sense.

colemanw · 2021-07-07T14:07:37Z

@kurund I meant the url you see at the top of your screen when you compose a search. For a new search (before hitting Save) you'll see the url change as you update criteria.
For an existing search, you can click the "Clone" button and it will bring you to an unsaved search with those criteria in the url.

kurund · 2021-07-07T14:14:01Z

b

http://localhost/civicrm/admin/search#/create/Activity?params=%7B%22version%22:4,%22select%22:%5B%22id%22,%22subject%22,%22GROUP_CONCAT(DISTINCT%20Activity_ActivityContact_Contact_01.display_name)%20AS%20GROUP_CONCAT_DISTINCT_Activity_ActivityContact_Contact_01_display_name%22%5D,%22orderBy%22:%5B%5D,%22where%22:%5B%5B%22activity_type_id:name%22,%22%3D%22,%22Transfer%22%5D%5D,%22groupBy%22:%5B%22id%22%5D,%22join%22:%5B%5B%22Contact%20AS%20Activity_ActivityContact_Contact_01%22,%22INNER%22,%22ActivityContact%22,%5B%22id%22,%22%3D%22,%22Activity_ActivityContact_Contact_01.activity_id%22%5D,%5B%22Activity_ActivityContact_Contact_01.record_type_id:name%22,%22%3D%22,%22%5C%22Activity%20Targets%5C%22%22%5D%5D%5D,%22having%22:%5B%5D%7D

c.

http://localhost/civicrm/admin/search#/create/Contact?params=%7B%22version%22:4,%22select%22:%5B%22id%22,%22display_name%22,%22Contact_RelationshipCache_Contact_01.far_relation:label%22,%22Contact_RelationshipCache_Contact_01.display_name%22,%22Contact_RelationshipCache_Contact_01.contact_sub_type:label%22%5D,%22orderBy%22:%5B%5D,%22where%22:%5B%5B%22contact_type:name%22,%22%3D%22,%22Organization%22%5D%5D,%22groupBy%22:%5B%5D,%22join%22:%5B%5B%22Contact%20AS%20Contact_RelationshipCache_Contact_01%22,%22LEFT%22,%22RelationshipCache%22,%5B%22id%22,%22%3D%22,%22Contact_RelationshipCache_Contact_01.far_contact_id%22%5D,%5B%22Contact_RelationshipCache_Contact_01.near_relation:name%22,%22IN%22,%5B%22Manufactured%20by%22,%22Supplied%20by%22%5D%5D%5D%5D,%22having%22:%5B%5D%7D

Hope this helps!

colemanw · 2021-07-07T18:26:01Z

Yes that helped. I can now reproduce b and c. On investigation, I don't think either should block this PR.

B. Is out-of-scope, as this PR fixes links for multi-select fields, not single fields aggregated by grouping. I do hope to tackle that issue at some point, but this PR is already quite large.

C. is a pre-existing condition. I was able to reproduce it in a test and will work on a fix.
UPDATE: here is the fix: #20799

Based on the review & testing I think this is merge-ready.

colemanw · 2021-07-07T20:59:50Z

retest this please

eileenmcnaughton · 2021-07-07T22:04:17Z

Thanks for the testing @kurund - I wonder how much it feels like things have moved on in your time away from Civi - hopefully a little bit :-)

Anyway - I'm upping this to merge-on-pass after reading through the above

colemanw · 2021-07-07T23:29:47Z

Unrelated test fail.

kurund · 2021-07-08T16:07:09Z

@colemanw @eileenmcnaughton

Thanks :)

colemanw added 2 commits July 4, 2021 13:39

APIv4 - Fix saving multi-contactRef field values

ef9170e

APIv4 - Support returning join data from serialized fields

1c33974

This allows the API to return e.g. an array of display names from a serialized contact reference field.

civibot bot added the master label Jul 4, 2021

colemanw changed the title ~~Multi contact ref joins~~ SearchKit - Add ability to join on multi-select ContactRef fields Jul 4, 2021

colemanw added the has-test label Jul 4, 2021

colemanw added 3 commits July 6, 2021 12:17

SearchKit - Allow multi-valued contact ref fields to be formatted as …

91b6118

…links This handles multivalued fields by looping through each item and formatting it as a link

SearchKit - Fix array filters with serialized fields

84b531d

Afform - Support implicit joins in SearchDisplay filter fields

96ff473

colemanw force-pushed the multiContactRefJoins branch from 0a99630 to 96ff473 Compare July 6, 2021 16:18

colemanw added the merge ready PR will be merged after a few days if there are no objections label Jul 7, 2021

eileenmcnaughton added the merge on pass label Jul 7, 2021

colemanw merged commit 2beecf9 into civicrm:master Jul 7, 2021

eileenmcnaughton deleted the multiContactRefJoins branch July 7, 2021 23:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SearchKit - Add ability to join on multi-select ContactRef fields #20764

SearchKit - Add ability to join on multi-select ContactRef fields #20764

colemanw commented Jul 4, 2021 •

edited

civibot bot commented Jul 4, 2021

eileenmcnaughton commented Jul 5, 2021

kurund commented Jul 5, 2021

eileenmcnaughton commented Jul 5, 2021

kurund commented Jul 6, 2021

colemanw commented Jul 6, 2021

kurund commented Jul 6, 2021 •

edited

kurund commented Jul 6, 2021

kurund commented Jul 6, 2021

colemanw commented Jul 6, 2021

colemanw commented Jul 6, 2021

kurund commented Jul 7, 2021

kurund commented Jul 7, 2021

kurund commented Jul 7, 2021

colemanw commented Jul 7, 2021

kurund commented Jul 7, 2021 •

edited

colemanw commented Jul 7, 2021 •

edited

colemanw commented Jul 7, 2021

eileenmcnaughton commented Jul 7, 2021

colemanw commented Jul 7, 2021

kurund commented Jul 8, 2021

SearchKit - Add ability to join on multi-select ContactRef fields #20764

SearchKit - Add ability to join on multi-select ContactRef fields #20764

Conversation

colemanw commented Jul 4, 2021 • edited

Overview

Before

After

Comments

civibot bot commented Jul 4, 2021

eileenmcnaughton commented Jul 5, 2021

kurund commented Jul 5, 2021

eileenmcnaughton commented Jul 5, 2021

kurund commented Jul 6, 2021

colemanw commented Jul 6, 2021

kurund commented Jul 6, 2021 • edited

kurund commented Jul 6, 2021

kurund commented Jul 6, 2021

colemanw commented Jul 6, 2021

colemanw commented Jul 6, 2021

kurund commented Jul 7, 2021

kurund commented Jul 7, 2021

kurund commented Jul 7, 2021

colemanw commented Jul 7, 2021

kurund commented Jul 7, 2021 • edited

colemanw commented Jul 7, 2021 • edited

colemanw commented Jul 7, 2021

eileenmcnaughton commented Jul 7, 2021

colemanw commented Jul 7, 2021

kurund commented Jul 8, 2021

colemanw commented Jul 4, 2021 •

edited

kurund commented Jul 6, 2021 •

edited

kurund commented Jul 7, 2021 •

edited

colemanw commented Jul 7, 2021 •

edited