Patterns: Combine local and w.org search results for Inserter

[iandunn]

#26578 is adding an API endpoint to fetch patterns from the w.org pattern directory. Plugins can also create local patterns via register_block_pattern() too. When the Inserter displays pattern search results, it'll need to combine local and remote patterns.

I'm not sure how they should be sorted, though. The remote patterns will have high-quality ElasticSearch rankings, but the local patterns will have low-quality MySQL rankings.

Normalized scores

Is it possible normalize the relevance scores from MySQL and ElasticSearch?

If so, we could simply sort them by the score, and maybe weight local ones a bit higher? We'd need to make sure Jetpack/ES returns the score along w/ the results (related tellyworth/plugin-directory#1)

ElasticSearch & Privacy

We could potentially pass the local patterns to the w.org API, so that ElasticSearch can rank them all.

That raises some potential privacy concerns, though, especially since ES is done on Automattic servers (via Jetpack Search). I don't imagine there'd be much sensitive content in most, but some custom patterns might reveal something about a site's content, or have placeholder content that reveals something sensitive.

We could add a filter to disable passing local patterns to w.org, and/or add a private={true | false} parameter to register_block_pattern(). If that were disabled, we'd have to have some fallback mechanism for sorting.

w.org wouldn't have any interest in using that data for anything other than sorting, so we could avoid storing it. It'd probably be desirable from a technical/performance standpoint to store them (semi?-)permanently, though.

cc @mrxkon, @birgire, @postphotos for feedback

Other Solutions

• Maybe the UI should show them separately? We'll need some feedback from a designer.
• Maybe they'd just be tacked on to the beginning or end of the list?

Tasks

• Combine local patterns when browsing
• Adjust sanitization/validation so that local patterns can add JS, unfiltered HTML, etc if they want?
• Combine local patterns when searching

[mrxkon]

Hey @iandunn !

Since patterns aren't meant to be actual reusable content for any given site (like reusable blocks) most (I hope) won't use them along with actual data as it will be a pain to maintain if something needs to be updated either way.

I do agree & understand that there will eventually be edge-cases of let's say "misuse", but we can't do much for that in my opinion. That falls into user error and I guess we can agree that we can't avoid that no matter what. It's like creating a supposed "private CPT" but forgetting to exclude it from search 😁.

Do we really want to "merge" the results and mix-match and sort them along with the remote though? I would personally prefer to always see what I have locally first when searching and then view any potential remote patterns to use (but that's my opinion at least). So in this case there isn't much of a need of communicating something in general as far as I can understand.

If we go for the ES way I'd say yes please for a filter. This way users that do want to keep their minds at ease from the get-go when creating a site and don't want anything to be communicated will at least have an option (no need to delve into the "why"). I prefer the filter over a param as well, as the filter will just stop everything but a param can accidentally be missed again.

Side note: thanks to @TimothyBJacobs for helping me get up to speed as I've been away for quite a while and apparently I've missed "just a couple" of things here and there 😂 .

[iandunn]

🤔 , yeah, displaying them separately is definitely worth considering, thanks!

[tellyworth]

Another option to consider is that the plugin and theme directories could index patterns found in plugins and themes by parsing out register_block_pattern() calls in code that is hosted there. Then the remote API could return those in all searches.

Obviously that wouldn't cover patterns that are registered in custom code or from plugins not hosted in the directory. But that might be fine, since as you note those kind of custom patterns might have privacy or licensing implications anyway.

[iandunn]

Ah, that's a good idea 👍🏻

[iandunn]

@shaunandrews , do you have any thoughts on how local vs remote patterns should be shown in the Inserter UI (and other places)? Should they be combined into a single list, or shown in separate lists?