Configure on a namespace basis the fields to index in the thing-search

[thjaeckle]

Currently, the complete JSON of a thing (including all attributes+features) is automatically indexed in the search.
We learned from input of the community that this might not always be wished as this could lead to a lot of load on the search DB - which could e.g. be prevented if some fields are never used in the search "fields".

[an1310]

This is of strategic interest to us -- when would be a good time to go over requirements/design ideas for a possible contribution?

[thjaeckle]

@an1310 I will try to find out where this could be placed the best.
As far as I remember there was an extension point for just that purpose used by Bosch's commercial offering of Ditto.

[thjaeckle]

@an1310 I took myself a closer look and I think I found the mentioned extension point to do index only custom fields in namespaces.

I think to remember that we explicitly added CachingSignalEnrichmentFacade extension point in Ditto to retrieve only selectively parts of "things" (based on their namespace "search index configuration") when processing - retrieving a "partial thing" in order to update the search index:

ditto/internal/models/signalenrichment/src/main/java/org/eclipse/ditto/internal/models/signalenrichment/DittoCachingSignalEnrichmentFacade.java

Lines 113 to 158 in 022dffe

	public CompletionStage<JsonObject> retrievePartialThing(final ThingId thingId,
	@Nullable final JsonFieldSelector jsonFieldSelector, final DittoHeaders dittoHeaders,
	@Nullable final Signal<?> concernedSignal) {
	final List<ThingEvent<?>> thingEvents =
	(concernedSignal instanceof ThingEvent) && !(ProtocolAdapter.isLiveSignal(concernedSignal)) ?
	List.of((ThingEvent<?>) concernedSignal) : List.of();
	// as second step only return what was originally requested as fields:
	final var cachingParameters =
	new CachingParameters(jsonFieldSelector, thingEvents, true, 0);
	return doRetrievePartialThing(thingId, dittoHeaders, cachingParameters)
	.thenApply(jsonObject -> applyJsonFieldSelector(jsonObject, jsonFieldSelector));
	}
	/**
	* Retrieve parts of a thing.
	*
	* @param thingId ID of the thing.
	* @param jsonFieldSelector the selected fields of the thing.
	* @param dittoHeaders Ditto headers containing authorization information.
	* @param concernedSignals the Signals which caused that this partial thing retrieval was triggered
	* (e.g. a {@code ThingEvent})
	* @param minAcceptableSeqNr minimum sequence number of the concerned signals to not invalidate the cache.
	* @return future that completes with the parts of a thing or fails with an error.
	*/
	@SuppressWarnings("java:S1612")
	public CompletionStage<JsonObject> retrievePartialThing(final EntityId thingId,
	final JsonFieldSelector jsonFieldSelector,
	final DittoHeaders dittoHeaders,
	final Collection<? extends Signal<?>> concernedSignals,
	final long minAcceptableSeqNr) {
	final List<ThingEvent<?>> thingEvents = concernedSignals.stream()
	.filter(signal -> signal instanceof ThingEvent && !Signal.isChannelLive(signal))
	.map(signal -> (ThingEvent<?>) signal)
	.collect(Collectors.toList());
	// as second step only return what was originally requested as fields:
	final var cachingParameters =
	new CachingParameters(jsonFieldSelector, thingEvents, true, minAcceptableSeqNr);
	return doRetrievePartialThing(thingId, dittoHeaders, cachingParameters)
	.thenApply(jsonObject -> applyJsonFieldSelector(jsonObject, jsonFieldSelector));
	}

We would need a custom JsonFieldSelector configured per namespace name, defining which fields of a "thing" to retrieve (and cache) for updating the search index.
The configuration could look like:

• a list/map of namespaces
  • namespace
    • indexedFields as JsonFieldSelector

Example what I have in mind (to put in the search.conf as configuration of the mentioned extension point):

ditto/thingsearch/service/src/main/resources/search.conf

Line 11 in 022dffe

caching-signal-enrichment-facade-provider = org.eclipse.ditto.thingsearch.service.persistence.write.streaming.DittoCachingSignalEnrichmentFacadeProvider

namespaces = [
  {
    namespace-name = "org.eclipse.ditto.foo"
    indexed-fields = [ 
      "thingId", 
      "policyId", 
      "_revision", 
      "_created", 
      "_modified", 
      "attributes/indexed-one"
      "attributes/complex-jsonobj"
    ]
  }
]

I hope that helps a little. Maybe a Ditto committer still at Bosch can share more details how this can be done, based on the identified extension point.

[thjaeckle]

@an1310 are you working on this? Any update or PR for providing early feedback?

[an1310]

Yes I am.