Inquiry: Increasing default result size for Search API and SSE stream overhead for 100+ Things

[radhikais07]

Context: I am working on a Digital Twin project for a building involving ~100+ things. My current pipeline streams AQI data via Sensor → MQTT → Kafka → Eclipse Ditto.

Problem: When fetching the initial state of the city to cache in a Flask backend, the Search API (/api/2/search/things) defaults to a size of 25. While I can use ?limit=100, I am concerned about the performance implications and potential "throttling" when frequently requesting large batches.

Questions for Developers:

Batch Limits: Is there a hard internal limit to the size parameter in the Search API beyond which performance significantly degrades for the search index?

SSE Scaling: For a twin with 100+ things, is the Server-Sent Events (SSE) /things endpoint the recommended way to keep a secondary cache synced, or should I be looking at the Managed Connections (Kafka/MQTT) for outbound synchronization?

Pagination vs. Bulk: For initial state synchronization of 100+ items, is it better to use a single large request or multiple paginated requests using the cursor or Last-Event-ID approach?

[thjaeckle]

That depends on the size of the things.
If they are 100kB in size each, I would not recommend using a page size of 100 - then you would load 10MB in a single HTTP request.

If they are small, 100 page size is not an issue.

Batch Limits: Is there a hard internal limit to the size parameter in the Search API beyond which performance significantly degrades for the search index?

200 is the default upper limit for page size - this can be configured, but I would not recommend configuring it higher.

SSE

The SSE for the search is great if you e.g. want to query several thousands in a "streaming way" - so already do something with the first ones of the stream during the results are still streamed.

to keep a secondary cache synced

If you really need to do that (Ditto is already a cache itself), I would recommend setting up a Kafka connection which feeds the cache with updates.
Regular polling (via search) is much more inefficient.