AppEngine: number of returned result does not match downsample_to when default limit is reached

[Annopaolo]

If a device has sent more data than the default query limit (i.e. 10000), if the following query is performed:

curl -X GET -H "Authorization: Bearer $JWT" \
        "http://$ASTARTE_HOST/appengine/v1/$REALM/devices/$DEVICE_ID/interfaces/org.astarte-platform.genericsensors.Values/streamTest/value?downsample_to=100"

The number of returned results is strictly lower than the downsample_to value. Also, the number of returned samples decreases as the total sample count increases.
If the limit parameter is specified, the number of returned results does match downsample_to, i.e. the following query

curl -X GET -H "Authorization: Bearer $JWT" \
       "http://$ASTARTE_HOST/appengine/v1/$REALM/devices/$DEVICE_ID/interfaces/org.astarte-platform.genericsensors.Values/streamTest/value?downsample_to=100&limit=200"

correctly returns 100 samples. However, if limit > 10000, the incorrect behavior comes back.

This could be due to the fact that two queries are performed: one for retrieving actual data, and one to count the number of returned samples. See

astarte/apps/astarte_appengine_api/lib/astarte_appengine_api/device/queries.ex

Lines 1278 to 1295 in d7997ac

	values_query =
	DatabaseQuery.new()
	|> DatabaseQuery.statement(values_query_statement)
	|> DatabaseQuery.put(:device_id, device_id)
	|> DatabaseQuery.put(:interface_id, interface_row[:interface_id])
	|> DatabaseQuery.put(:endpoint_id, endpoint_id)
	|> DatabaseQuery.put(:path, path)
	|> DatabaseQuery.merge(q_params)
	values = DatabaseQuery.call!(client, values_query)
	count_query =
	values_query
	|> DatabaseQuery.statement(count_query_statement)
	count = get_results_count(client, count_query, opts)
	{:ok, count, values}

There have been some issues in Cassandra regarding the behavior of the LIMIT param together with count() (see e.g. https://issues.apache.org/jira/browse/CASSANDRA-8216?focusedCommentId=15235570&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15235570).

When the actual issue is fixed, it would be great to thoroughly document the expected behaviour ( #629 👀).

[Annopaolo]

Closed by #827.