Skip to content

Commit

Permalink
client: Expose future based submissions
Browse files Browse the repository at this point in the history
Currently all the KCIDB submission APIs are synchronous, meaning we have
network latency in the call which isn't ideal for large numbers of small
submissions. The underlying BigQuery PubSub APIs offer an API based on
futures which should hopefully allow us to do better here, allowing the
client to submit data and then later check that it has actually been sent
so that the PubSub library can batch I/O or do it asynchronously.

This is already wired up at the publisher level so just add a top level
client method which exposes the future based API. We may want to also
expose something of the flow control parameters via the PublishFlowControl
passed when creating the client, though the current defaults (1000 messages
and 10Mb in flight) look reasonable.

Signed-off-by: Mark Brown <broonie@kernel.org>
  • Loading branch information
broonie authored and spbnick committed Aug 24, 2023
1 parent 4593eed commit 84f8a33
Showing 1 changed file with 24 additions and 0 deletions.
24 changes: 24 additions & 0 deletions kcidb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,30 @@ def submit(self, data):
raise NotImplementedError
return self.mq_publisher.publish(data)

def submit_future(self, data):
"""
Submit reports without blocking for the interaction with the
database.
Args:
data: A JSON object with the report data to submit.
Must adhere to the current, or an earlier version of I/O
schema. Note that this function will not validate the
submitted data.
Returns:
A future which will return the Submission ID string.
Raises:
`NotImplementedError`, if not supplied with a project ID or an MQ
topic name at initialization time.
"""
assert io.SCHEMA.is_compatible(data)
assert LIGHT_ASSERTS or io.SCHEMA.is_valid(data)
if not self.mq_publisher:
raise NotImplementedError
return self.mq_publisher.publish_future(data)

def submit_iter(self, data_iter, done_cb=None):
"""
Submit reports returned by an iterator.
Expand Down

0 comments on commit 84f8a33

Please sign in to comment.