Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specification of metadata to be fetched for Websocket Queries #234

Merged
merged 1 commit into from
Dec 3, 2021

Conversation

silvanheller
Copy link
Member

Introduces the MetadataAccessSpecification, of which a list can be handed to Cineast. * for either domain or keys mean all elements should be fetched. For backwards-compatibility, null is treated as *, *, which means all metadata is fetched.

Implementation is unit-tested.

Closes #166

As a simple comparison, on the LSC 21 dataset, this reduces metadata fetch time for a tag query sushi_bar (5k results) from approx. 2.5 to 2 seconds:

(old)

[11-18 19:19:04.449][DEBUG][metadata-load-write-1db170ea-e7db-4117-b49c-d1bd4afc0a4a] AbstractMetadataReader - Performed metadata lookup for 6671 ids in 2439 ms. 92879 results.
[11-18 19:19:04.514][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 44 ms
[11-18 19:19:04.656][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 187 ms

(new)

[11-18 19:15:10.310][DEBUG][metadata-load-write-c6917953-51e3-400b-b06a-ea492ce166ba] AbstractMetadataReader - Performed metadata lookup for 6671 ids in 1970 ms. 39914 results.
[11-18 19:15:10.372][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 44 ms
[11-18 19:15:10.435][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 108 ms

For larger result-sets (i.e. hitting the config limit, currently 20k) it goes from 9.4 to 6 seconds + there are (obviously) gains in serialization speed.

(old)

[11-18 19:20:46.498][DEBUG][metadata-load-write-5066239b-fbe4-4bcb-9af6-033e51b4249c] AbstractMetadataReader - Performed metadata lookup for 20000 ids in 9473 ms. 315522 results.
[11-18 19:20:46.600][TRACE][metadata-ws-write-queryId3] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 8 ms
[11-18 19:20:46.620][TRACE][metadata-ws-write-queryId3] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId3: Successfully wrote message QR_METADATA_S in 28 ms
[11-18 19:20:46.664][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 72 ms
[11-18 19:20:46.664][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 72 ms
[11-18 19:20:46.823][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 231 ms
[11-18 19:20:46.935][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId2: Successfully wrote message QR_METADATA_S in 343 ms
[11-18 19:20:47.054][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 463 ms
[11-18 19:20:47.170][TRACE][metadata-ws-write-queryId2] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId1: Successfully wrote message QR_METADATA_S in 578 ms

(new)

[11-18 19:23:36.383][DEBUG][metadata-load-write-6437f6c0-f2bc-4f2f-8653-45a3615d17ab] AbstractMetadataReader - Performed metadata lookup for 20000 ids in 6278 ms. 134793 results.
[11-18 19:23:36.486][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 50 ms
[11-18 19:23:36.534][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler - Serialization for QR_METADATA_S in 98 ms
[11-18 19:23:36.541][TRACE][metadata-ws-write-queryId1] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId1: Successfully wrote message QR_METADATA_S in 105 ms
[11-18 19:23:36.674][TRACE][metadata-ws-write-queryId0] AbstractWebsocketMessageHandler$1 - metadata-ws-write-queryId0: Successfully wrote message QR_METADATA_S in 238 ms

@sauterl
Copy link
Collaborator

sauterl commented Nov 18, 2021

Generally speaking I like this PR very much, there are a few minor engineering points I think are to be improved.

Copy link
Collaborator

@sauterl sauterl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks @silvanheller for the PR

@silvanheller silvanheller merged commit 6037f63 into dev Dec 3, 2021
@silvanheller silvanheller deleted the feature/md-spec-166 branch December 3, 2021 08:22
silvanheller added a commit that referenced this pull request Jun 10, 2022
Allow specification of metadata to be fetched for Websocket Queries

Former-commit-id: 6037f63
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants