Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved streaming experience #409

Merged
merged 3 commits into from
Mar 8, 2024

Conversation

massi-ang
Copy link
Collaborator

@massi-ang massi-ang commented Mar 8, 2024

Issue #, if available:

Description of changes:
When using streaming mode, single tokens are published to SQS and then processed by a lambda function in micro-batches of 10 items.
Each message is then published as a mutation to AppSync and triggers a refresh of the UI. Tokens are reordered client side before the resulting text is displayed.

We have noticed that the text is often scrambled and upon each refresh it gets reordered. Upon investigation this is due to the fact that records in SQS batches are not in order and Lambda Powertools batch processor processes the records in the same order as they are received. This causes a lot of reordering client side since token are received individually.

Eg:

Batch: [friend!, Hello, my]

Client:
friend -> "friend!"

Client:
Hello -> "Hello friend!"

Client:
my -> "Hello my friend!"

To improve the experience we sort the records in he batch based on their sequenceNumber before passing them to the Lambda Powertools batch processor.
In the same example as above the batch will be reordered as [Hello, my, friend!] and the tokens sent in the correct order.
This is not an absolute ordering but improves the experience significantly already.

With this implementation we observed that client side the maximum number of token reordering is massively reduced from an average of 6-10 per received token to an average of 0-1.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@bigadsoleiman bigadsoleiman merged commit 40667ec into aws-samples:main Mar 8, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

2 participants