-
Notifications
You must be signed in to change notification settings - Fork 21
[Processing] Simplify incoming message functions #227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Processing] Simplify incoming message functions #227
Conversation
|
If all jobs are "big batches", it means they are processed asynchronously... It might get a big delay on p2p-pubsub/ipfs-pubsub incoming messages, leading to non-ui-proof latency. Current applications are expecting actions to be done sub-500ms after a use clicks on something and signs the message. If we can still process them fast enough it's ok, if not we need to rethink this a bit. |
|
@moshemalawach I introduced a bug here indeed, I need to change the logic in the IPFS pubsub part to apply the operations from
In any case I'm keeping the performance aspect in mind, we already have integration tests on the aleph_client repository that validate the time it takes for a message to propagate on the network. |
b70781c to
db4ec19
Compare
Improved the `incoming` function to simplify it and make it more generic. Removed the `bulk_operation` flag as we always wish for bulk DB operations. The function now returns a status + a list of DB operations. DB operations are a new class that contains a Mongo operation and the name of the collection to target. This avoids passing lists as arguments and modifying them in place. The loop to await individual message tasks now waits for all tasks to complete instead of failing on the first operation. This fixes an issue where, if an exception occurs soon enough in the batch, the output of still-running tasks could be lost. Added type hints.
db4ec19 to
d65eacf
Compare
|
Ok, great to hear :) |
Improved the `incoming` function to simplify it and make it more generic. Removed the `bulk_operation` flag as we always wish for bulk DB operations. The function now returns a status + a list of DB operations. DB operations are a new class that contains a Mongo operation and the name of the collection to target. This avoids passing lists as arguments and modifying them in place. The loop to await individual message tasks now waits for all tasks to complete instead of failing on the first operation. This fixes an issue where, if an exception occurs soon enough in the batch, the output of still-running tasks could be lost. Added type hints.
Improved the
incomingfunction to simplify it and make it moregeneric.
Removed the
bulk_operationflag as we always wish for bulk DBoperations. The function now returns a status + a list of DB
operations. DB operations are a new class that contains a Mongo
operation and the name of the collection to target.
This avoids passing lists as arguments and modifying them in place.
The loop to await individual message tasks now waits for all tasks
to complete instead of failing on the first operation.
This fixes an issue where, if an exception occurs soon enough in
the batch, the output of still-running tasks could be lost.
Added type hints.