Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement a per-node request queue for gossiping #1149

Closed
AndrejMitrovic opened this issue Aug 27, 2020 · 1 comment
Closed

Implement a per-node request queue for gossiping #1149

AndrejMitrovic opened this issue Aug 27, 2020 · 1 comment
Assignees
@AndrejMitrovic
Labels
C. Network Communication An issue which is related to network communication Story-Points:1 This takes 0.5 day to complete type-feature An addition to the system introducing new functionalities
Milestone
2. Validator

Comments

@AndrejMitrovic
Copy link
Contributor

#1145 revealed that there are numerous fibers spawned for every single gossiping requests.

That means in the example of 10 nodes gossiping 8 transactions would lead to the creation of 80 fibers.

The ManyValidators test would spawn over 1600 fibers over the course of the run of the test.
@AndrejMitrovic AndrejMitrovic added Story-Points:1 This takes 0.5 day to complete type-feature An addition to the system introducing new functionalities labels Aug 27, 2020
@AndrejMitrovic AndrejMitrovic added this to the 2. Validator milestone Aug 27, 2020
@AndrejMitrovic AndrejMitrovic self-assigned this Aug 27, 2020
@AndrejMitrovic AndrejMitrovic mentioned this issue Aug 27, 2020
Closed
AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this issue Aug 27, 2020
@AndrejMitrovic
Implement a per-node request queue
3251414 
Previously a new fiber was spawned for every new gossiping
request to every node.

For example, to gossip 8 transactions to 10 different nodes
the total number of active fibers spawned would be 80.

Now each connected node has its own request queue for
all gossiping requests.

In the preliminary design the requests are served
first-in first-out - a queue. Later the design can be
complemented with request priority support and prunning
of stale requests based on their timestamps.

Fixes bosagora#1149
AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this issue Aug 27, 2020
@AndrejMitrovic
Implement a per-node request queue
f468bc8 
Previously a new fiber was spawned for every new gossiping
request to every node.

For example, to gossip 8 transactions to 10 different nodes
the total number of active fibers spawned would be 80.

Now each connected node has its own request queue for
all gossiping requests.

In the preliminary design the requests are served
first-in first-out - a queue. Later the design can be
complemented with request priority support and prunning
of stale requests based on their timestamps.

Fixes bosagora#1149
AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this issue Aug 27, 2020
@AndrejMitrovic
Implement a per-node request queue
ab2587d 
Previously a new fiber was spawned for every new gossiping
request to every node.

For example, to gossip 8 transactions to 10 different nodes
the total number of active fibers spawned would be 80.

Now each connected node has its own request queue for
all gossiping requests.

In the preliminary design the requests are served
first-in first-out - a queue. Later the design can be
complemented with request priority support and prunning
of stale requests based on their timestamps.

Fixes bosagora#1149
AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this issue Aug 28, 2020
@AndrejMitrovic
Implement a per-node request queue
64a8a1b 
Previously a new fiber was spawned for every new gossiping
request to every node.

For example, to gossip 8 transactions to 10 different nodes
the total number of active fibers spawned would be 80.

Now each connected node has its own request queue for
all gossiping requests.

In the preliminary design the requests are served
first-in first-out - a queue. Later the design can be
complemented with request priority support and prunning
of stale requests based on their timestamps.

Fixes bosagora#1149
@AndrejMitrovic
Copy link
Contributor Author

This is solving the problem at the wrong level. We should have a fiber pool, with potentially a more intelligent design (e.g. don't allow a single node to consume all fibers). But it should be solved in vibe.d & localrest.

@bpalaggi bpalaggi added the C. Network Communication An issue which is related to network communication label Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndrejMitrovic AndrejMitrovic

Labels
C. Network Communication An issue which is related to network communication Story-Points:1 This takes 0.5 day to complete type-feature An addition to the system introducing new functionalities
Projects
None yet
Milestone
2. Validator
Development

Successfully merging a pull request may close this issue.

[RFC] Implement a per-node request queue for gossiping AndrejMitrovic/agora
2 participants
@AndrejMitrovic @bpalaggi