Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rework conveyor heartbeat handling #5252

Closed
rcarpa opened this issue Feb 18, 2022 · 0 comments
Closed

rework conveyor heartbeat handling #5252

rcarpa opened this issue Feb 18, 2022 · 0 comments
Assignees
@rcarpa
Labels
enhancement Transfers
Milestone
1.27.9

Comments

@rcarpa
Copy link
Contributor

rcarpa commented Feb 18, 2022 •
edited

Motivation

Right now heartbeats are updated in the database at each iteration (each activity for multi-activity daemons). This forces us to use a big "older_than" value to avoid the race condition when a big bulk size and slow access to transfertool makes hearbeats expire.

At the same time, this puts strain on database with un-needed heartbeat updates when the daemons are lightly loaded.

Modification

An idea for improvement is to perform a hearbeat_handler.live() more frequently (each submission, for example), but modify live() to only perform a database update if enough time had passed from last update. The frequency of updates and older_than should probably defined as a function of sleep_time
@rcarpa rcarpa self-assigned this Feb 18, 2022
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 1, 2022
@rcarpa
Transfers: rework heartbeat handling. Closes rucio#5252
6ead54d 
Right now heartbeats are updated in the database at each iteration
(each activity for multi-activity daemons). This forces us to use
a big "older_than" value to avoid the race condition when a big bulk
size and slow access to transfertool makes hearbeats expire.
At the same time, this puts strain on database with un-needed heartbeat
updates when the daemons are lightly loaded.

Rework this behavior to perform hearbeat_handler.live() more frequently
(each submission, for example), but modify live() to only perform a
database update if enough time has passed from last update.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 1, 2022
@rcarpa
Transfers: rework heartbeat handling. Closes rucio#5252
19f2eba 
Right now heartbeats are updated in the database at each iteration
(each activity for multi-activity daemons). This forces us to use
a big "older_than" value to avoid the race condition when a big bulk
size and slow access to transfertool makes hearbeats expire.
At the same time, this puts strain on database with un-needed heartbeat
updates when the daemons are lightly loaded.

Rework this behavior to perform hearbeat_handler.live() more frequently
(each submission, for example), but modify live() to only perform a
database update if enough time has passed from last update.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 8, 2022
@rcarpa
Transfers: rework heartbeat handling. Closes rucio#5252
9aa6acc 
Right now heartbeats are updated in the database at each iteration
(each activity for multi-activity daemons). This forces us to use
a big "older_than" value to avoid the race condition when a big bulk
size and slow access to transfertool makes hearbeats expire.
At the same time, this puts strain on database with un-needed heartbeat
updates when the daemons are lightly loaded.

Rework this behavior to perform hearbeat_handler.live() more frequently
(each submission, for example), but modify live() to only perform a
database update if enough time has passed from last update.
@bari12 bari12 closed this as completed in ded5e90 Mar 14, 2022
bari12 added a commit that referenced this issue Mar 14, 2022
@bari12
Merge pull request #5306 from rcarpa/patch-5252-conveyor_heartbeat_ha…
59cf849 
…ndling

Transfers: rework heartbeat handling. Closes #5252
bari12 pushed a commit that referenced this issue Mar 14, 2022
@rcarpa @bari12
Transfers: rework heartbeat handling. Closes #5252
4e19a50 
Right now heartbeats are updated in the database at each iteration
(each activity for multi-activity daemons). This forces us to use
a big "older_than" value to avoid the race condition when a big bulk
size and slow access to transfertool makes hearbeats expire.
At the same time, this puts strain on database with un-needed heartbeat
updates when the daemons are lightly loaded.

Rework this behavior to perform hearbeat_handler.live() more frequently
(each submission, for example), but modify live() to only perform a
database update if enough time has passed from last update.
@bari12 bari12 added this to the 1.27.9 milestone Mar 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcarpa rcarpa

Labels
enhancement Transfers
Projects
None yet
Milestone
1.27.9
Development

No branches or pull requests
2 participants
@bari12 @rcarpa