Skip to content

[bugfix] Fix Slurm backend deadlock when ReFrame runs through midnight #1231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Mar 17, 2020
Merged

[bugfix] Fix Slurm backend deadlock when ReFrame runs through midnight #1231

merged 3 commits into from
Mar 17, 2020

Conversation

@teojgo
Copy link
Contributor

@teojgo teojgo commented Mar 13, 2020

  • Use the submission datetime instead of the current datetime with sacct.

Fixes #1083

Fix sacct-based backend blocking for midnight submission
3d962ec 
* Use the submission datetime instead of the current datetime with sacct.
@teojgo teojgo added this to the ReFrame sprint 20.04 milestone Mar 13, 2020
@teojgo teojgo requested review from ekouts and vkarak March 13, 2020 08:31
@teojgo teojgo self-assigned this Mar 13, 2020
@codecov-io
Copy link

codecov-io commented Mar 13, 2020 •
edited
Loading

Codecov Report

Merging #1231 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1231   +/-   ##
=======================================
  Coverage   91.92%   91.92%           
=======================================
  Files          84       84           
  Lines       12209    12209           
=======================================
  Hits        11223    11223           
  Misses        986      986
Impacted Files Coverage Δ
reframe/core/schedulers/slurm.py 53.51% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1443a7d...fdcd61c. Read the comment docs.

ekouts
ekouts approved these changes Mar 16, 2020
View reviewed changes
Copy link
Contributor

@ekouts ekouts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

vkarak
vkarak requested changes Mar 16, 2020
View reviewed changes
Copy link
Contributor

@vkarak vkarak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set the _submit_time also inside the constructor of the SlurmJobScheduler and not just in its submit() method.

vkarak
vkarak approved these changes Mar 17, 2020
View reviewed changes
Copy link
Contributor

@vkarak vkarak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@vkarak vkarak changed the title [bugfix] Fix sacct-based backend blocking for midnight submission [bugfix] Fix Slurm backend deadlock when ReFrame runs through midnight Mar 17, 2020
@vkarak vkarak merged commit 52ac63c into reframe-hpc:master Mar 17, 2020
@teojgo teojgo deleted the bugfix/sacct_midnight_submission branch May 12, 2020 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vkarak vkarak vkarak approved these changes

@ekouts ekouts ekouts approved these changes

Assignees

@teojgo teojgo

Labels

bugfix prio: normal

Projects

None yet

Milestone

ReFrame sprint 20.04

Development

Successfully merging this pull request may close these issues.

Slurm sacct-based backend will block indefinitely if a submitted job lives across midnight

4 participants

@teojgo @codecov-io @vkarak @ekouts