Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sched crash when array job and reservation is submitted #1109

Merged
merged 2 commits into from May 1, 2019

Conversation

Projects
None yet
3 participants
@arungrover
Copy link
Contributor

commented May 1, 2019

Describe Bug or Feature

Scheduler crashes when it encounters an array job and a reservation in the same scheduling cycle.

Describe Your Change

As part of speeding up scheduler, we added a way to find a job/reservation based on their indexes in the all_resresv array.
But after these indexes are created, all_resresv array gets sorted to bring timed events in front in order to create a calendar. This sort messes up the indexes and causes the crash. While the sort is needed there is no reason to have all_resresv sorted.
As part of the fix, create_events code which creates the calendar will create a local copy of the array and sort that local copy instead of sorting all_resresv array.

Link to Design Doc

None

Attach Test Logs or Output

sched_dump_valgrind.txt
test_out.txt

*/
all = sinfo->all_resresv;
all_resresv_len = count_array((void **)sinfo->all_resresv);
all_resresv_copy = (resource_resv **)malloc((all_resresv_len + 1) * sizeof(resource_resv *));

This comment has been minimized.

Copy link
@bhroam

bhroam May 1, 2019

Contributor

No need to cast the return value of malloc()

return 0;
}

This comment has been minimized.

Copy link
@bhroam

bhroam May 1, 2019

Contributor

I think this is actually a bug. If we return here, we'll leak events. Instead of returning, errflag++ and break (that is what happens above). That will cause the code below to free everything and return.

r = Reservation()
a = {'Resource_List.select': '1:ncpus=1'}
a = {'resources_available.ncpus': 4}
self.server.manager(MGR_CMD_SET, NODE, a, id=self.mom.shortname)

This comment has been minimized.

Copy link
@bhroam

bhroam May 1, 2019

Contributor

If you do this, I think you need to add the skip on cpuset decorator to this test. You can't modify the resources on a cpuset machine to differ from what the mom reported.

r = Reservation(TEST_USER)
now = int(time.time())
a = {'Resource_List.select': '1:ncpus=4',
'reserve_start': now + 5,

This comment has been minimized.

Copy link
@bhroam

bhroam May 1, 2019

Contributor

5s might be too short on slow machines. I'd use 10s.

jid1 = self.server.submit(j1)

a = {'Resource_List.select': '1:ncpus=1',
ATTR_q: rid.split('.')[0], ATTR_J: '1-2'}

This comment has been minimized.

Copy link
@bhroam

bhroam May 1, 2019

Contributor

Since you're splitting the rid twice, why not do it once and store it into a local variable?

@arungrover

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2019

Thanks for quick review @bhroam. I've addressed your review comments.

@bhroam

bhroam approved these changes May 1, 2019

Copy link
Contributor

left a comment

Looks good. I sign off.

@lisa-altair
Copy link
Contributor

left a comment

LGTM

@bhroam bhroam merged commit af90831 into PBSPro:master May 1, 2019

3 checks passed

Travis CI - Pull Request Build Passed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.