Sched crash when array job and reservation is submitted #1109
Conversation
src/scheduler/simulate.c
Outdated
| */ | ||
| all = sinfo->all_resresv; | ||
| all_resresv_len = count_array((void **)sinfo->all_resresv); | ||
| all_resresv_copy = (resource_resv **)malloc((all_resresv_len + 1) * sizeof(resource_resv *)); |
There was a problem hiding this comment.
No need to cast the return value of malloc()
| return 0; | ||
| } |
There was a problem hiding this comment.
I think this is actually a bug. If we return here, we'll leak events. Instead of returning, errflag++ and break (that is what happens above). That will cause the code below to free everything and return.
| r = Reservation() | ||
| a = {'Resource_List.select': '1:ncpus=1'} | ||
| a = {'resources_available.ncpus': 4} | ||
| self.server.manager(MGR_CMD_SET, NODE, a, id=self.mom.shortname) |
There was a problem hiding this comment.
If you do this, I think you need to add the skip on cpuset decorator to this test. You can't modify the resources on a cpuset machine to differ from what the mom reported.
test/tests/pbs_smoketest.py
Outdated
| r = Reservation(TEST_USER) | ||
| now = int(time.time()) | ||
| a = {'Resource_List.select': '1:ncpus=4', | ||
| 'reserve_start': now + 5, |
There was a problem hiding this comment.
5s might be too short on slow machines. I'd use 10s.
test/tests/pbs_smoketest.py
Outdated
| jid1 = self.server.submit(j1) | ||
|
|
||
| a = {'Resource_List.select': '1:ncpus=1', | ||
| ATTR_q: rid.split('.')[0], ATTR_J: '1-2'} |
There was a problem hiding this comment.
Since you're splitting the rid twice, why not do it once and store it into a local variable?
|
Thanks for quick review @bhroam. I've addressed your review comments. |
Describe Bug or Feature
Scheduler crashes when it encounters an array job and a reservation in the same scheduling cycle.
Describe Your Change
As part of speeding up scheduler, we added a way to find a job/reservation based on their indexes in the all_resresv array.
But after these indexes are created, all_resresv array gets sorted to bring timed events in front in order to create a calendar. This sort messes up the indexes and causes the crash. While the sort is needed there is no reason to have all_resresv sorted.
As part of the fix, create_events code which creates the calendar will create a local copy of the array and sort that local copy instead of sorting all_resresv array.
Link to Design Doc
None
Attach Test Logs or Output
sched_dump_valgrind.txt
test_out.txt