Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use empty string for all-inclusive pod-label-selector #113

Merged
merged 2 commits into from
Mar 16, 2022

Conversation

odp
Copy link
Collaborator

@odp odp commented Mar 16, 2022

Two successive non-preemptible AdaptDL jobs received same allocation by the single job allocation optimization introduced by #66

INFO:__main__:Patch AdaptDLJob  namespace/a-b-c-dzjr4: {'status': {'allocation': ['X.Y.0.232']}}
INFO:__main__:Patch AdaptDLJob  namespace/a-b-c-qczpw: {'status': {'allocation': ['X.Y.0.232']}}

The jobs being non-preemptible, this later crashes in the Pollux optimizer (because two jobs have same allocation) when a full allocation cycle starts.

    await self._optimize_all()
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/allocator.py", line 126, in _optimize_all
    allocations = self._allocate(jobs, nodes, prev_allocations,
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/allocator.py", line 259, in _allocate
    allocations, desired_nodes = self._policy.optimize(
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/policy/pollux.py", line 191, in optimize
    problem = Problem(list(jobs.values()), list(nodes.values()) +
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/policy/pollux.py", line 275, in __init__
    self._max_replicas[j, n] = min(
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/policy/pollux.py", line 276, in <genexpr>
    self._get_avail_resource(
  File "/usr/local/lib/python3.8/site-packages/adaptdl_sched/policy/pollux.py", line 299, in _get_avail_resource
    assert resource >= 0
AssertionError

The issue happens because a None pod label qualifier is used to include all pods instead of an empty string "". This causes the code to always assume all nodes are available (because None returns no pods) and it assigns the first one to every incoming job. This bug also affects preemptible jobs but without any visible implications because the allocation gets fixed by a later full allocation cycle.

With the fix, every new job gets the next available node.

@odp odp requested a review from rmfan March 16, 2022 16:54
@odp odp assigned odp and unassigned odp Mar 16, 2022
@odp odp merged commit 66bc1ea into petuum:master Mar 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants