Skip to content

Commit

Permalink
Use a defaultdict for BlockInfo in htex scale-in (#3300)
Browse files Browse the repository at this point in the history
This is to accomodate upcoming expansion of the block_info dictionary to
include data from an additional source (the list of blocks known to the
scaling code)

This change arises from a review of PR #3232, an earlier prototype of that
expansion work.

Behaviourally, this shouldn't change anything: it changes invalid
block accesses from key errors into blocks that are infinitely idle and
completely unloaded. However there are no indexes into block_info except
in code that is adding block information, so this situation should
not arise.
  • Loading branch information
benclifford committed Mar 28, 2024
1 parent 7dea346 commit aed4256
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions parsl/executors/high_throughput/executor.py
@@ -1,4 +1,5 @@
import typing
from collections import defaultdict
from concurrent.futures import Future
import typeguard
import logging
Expand Down Expand Up @@ -730,13 +731,11 @@ class BlockInfo:
idle: float # shortest idle time of any manager in this block

managers = self.connected_managers()
block_info: Dict[str, BlockInfo] = {}
block_info: Dict[str, BlockInfo] = defaultdict(lambda: BlockInfo(tasks=0, idle=float('inf')))
for manager in managers:
if not manager['active']:
continue
b_id = manager['block_id']
if b_id not in block_info:
block_info[b_id] = BlockInfo(tasks=0, idle=float('inf'))
block_info[b_id].tasks += manager['tasks']
block_info[b_id].idle = min(block_info[b_id].idle, manager['idle_duration'])

Expand Down

0 comments on commit aed4256

Please sign in to comment.