DM-34915: Convert from bps to idds workflow in prepare. #34
Conversation
MichelleGower
force-pushed
the
tickets/DM-34915
branch
from
783723b
to
1ff331c
Compare
Codecov ReportBase: 29.93% // Head: 29.17% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #34 +/- ##
==========================================
- Coverage 29.93% 29.17% -0.76%
==========================================
Files 9 10 +1
Lines 598 449 -149
Branches 100 76 -24
==========================================
- Hits 179 131 -48
+ Misses 416 315 -101
Partials 3 3
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
MichelleGower
force-pushed
the
tickets/DM-34915
branch
from
1ff331c
to
e4a120c
Compare
python/lsst/ctrl/bps/panda/utils.py
Outdated
| os.path.join(file_distribution_uri, folder_name) | ||
| ) | ||
|
|
||
| copy_executor = concurrent.futures.ThreadPoolExecutor(max_workers=10) |
There was a problem hiding this comment.
Can we use a cfg variable or an environment to control max_workers here?
There was a problem hiding this comment.
Added default constant: PANDA_DEFAULT_MAX_COPY_WORKERS = 10, and changed code to look for maxCopyWorkers in the submit yaml.
python/lsst/ctrl/bps/panda/utils.py
Outdated
| "available": False, | ||
| } | ||
| ) | ||
| work.dependency_map.append({"name": pseudo_filename, "submitted": False, "dependencies": deps}) |
There was a problem hiding this comment.
'"submitted": False' can be removed. It's not used by iDDS.
MichelleGower
force-pushed
the
tickets/DM-34915
branch
from
e4a120c
to
7f54dda
Compare
Checklist
doc/changes