Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-32201: Fix execution butler with HTCondor plugin bug #58

Merged
merged 1 commit into from
Oct 12, 2021
Merged

DM-32201: Fix execution butler with HTCondor plugin bug #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/changes/DM-32201.bugfix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix execution butler with HTCondor plugin bug when output collection has period.
1 change: 1 addition & 0 deletions doc/changes/DM-32201.misc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Change default for bpsUseShared to True.
2 changes: 1 addition & 1 deletion python/lsst/ctrl/bps/etc/bps_defaults.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ requestMemory: 2048
requestCpus: 1

wmsServiceClass: lsst.ctrl.bps.wms.htcondor.htcondor_service.HTCondorService
bpsUseShared: True
clusterAlgorithm: lsst.ctrl.bps.quantum_clustering_funcs.single_quantum_clustering

# Templates for bps filenames
Expand All @@ -74,7 +75,6 @@ saveClusteredQgraph: False

# Temporary backward-compatibility settings
useLazyCommands: True
bpsUseShared: False

executionButler:
whenCreate: "SUBMIT"
Expand Down
11 changes: 6 additions & 5 deletions python/lsst/ctrl/bps/wms/htcondor/htcondor_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -611,7 +611,7 @@ def _replace_file_vars(config, arguments, workflow, gwjob):
# Taking advantage of inside knowledge. Not future-proof.
# Temporary fix until have job wrapper that pulls files
# within job.
if gwfile.name == "butlerConfig" and not Path(gwfile.src_uri).suffix:
if gwfile.name == "butlerConfig" and Path(gwfile.src_uri).suffix != ".yaml":
uri = "butler.yaml"
else:
uri = os.path.basename(gwfile.src_uri)
Expand Down Expand Up @@ -703,10 +703,11 @@ def _handle_job_inputs(generic_workflow: GenericWorkflow, job_name: str, use_sha
# knowledge for temporary fix until have job wrapper that pulls
# files within job.
if gwf_file.name == "butlerConfig":
# The execution butler directory doesn't normally exist
# until the submit phase so checking for suffix instead
# of using is_dir().
if uri.suffix: # Single file, so just copy.
# The execution butler directory doesn't normally exist until
# the submit phase so checking for suffix instead of using
# is_dir(). If other non-yaml file exists they would have a
# different gwf_file.name.
if uri.suffix == ".yaml": # Single file, so just copy.
inputs.append(f"file://{uri}")
else:
inputs.append(f"file://{uri / 'butler.yaml'}")
Expand Down