DM-31476: Clean up panda's yaml and provide one default config #50
Conversation
hsinfang
force-pushed
the
tickets/DM-31476
branch
from
5915179
to
71027b5
Compare
There was a problem hiding this comment.
I've commented on things that jumped out at me. None of those lines I commented on should cause any problems running. Presumably the example_panda_SLAC.yaml and maybe the HSC-PANDA.yaml should also be cleaned up, but that can be done later in a different ticket.
Changes approved
| includeConfigs: | ||
| - ${CTRL_BPS_DIR}/config/bps_idf.yaml | ||
|
|
||
| operator: ${USER} |
There was a problem hiding this comment.
Double check whether this is needed. If operator isn't set in the submit yaml, bps should grab the value of that environment variable right after reading the yaml file.
There was a problem hiding this comment.
The default sounds good. I removed this line.
| inCollection: HSC/calib,HSC/raw/all,refcats | ||
| dataQuery: "tract = 9615 and patch=30 and detector IN (10..11) and instrument='HSC' and skymap='hsc_rings_v1' and band in ('r')" | ||
| output: u/{operator}/pcheck_ebutler | ||
| output: "u/{operator}/{payloadName}" |
There was a problem hiding this comment.
This is the default. I don't object if you want to put it here to make it easy for someone to change it.
There was a problem hiding this comment.
I'd leave this here as I think we might want to change it in most DP0.2 runs.
| fileDistributionEndPoint: "s3://butler-us-central1-panda-dev/hsc/{payload_folder}/{uniqProcName}/" | ||
| s3_endpoint_url: "https://storage.googleapis.com" | ||
| payload_folder: payload | ||
| fileDistributionEndPoint: "s3://butler-us-central1-panda-dev/dc2/{payload_folder}/{uniqProcName}/" |
There was a problem hiding this comment.
Is this something regular users tend to change? If not it can go into the bps_idf.yaml.
There was a problem hiding this comment.
Most people won't change it but I hesitate if it should match the butler repo. I moved it to bps_idf.yaml.
| #PANDA plugin specific settings: | ||
| idds_server: "https://aipanda015.cern.ch:443/idds" | ||
| placeholderParams: ['qgraphNodeId', 'qgraphId'] | ||
| defaultPreCmdOpts: "--long-log --log-level=VERBOSE --log-file payload-log.json" |
There was a problem hiding this comment.
This is the line I commented in slack about. defaultPreCmdOpts is in every pipetask and butler command line. This means those commands that are executed during submit will also output that json file. Currently it looks like at least one command is executed while in the cwd and then at least one after switching to the run directory. This is something that probably could be changed in bps, but I'd rather not rush to get it in tonight.
There was a problem hiding this comment.
I find the side effect of having payload-log.json in cwd inconsistent with the rest of artifacts. But can live with it for now.
hsinfang
force-pushed
the
tickets/DM-31476
branch
from
71027b5
to
5219273
Compare
|
I removed HSC-PANDA.yaml but am not sure about touching example_panda_SLAC.yaml; I'm leaving it to somebody who can test at SLAC's panda. |
hsinfang
force-pushed
the
tickets/DM-31476
branch
2 times, most recently
from
80c0cd4
to
589024b
Compare
hsinfang
force-pushed
the
tickets/DM-31476
branch
from
589024b
to
dc411c3
Compare
hsinfang
force-pushed
the
tickets/DM-31476
branch
from
dc411c3
to
c44270e
Compare
|
@yesw2000 Don't those follow the defaults @MichelleGower set? |
Checklist
doc/changes