DM-32048: Change tests to use the new pipetask API. #73
Conversation
tests/fgcmcalTestBase.py
Outdated
| configFiles : `list` [`str`], optional | ||
| List of config files to use (with "-C"). | ||
| List of config files to use. Each string will be of the form |
There was a problem hiding this comment.
I'd maybe replace "to use" with "to load, in order".
Similarly for the configOptions below.
tests/fgcmcalTestBase.py
Outdated
| pipeline.addConfigFile(taskName, fileName) | ||
| for configOption in configOptions: | ||
| taskName, others = configOption.split(':') | ||
| option, value = others.split('=') |
There was a problem hiding this comment.
I'm not keen on the string manipulation here: I feel like these could be dictionaries instead, so you don't have to muck with strings. I found that to be an improvement in jointcal: just pass a dict of the config field:value you want to apply. But you're also specifying different tasks, so maybe it's not as nice? Would a nested dict be bad?
There was a problem hiding this comment.
I don't love nested dicts, but they do the trick in this case.
tests/fgcmcalTestBase.py
Outdated
| @@ -92,16 +88,18 @@ def _runPipeline(self, repo, pipelineFile, queryString=None, | |||
| Pipeline definition file. | |||
| queryString : `str`, optional | |||
| String to use for "-d" data query. | |||
There was a problem hiding this comment.
This should probably be "where query that defines the data to use" or something like that, since -d is no longer relevant.
tests/fgcmcalTestBase.py
Outdated
| # We are reusing the outputCollection so we can't specify the input | ||
| inputCollections = None | ||
| # We need to know the correct input collections to be able to | ||
| # reuse the output collection with the pipetask API. |
There was a problem hiding this comment.
I think I'm having a little trouble parsing this comment. If I didn't know about the discussion you and Jim had on slack, I'd be a bit lost. Not sure how to reword it though.
There was a problem hiding this comment.
How about:
# In these tests we are running the fit cycle task multiple
# times into the same output collection. This code allows
# us to find the correct chained input collections to use
# so that we can both read from previous runs in the output
# collection and write to a new run in the output collection.
# Note that this behavior is handled automatically by the
# pipetask command-line interface, but not by the python
# API.
tests/fgcmcalTestBase.py
Outdated
| @@ -608,7 +602,7 @@ def _testFgcmMultiFit(self, instName, testName, queryString, visits, zpOffsets): | |||
| 'pipelines', | |||
| f'fgcmFullPipeline{instCamel}.yaml'), | |||
| configFiles=configFiles, | |||
| inputCollections=f'{instName}/{testName}/lut,refcats/gen2', | |||
| inputCollections=[f'{instName}/{testName}/lut', 'refcats/gen2'], | |||
There was a problem hiding this comment.
You did the same thing above, but put a newline instead of a space before refcats. I'm not sure it matters, but the newline was maybe a bit more easy to parse.
erykoff
force-pushed
the
tickets/DM-32048
branch
from
a211b69
to
7df4030
Compare
No description provided.