DM-22277: Gen3 object tables #490
Conversation
| catalogs[band] = {} | ||
| catalogs[band]['meas'] = measDict[band]['meas'] | ||
| catalogs[band]['forced_src'] = forcedSourceDict[band]['forced_src'] | ||
| catalogs[band]['ref'] = inputs['inputCatalogRef'] |
There was a problem hiding this comment.
Would this read a bit better with:
catalogs[band] = {'meas': measDict[band]['meas'],
'forced_src': forcedSourceDict[band]['forced_src'],
'ref': inputs['inputCatalogRef']}
|
|
||
| dataId = butlerQC.quantum.dataId | ||
| parq = self.run(catalogs=catalogs, tract=dataId['tract'], patch=dataId['patch']) | ||
| outputs = pipeBase.Struct(outputCatalog=parq.toDataFrame()) |
There was a problem hiding this comment.
I don't know if there's a big overhead to converting to and from parquet and dataFrames, but I think it's odd that the last thing the run method does is convert the dataframe to parquet, and the first thing this code does is convert it back to a dataframe. I know that's what the gen2 code expects, but in terms of planning for gen2 removal, I think that run should return a dataframe and runDataRef could then parquet-ify the table?
There was a problem hiding this comment.
I agree. ParquetTable will disappear with along with Gen2.
Was avoiding API changes on this one, but if the reviewer advocates for API changes, well then I guess I have to :)
| dimensions=("tract", "patch", "band", "skymap"), | ||
| storageClass="SourceCatalog", | ||
| name="{coaddName}Coadd_meas", | ||
| multiple=True |
There was a problem hiding this comment.
I guess there's no advantage to these being deferLoad=True because we need to hold them all in memory at the same time anyway?
There was a problem hiding this comment.
That's right. We're vertically concatenating them.
yalsayyad
force-pushed
the
tickets/DM-22277
branch
4 times, most recently
from
ce9e814
to
789e1b7
Compare
yalsayyad
force-pushed
the
tickets/DM-22277
branch
from
789e1b7
to
a65dc2e
Compare
No description provided.