initial task creation #1

esheldon · 2021-04-08T11:09:20Z

No description provided.

esheldon · 2021-04-08T11:13:18Z

python/lsstdesc/pipe/task/metadetection.py

+    _DefaultName = "metadetect"
+
+    # @pipeBase.timeMethod
+    def run(self, calExpList: typing.List[lsst.afw.image.ExposureF], skyInfo: pipeBase.Struct) -> pipeBase.Struct:


How can we split this list by band, so we can coadd each band separately?

If I do calexp.get() I can see the filter but not before I load it

At what stages will the different bands come together? If we're coadding each band separately (and not making a measurement on a multiband coadd), we could invoke a task with only tracts and patches from a single band. Let me check what API exists to see the calexp info without loading it.

OK, so calexp.dataId['band'] will give the abstract band like z for the example, and calexp.dataId['physical_filter'] will output HSC-Z.

python/lsstdesc/pipe/task/version.py

erykoff · 2021-04-08T20:59:49Z

python/lsstdesc/pipe/task/metadetection.py

+        doc=("Coadded image"),
+        name="ngmixCoadd",
+        storageClass="ExposureF",
+        dimensions=("tract", "patch", "skymap", "visit", "instrument")


"visit" does not make sense for a coadd output dimension. @esheldon says he wants one output per band so this should be band here. I don't know if skymap and instrument are redundant.

You're probably right. I think skymap does include instrument but I also see some examples in the stack where both were mentioned, so I kept them both just in case.

esheldon · 2021-04-09T12:55:30Z

with the new changes I get this error

Traceback (most recent call last):
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-25-g2bc75c1+976188b8f0/bin/pipetask", line 29, in <module>
    sys.exit(main())
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-25-g2bc75c1+976188b8f0/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main
    return cli()
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/software/lsstsw/stack_20210211/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-25-g2bc75c1+976188b8f0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 102, in run
    qgraph = script.qgraph(pipelineObj=pipeline, **kwargs)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-25-g2bc75c1+976188b8f0/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 132, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/ctrl_mpexec/21.0.0-25-g2bc75c1+976188b8f0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 541, in makeGraph
    qgraph = graphBuilder.makeGraph(pipeline, collections, run, args.data_query)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/graphBuilder.py", line 841, in makeGraph
    scaffolding = _PipelineScaffolding(pipeline, registry=self.registry)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/graphBuilder.py", line 405, in __init__
    datasetTypes = PipelineDatasetTypes.fromPipeline(pipeline, registry=registry)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/pipeline.py", line 868, in fromPipeline
    thisTask = TaskDatasetTypes.fromTaskDef(taskDef, registry=registry)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/pipeline.py", line 751, in fromTaskDef
    outputs = makeDatasetTypesSet("outputs", freeze=False)
  File "/software/lsstsw/stack_20210211/stack/miniconda3-py38_4.9.2-0.4.1/Linux64/pipe_base/21.0.0-16-g0fb55c1+b60e2d390c/python/lsst/pipe/base/pipeline.py", line 742, in makeDatasetTypesSet
    raise ValueError(f"Supplied dataset type ({datasetType}) inconsistent with "
ValueError: Supplied dataset type (DatasetType('ngmixCoadd', {band, instrument, skymap, tract, patch}, ExposureF)) inconsistent with registry definition (DatasetType('ngmixCoadd', {band, instrument, skymap, physical_filter, tract, visit_s
ystem, patch, visit}, ExposureF)) for metadetect.

erykoff · 2021-04-09T13:16:45Z

Ugh. You need to erase the type in the database and I forget how to do this. It’s frustrating for debugging.

esheldon · 2021-04-09T14:00:21Z

So is this going to run the code once per band? That would not work for metadetect, we need all the bands.

It would work if we decide to do the coadds as a separate task

erykoff · 2021-04-09T14:02:48Z

The output connections will just allow you to save one file per band. The quantum dimension in the connections constructor say whether it will be run per band.

esheldon · 2021-04-09T14:03:04Z

by "all the bands" I mean riz

arunkannawadi · 2021-04-09T14:34:26Z

Ugh. You need to erase the type in the database and I forget how to do this. It’s frustrating for debugging.

It cannot be done unless the associate datasets themselves are removed unfortunately. Jim's recommendation (for now) is to rename the connection output (thereby polluting the namespace) and cleaning it up once we have a basic version working.

esheldon · 2021-04-09T15:04:10Z

I tried coaddsInCellsV1 and it seems to be working

esheldon · 2021-04-09T15:07:31Z

How do I request only bands riz ?

esheldon · 2021-04-09T15:11:07Z

Another question: the last line currently reads

return pipeBase.Struct(coadd=coaddedImage, catalog=cat)

but that is only sending on coadd image, not one for each band. How is that working?

arunkannawadi · 2021-04-09T15:34:21Z

Where you specify the dataId, include the band as well. In the example given in README, that would become tract=9615 AND patch=54 AND band in ('r','i','z')

arunkannawadi · 2021-04-09T15:38:47Z

Would you have the Struct that is returned to be modified so that it has coadded image per band? I just added one image as a placeholder and we could have several. I have to think about how to not make it specific to the riz bands alone though.

esheldon · 2021-04-09T15:52:05Z

I guess it doesn't really matter, since we are going to convert this into a coaddsInCellsTask anyway. But I didn't understand how it worked

esheldon · 2021-04-09T17:08:48Z

Where can I find the output files?

arunkannawadi · 2021-04-09T17:53:38Z

The data products will be stored in sub-folder within $REPO/u/$USER/mdetTest with a timestamp. However, it is not recommended directly access them from the folders but through the butler. Here's an example (I'll put this in README soon):

 from lsst.daf.butler import Butler
 import  os
 REPO=os.environ['REPO']
 USER=os.environ['USER']
 butler = Butler(REPO, collections=f'u/{USER}/mdetTest')
 dataId = {'tract': 9615, 'patch':54, 'skymap':'hsc_rings_v1'}
 metadetectObj = butler.get('metadetectObj', dataId=dataId)

If you ran it with the changed dimensions with band instead of skymap you'll have to modify the dataId dict.

esheldon · 2021-04-09T17:55:28Z

do you know why REPO is not defined for me?

arunkannawadi · 2021-04-09T18:02:07Z

Did you set it in your environment as given in the README example? This snippet was assuming you did that. I see on lsst-devl that you've run using /project/hsc/gen3repo/rc2v21_0_0_rc1_ssw48 as your REPO. You probably opened Python from a different terminal if you did follow the README example.

…tect-task into dev/arunkannawadi

DM uses non-standard naming conventions that flake8 flags

currently not seeding rng

arunkannawadi added 5 commits November 19, 2020 16:13

Second commit

8fc6810

Ran with HSC repo on lsst-devl machines

cf1f997

add a runQuantum method

0d182bd

Persist an image output

dc49ad7

Write an image and a SourceCatalog as output

62e2f7d

esheldon commented Apr 8, 2021

View reviewed changes

just load 10 for now, reduce memory usage

65cfa1d

arunkannawadi reviewed Apr 8, 2021

View reviewed changes

python/lsstdesc/pipe/task/version.py Outdated Show resolved Hide resolved

Remove autogenerated version.py

2e4b5d8

erykoff reviewed Apr 8, 2021

View reviewed changes

arunkannawadi added 2 commits April 8, 2021 16:39

Remove visit from output dimensions

ab5052c

Add band to output dimensions

d5dbdd0

make new connections work

e2961d1

Add in README an example to read the output

44e6ba7

esheldon added 7 commits April 27, 2021 13:14

get working with descwl_coadd coadder

47306d1

Merge branch 'dev/arunkannawadi' of github.com:LSSTDESC/descwl-metade…

7c4bb77

…tect-task into dev/arunkannawadi

add setup.cfg to tell flake8 to ignore DM names

79fab9f

DM uses non-standard naming conventions that flake8 flags

create noise image

dc03689

currently not seeding rng

get seed from command line and seed rng

690736d

docs

3b32e7b

use logger

024fc77

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial task creation #1

initial task creation #1

esheldon commented Apr 8, 2021

esheldon Apr 8, 2021

arunkannawadi Apr 8, 2021

arunkannawadi Apr 8, 2021

erykoff Apr 8, 2021

arunkannawadi Apr 8, 2021

esheldon commented Apr 9, 2021

erykoff commented Apr 9, 2021

esheldon commented Apr 9, 2021

erykoff commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021 •

edited

Loading

initial task creation #1

Are you sure you want to change the base?

initial task creation #1

Conversation

esheldon commented Apr 8, 2021

esheldon Apr 8, 2021

Choose a reason for hiding this comment

arunkannawadi Apr 8, 2021

Choose a reason for hiding this comment

arunkannawadi Apr 8, 2021

Choose a reason for hiding this comment

erykoff Apr 8, 2021

Choose a reason for hiding this comment

arunkannawadi Apr 8, 2021

Choose a reason for hiding this comment

esheldon commented Apr 9, 2021

erykoff commented Apr 9, 2021

esheldon commented Apr 9, 2021

erykoff commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021

esheldon commented Apr 9, 2021

arunkannawadi commented Apr 9, 2021 • edited Loading

arunkannawadi commented Apr 9, 2021 •

edited

Loading