implement data quality / bad pixels

[zonca]

stpipe has a dq_init step that initializes the masks/flags for bad pixels, see https://jwst-pipeline.readthedocs.io/en/latest/jwst/dq_init/description.html

the structure of the fits file is:

PIXELDQ is common to all readouts, GROUPDQ instead has the same shape as the science extension.

dq_init connects to the CRDS and gets a static mask based on file metadata and prepares the PIXELDQ extension based on it, then initializes a zero array for the GROUPDQ.

See the implementation at https://github.com/spacetelescope/jwst/tree/master/jwst/dq_init

The definition of all flags used by JWST is at https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags

[zonca]

@arunsurya77 can you prepare an example reference mask I can use for testing? do we have our own definition for flags or we can borrow the one from JWST?

[arunsurya77]

Hi Andrea, Sorry for the delay. I talked to Shelley about this last group meeting, since this applies at the detector level ROP Ramps and not the Level-2 image models I was not sure how to proceed. She suggested we talk about it sometime. I can have another look today and update you. For the flags, I think most of what is in JWST can be reused. Let me check once more today and get back to you on this. Cheers Arun

…

On Tue, 2 Apr 2019 at 18:50, Andrea Zonca ***@***.***> wrote: @arunsurya77 <https://github.com/arunsurya77> can you prepare an example reference mask I can use for testing? do we have our own definition for flags or we can borrow the one from JWST? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AoO_ri4BL2JEqvq5W5QbS3nPICd2vPPVks5vdAjVgaJpZM4cZbps> .

[arunsurya77]

Hi Andrea, I am attaching a mask reference file used for the static badpixel mapping. I just have 100 random hot pixels and 100 dead pixels. The flag values are per the following. It follows the MaskModel schema used for the initdq steps. mask_test.fits <https://drive.google.com/file/d/196Sj89ziqH8KXxozaw2wrgzjqNqVhqWh/view?usp=drive_web> https://jwst-pipeline.readthedocs.io/en/latest/jwst/references_general/references_general.html#data-quality-flags Cheers Arun

…

On Wed, 3 Apr 2019 at 09:57, Arun Surya ***@***.***> wrote: Hi Andrea, Sorry for the delay. I talked to Shelley about this last group meeting, since this applies at the detector level ROP Ramps and not the Level-2 image models I was not sure how to proceed. She suggested we talk about it sometime. I can have another look today and update you. For the flags, I think most of what is in JWST can be reused. Let me check once more today and get back to you on this. Cheers Arun On Tue, 2 Apr 2019 at 18:50, Andrea Zonca ***@***.***> wrote: > @arunsurya77 <https://github.com/arunsurya77> can you prepare an example > reference mask I can use for testing? do we have our own definition for > flags or we can borrow the one from JWST? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#4 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AoO_ri4BL2JEqvq5W5QbS3nPICd2vPPVks5vdAjVgaJpZM4cZbps> > . >

[zonca]

thanks @arunsurya77 I need to create a RampModel to test the bad pixel map,
have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work.

[arunsurya77]

Hi Andrea, I had created a RampModel of 10x10 pixel with in x-y. So I think we will need 4096x4096 window size to test bad pixel stuff. I will create it and send it to you. I have a script to make those. Cheers Arun

…

On Wed, 3 Apr 2019 at 19:54, Andrea Zonca ***@***.***> wrote: thanks @arunsurya77 <https://github.com/arunsurya77> I need to create a RampModel to test the bad pixel map, have you done that already for your work on the ROP? in any case it would be very useful if you can create a pull request even with a temporary version of your code, so I don't risk of duplicating your work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps> .

[zonca]

thanks, can you please point to the location on Galactica?

…

On Thu, Apr 4, 2019 at 9:55 AM Arun Surya ***@***.***> wrote: Hi Andrea, I had created a RampModel of 10x10 pixel with in x-y. So I think we will need 4096x4096 window size to test bad pixel stuff. I will create it and send it to you. I have a script to make those. Cheers Arun On Wed, 3 Apr 2019 at 19:54, Andrea Zonca ***@***.***> wrote: > thanks @arunsurya77 <https://github.com/arunsurya77> I need to create a > RampModel to test the bad pixel map, > have you done that already for your work on the ROP? in any case it would > be very useful if you can create a pull request even with a temporary > version of your code, so I don't risk of duplicating your work. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#4 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXYcvGZeDLO0HFhvudJ3AuYHHzBYdGIks5vdi6CgaJpZM4cZbps> .

[arunsurya77]

Hi, It is saved in /data/group/packages/idlpro/iris/readout_iris/test_ramp.fits Cheers Arun

…

On Thu, 4 Apr 2019 at 10:12, Andrea Zonca ***@***.***> wrote: thanks, can you please point to the location on Galactica? On Thu, Apr 4, 2019 at 9:55 AM Arun Surya ***@***.***> wrote: > Hi Andrea, > I had created a RampModel of 10x10 pixel with in x-y. So I think we will > need 4096x4096 window size to test bad pixel stuff. I will create it and > send it to you. I have a script to make those. > > Cheers > Arun > > On Wed, 3 Apr 2019 at 19:54, Andrea Zonca ***@***.***> > wrote: > > > thanks @arunsurya77 <https://github.com/arunsurya77> I need to create a > > RampModel to test the bad pixel map, > > have you done that already for your work on the ROP? in any case it would > > be very useful if you can create a pull request even with a temporary > > version of your code, so I don't risk of duplicating your work. > > > > — > > You are receiving this because you were mentioned. > > Reply to this email directly, view it on GitHub > > < #4 (comment) > >, > > or mute the thread > > < > https://github.com/notifications/unsubscribe-auth/AoO_roC8DG5cfFaK8_MBnciq0GFhFE5Lks5vdWlbgaJpZM4cZbps > > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#4 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAXYcvGZeDLO0HFhvudJ3AuYHHzBYdGIks5vdi6CgaJpZM4cZbps > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AoO_ruOuyXFOlOXk9Gld98CkfxsF5Ujcks5vdjKLgaJpZM4cZbps> .

[zonca]

thanks @arunsurya77, I added a schema for
a TMT ramp: zonca/jwst@48ec2e4

[zonca]

see the attached file where I changed the metadata so that it opens up as a TMTRampModel:

ramp.zip

[zonca]

ok, I have ported the dq_init code to iris_pipeline, see example usage: https://gist.github.com/zonca/e15620ff5d26652bc201b180ec00cdce

documentation is available in https://iris-pipeline.readthedocs.io/en/latest/algorithms.html#remove-detector-artifacts