Create commands.txt

[TomGlanzman]

This is the command/physics override file for running PhoSim for DC2.

[johnrpeterson]

looks fine to me. i'm not sure there was any discussion on "contamination mode" though. in DC1 it was turned off, because of blockiness of the implementation which wouldn't be there for DC2. but its fine either way.

[TomGlanzman]

I've attempted to associate @cwwalter early best guess for the phosim commands/overrides with those published in bitbucket. Chris' guess is:

BF
Tree rings
Cosmic Rays
Bleeding

will be on.

Fringing
Saturation/non-linearity
Crosstalk and hot pixels and colums etc

likely won't be on but might be.

Some of my comments reference phosim runs using the current provisional commands.txt content in this PR.

BF = Brighter/Fatter ==> This one is understood: "chargesharing 1"

Tree Rings ==> do not see an obvious association with published overrides (possibly "impurityvariation"?)

Cosmic Rays ==> appear to be on by default, as a recent log files indicates their presence. There are two published commands relating to cosmic rays: "raydensity" and "scalenumber", but defaults are not specified and their values do not appear in the log file.

Bleeding ==> do not see an obvious association with published overrides, but maybe some combination of "blooming" and "saturation"?

Fringing ==> published list includes "fringeflag" but historical commands file includes "fringing" which is confirmed as recognized by phosim in the log file which says:
[fringing] Fringing (0=off/1=on): 0

Saturation/non-linearity ==> "saturation" and "blooming" are registered in the documentation, nothing obvious for non-linearity

Crosstalk and hot columns/pixels ==> do not see an obvious association with published overrides

@johnrpeterson can you supply a coherent set of overrides that would satisfy these requirements?

There is also a possible inconsistency between the provisional command set and what the log file says. Given that "cleardefects" was included, is it surprising that the log file indicates:
[saturation] Saturation mode (0=off/1=on): 1
[blooming] Blooming mode (0=off/1=on): 1
Are these not considered 'defects'? Exactly which options are disabled by 'cleardefects'?

[cwwalter]

Ok the list we decided to turn ON was:

BF, Tree rings, Cosmic rays, bleeding, X-talk,

Also on: hot columns/pixels. Note: we will need to supply and an input defect mask to DM.

Also on: non-linearity/saturation. Note: we will need to supply non-linearity info to DM ourselves. The cannot measure it.

Off:

Fringing, debris/dust.

[cwwalter]

Corresponding to my list above and Tom's comments. I will add what I can here and @johnrpeterson should add more.

BF = Brighter/Fatter ==> This one is understood: "chargesharing 1"

Yes.

Tree Rings ==> do not see an obvious association with published overrides (possibly "impurityvariation"?)

At least the parameters are set in the instrument directory in (e.g.)

data/lsst/silicon.txt

Cosmic Rays ==> appear to be on by default, as a recent log files indicates their presence. There are two published commands relating to cosmic rays: "raydensity" and "scalenumber", but defaults are not specified and their values do not appear in the log file.

The source code values are for the defaults (at least in 3.6) were:

raydensity = 0.6;
scalenumber = 8.0;

These correspond to:

raydensity X (X is the density of cosmic rays)
scalenumber X (X is the scaling in electrons of the cosmic ray images)

These can be reset by some of the clear commands.

Bleeding ==> do not see an obvious association with published overrides, but maybe some combination of "blooming" and "saturation"?

I'm not sure on this. I think bleed trails is the same as blooming in PhoSim but John will need to confirm.

Fringing ==> published list includes "fringeflag" but historical commands file includes "fringing" which is confirmed as recognized by phosim in the log file which says:
[fringing] Fringing (0=off/1=on): 0

Looks right.

Saturation/non-linearity ==> "saturation" and "blooming" are registered in the documentation, nothing obvious for non-linearity

I don't remember for sure if saturation is a hard cutoff, but I thought it was non-linear as you approach full well.

Crosstalk and hot columns/pixels ==> do not see an obvious association with published overrides

This is handled in the instrument file in e.g. segmentation.txt

 (1): amplifier name
# (2-5): x low, x high, y low, y high
# (6): serialread
# (7): parallelread
# (8-9): gain, % variation
# (10-11): bias level, % variation
# (12-13): readnoise , % variation
# (14-15): dark current, % variation
# (16-19): parallel prescan, serial overscan, serial prescan, parallel overscan (pixel)
# (20): hot pixel rate
# (21): hot column rate
# (22-35): crosstalk

[cwwalter]

John should confirm, but I think if the instrument directory files are correct we want:

fringing 0
contaminationmode 0

backalpha 0.1
backbeta 4.0
backgamma 1000.0
backdelta 1.0
activebuffer 600

sourceperthread 100

To match my list above.

[TomGlanzman]

@cwwalter referenced some quantities in the phosim 'instrument' description. Do we need to review the content of the phosim data/lsst* directory tree or look at certain areas?

[cwwalter]

referenced some quantities in the phosim 'instrument' description. Do we need to review the content of the phosim data/lsst* directory tree or look at certain areas?

I think we should get started after @johnrpeterson confirms/modifies the list for protoDC2. But I think we should understand what is being set in the files and make sure everyone agrees these are the most up-to-date values give our recent knowledge from the actual rafts/sensors.

Things like: tree-ring parameters, x-talk, hot column frequency etc.

[cwwalter]

think we should get started after @johnrpeterson confirms/modifies the list for protoDC2. But I think we should understand what is being set in the files and make sure everyone agrees these are the most up-to-date values give our recent knowledge from the actual rafts/sensors.

Just to be clear: I'm not proposing we wait for this, I think it is fine to use whatever is there now for protoDC2 since the values are probably reasonable. But before run 2.0 starts we should make sure this is all up-to-date and everyone knows the list of what is being set in the files.

[cwwalter]

These changes look OK to me. @johnrpeterson so should confirm.

[jchiang87]

I think we would like to turn off column and pixel defects (e.g., bright or dark pixels). @johnrpeterson Is there a command to do that?

[cwwalter]

I think we would like to turn off column and pixel defects (e.g., bright or dark pixels). @johnrpeterson Is there a command to do that?

It looks like we have to set

# (20): hot pixel rate
# (21): hot column rate

to zero in the segmentation.txt file.

Jim explained to me that we have the masks for the sensors we have received and we want to use them. So this needs to be deterministic. We can just mask these pixels off and so it doesn't matter if the effect is turned on.

[jchiang87]

Since the bright defects are introduced in the e2adc step for phosim, I don't think we need to worry about turning them off for the protoDC2 runs. In fact, I would suggest to not generate the amplifier images for protoDC2 at this point. We can always generate them later from the eimages and we don't know enough about running ISR on simulated images now anyways (or at least I don't).

[cwwalter]

What about things like CRs, xtalk etc. The SN group is very interested in studying the false positive rate.

[jchiang87]

CRs are in the eimages. xtalk is in e2adc, but I don't think it's needed for protoDC2.

[TomGlanzman]

Cosmic rays appear to be enabled. One can see this in the log, e.g.,

Type Sources Photons (Sat,Rem,Rej,Acc)% Time (s) Photons/s
------------------------------------------------------------------------------------------
Cosmic Rays 23 22,157,630,023 ( 0, 96, 2, 2) 105.57 209,867,661

[SimonKrughoff]

CRs should be fine to leave in. XTalk would be fine too, but only if we are starting from amp level images.

[TomGlanzman]

The E2ADC step is currently enabled. Would there be any benefit to leaving it enabled for some fraction of visits (or for the entire protoDC2)?

[jchiang87]

The E2ADC step is currently enabled. Would there be any benefit to leaving it enabled for some fraction of visits (or for the entire protoDC2)?

I really don't think so.

[cwwalter]

I'm a bit nervous about suddenly deciding to turn this off after we told people we would do it. But I suppose as long as we can run it as a separate step it is OK.

It will save us some disk space which is good, but I am worried about the fact that we will have "perfect" ISR. The CR removal will run on the e-image?

Can someone list exactly what is added at the e2adc stage? It means we won't have any amplifier non-linearity, gain variations, x-talk. Can we we get a list of what else?

[jchiang87]

yes, CR repair ran for Twinkles using only e-images. Let's not lose sight of the fact that this is protoDC2 and not DC2 and we can run e2adc easily later since it's a tiny step. I'd really like to get this PR merged and start running the protoDC2 sims.

[TomGlanzman]

The E2ADC step is part of the raytrace step in the current workflow. Disabling/Enabling is quick, but pulling this out as a separate step (e.g., to run later) would cause some delay. Given how close we are to starting, I would recommend that if E2ADC is disabled, then it is run later in an independent job/workflow/script.

[cwwalter]

OK but keeping this in is not stopping us from merging and starting. We are talking about changing things now to take it out. This isn't stopping us from going ahead.

Also, recall that one of the reasons we are doing DC2 at a 10% level of DC3 is to learn how to do it. I don't think waiting until DC3 to make our system work for ISR on the amplifiers is necessarily a good idea.

[cwwalter]

BTW: I'm not trying to be obstructionist but in addition to my points above, last second changes are how we make mistakes.

[cwwalter]

Hi All,

Let me make a concrete proposal: Since this is protoDC2 why don't we just start as we had planned (of course we are still waiting on John to checkoff on the command file) as we configured things before.

Since, this will produce a small dataset there is no harm in producing the amplifier files too. This way we will have them and we can see what happens when we try to run ISR etc when we get to the DM step. We can then use that experience to figure out whether we should have them for run 2.

How do people feel about that?

[TomGlanzman]

From the perspective of workflow operations, that would be fine. We could even decide partway through the project to stop producing them.

[cwwalter]

From the perspective of workflow operations, that would be fine. We could even decide partway through the project to stop producing them.

OK. The reason I like doing this this is if, in the future if we decide we want to split the processing into these pieces for run 2, or not do it, we can will have time to do it carefully with review and not make mistakes.

This way we get a selection of amplifier chips to look at. Maybe we will decide it is worthless or too hard / too much extra work to deal with but at least we will have the opportunity to decide without doing extra work later which might not happen. It's no extra work for us to do this now.

[jchiang87]

The segmentation file that phosim would use right now produces amplifier files with the incorrect pixel geometry. I've said this before on numerous occasions. I honestly don't see the point of running e2adc under those circumstances.

[cwwalter]

The segmentation file that phosim would use right now producer amplifier files with the incorrect pixel geometry. I've said this before on numerous occasions. I honestly don't see the point of running e2adc under those circumstances.

Oh. Sorry... I actually missed this before somehow. What is the issue?

[cwwalter]

Of course I agree that producing bad output doesn't make sense.

[jchiang87]

The serial and parallel directions are swapped for each amp, but there are other issues.

[cwwalter]

I see; that's too bad. Well at some level figuring out this stuff is why we are doing this. I guess we will likely need to do a run1.2 to test with amplifiers after these issues are all worked out. Can we file an issue against bitbucket if it hasn't already been done?

As I said it would be good to have a list of exactly what is in amp files not in the e-images. Just generally, it is good to have this clearly laid out anyway. I think bleed trails are also in the e-images correct?

For my information: Is obs_lsstSim designed to do ISR of effects just on e-images? For amp files do we use the same code as for the camera, or does obs_lsstSim do those too?

So, if we aren't producing usable files, I have a slight preference for still making them so our pipeline still knows how to deal with them, move them around with the right filenames etc but don't object to turning it off if others want as long as the change is little more than a configuration change in the workflow engine that is already tested.

[RobertLuptonTheGood]

I think that the ISR will just run if the fits files are close to those from TS8. I'd much rather that you did that than generate images which have to be handled specially (after all, real cameras don't produce them).

[cwwalter]

I think that the ISR will just run if the fits files are close to those from TS8. I'd much rather that you did that than generate images which have to be handled specially (after all, real cameras don't produce them).

Is now defined what the camera will do? Is it decided that there will be no interim fits file as a exchange format into DM?

I mean obs_comcom is reading the TS data and it uses a fits file. What will happen on the mountain? Is this decided?

[cwwalter]

Several of us working in Chuck's group are interested in starting to write scripts for commissioning validation on the mountain. We would like to use DC2 to work on developing some of them and so I would like to get this as "right" as we can in terms of the readout.

Should we be using Merlin's work on obs_comcam as a guide?

[salmanhabib]

@cwwalter Doesn't DM have its own data challenges? Is it smart to (apparently) mix two activities?

[cwwalter]

Doesn't DM have its own data challenges? Is it smart to (apparently) mix two activities?

Are you asking about my comment about for commissioning? Sorry, maybe I was unclear.

What I am talking about isn't for DM. It is for commissioning studies and preparation (with Chuck Claver). We have been discussing trying to use the DCs for commissioning for a while now including sessions at the collaboration meetings and AHM. In fact we may want to focus DC3 on the commissioning camera and mini-surveys etc.

[cwwalter]

BTW: I am happy to discuss this along with people like Kieth Bechtel etc. But, let's not derail this issue, let's do it somewhere else.

I only brought it up because I was pointing out that in the past it was probably OK for amplifier readout to have technical issues in directions etc but we would now like to get it right as we are getting close to commissioning. Mostly, I was trying to understand from Robert how to think about the amplifier readout and what is coming off the test stands vs what he will eventually see from the camera.

[salmanhabib]

@cwwalter Frankly, I think it's confusing to bring in commissioning issues. That's not our problem to worry about as of now. If it is, someone from the science WGs should be making the argument. If they are not, we should move ahead with a reasonable config and see what comes out. With protoDC2 we are not aiming to be fully realistic (just repeating @jchiang87 at this point).

[astrostubbs]

Can someone point me to an example of an image with cosmic rays introduced? I'd like to compare to lab images. thanks Chris Christopher Stubbs Samuel C. Moncher Professor of Physics and of Astronomy 17 Oxford Street Harvard University Cambridge MA 02138 stubbs@physics.harvard.edu Think this email is terse? Here's why: http://www.emailcharter.org

…

On Dec 19, 2017, at 6:03 PM, Tom Glanzman ***@***.***> wrote: Cosmic rays appear to be enabled. One can see this in the log, e.g., Type Sources Photons (Sat,Rem,Rej,Acc)% Time (s) Photons/s Cosmic Rays 23 22,157,630,023 ( 0, 96, 2, 2) 105.57 209,867,661 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[johnrpeterson]

sorry, didn't realize there were going to be 1000 emails last night.

to turn off cosmic rays do:
raydensity 0.0
to keep it on, you don't need a command

hot pixels are deterministic.

i don't see why you would turn off the readout (e2adc), as it is negligible CPU cost.

to turn off fringing do:
fringing 0

i think this is it. anything else?

[cwwalter]

Hi @johnrpeterson

We want Cosmic rays on:

The proposed file (look at the code in this PR is):

# Enable centroid file
centroidfile 1

# Disable fringing
fringing 0

# Disable dirt
contaminationmode 0

# Quick background
backalpha 0.1
backbeta 4.0
backgamma 1000.0
backdelta 1.0
activebuffer 600

# Number of sources handled by a single thread
sourceperthread 100

Look at comment #63 (comment)

and then the 3 or 4 after that.

[cwwalter]

hot pixels are deterministic.

Can you say a bit about how it works? I see the rate lines for each sensor in the segmentation.txt file. How would you set particular columns and pixels on?

[johnrpeterson]

what is the logic for:

contaminationmode 0

i think this should be on.
also, what about distortion near the edges? this would be on by what you say above.

non-linearity can be given to DM along with hot pixel maps. those are easy. bleeding is on.

[johnrpeterson]

it is deterministic so that it will choose the same pixels as hot pixels every time you run a particular version of phosim. the rate is set in the segmentation file as you've noticed. its based on real devices.

[cwwalter]

OK. Just to make sure I understand: We can set the rate per sensor and it will repeat the patter from run-to-run but we can't choose ourselves which pixels to make hot.

Correct?

[TomGlanzman]

@astrostubbs you may find the images for the partially complete first visit here at NERSC:
/global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-2-r/output/000000
At present there are 13 sensor-visits complete:
lsst_e_151687_f2_R01_S01_E000.fits.gz
lsst_e_151687_f2_R01_S02_E000.fits.gz
lsst_e_151687_f2_R01_S10_E000.fits.gz
lsst_e_151687_f2_R01_S11_E000.fits.gz
lsst_e_151687_f2_R01_S12_E000.fits.gz
lsst_e_151687_f2_R11_S00_E000.fits.gz
lsst_e_151687_f2_R11_S01_E000.fits.gz
lsst_e_151687_f2_R11_S02_E000.fits.gz
lsst_e_151687_f2_R11_S11_E000.fits.gz
lsst_e_151687_f2_R11_S12_E000.fits.gz
lsst_e_151687_f2_R11_S20_E000.fits.gz
lsst_e_151687_f2_R11_S21_E000.fits.gz
lsst_e_151687_f2_R11_S22_E000.fits.gz

[RobertLuptonTheGood]

We really don't care about formats, but for now I think that the thing to do is to write fits files as they are written on TS8. We can handle other formats (it's not even really all that hard), but if we have an LSST simulator it should simulate the data as read using LSST chips (e2V or ITL or mixed) and REBs R

[cwwalter]

We really don't care about formats, but for now I think that the thing to do is to write fits files as they are written on TS8. We can handle other formats (it's not even really all that hard), but if we have an LSST simulator it should simulate the data as read using LSST chips (e2V or ITL or mixed) and REBs

Thanks Robert.

[johnrpeterson]

yes, you can't define the specific pixel, but it will use the specified rate and make them deterministically. they are rather easy to find (use a dark for instance), so you can make a mask for them afterwards.

[cwwalter]

Thanks John.