Add WALKER and RUNNER bits (was: propagate additional bits from tractor-i to tractor files)

[dstndstn]

• hit_limit
• fit_background
• force_pointsource

As a set of boolean columns? Or (yet another!) new bitmask?

[djschlegel]

We shouldn't use MASKBITS, since those have a 1-to-1 mapping to pixels in the mask bits file.
We shouldn't use ANYMASK_,ALLMASK_, since those map to the CP images in each filter.
Therefore, I think we should define a new bit mask. We could call it FITBITS ?

[djschlegel]

A concrete suggestion for FITBITS:
0 - FORCE_POINTSOURCE - Force to be TYPE=PSF for Gaia and Tycho stars at G<=13, or Gaia stars at 13<G<16 with astrometric_excess_noise < sqrt(10)
1 - FIT_BACKGROUND - Fit an additional constant background term per source per filter
2 - HIT_LIMIT - Hit optimization limits (for which parameters?)

Do we want to add bits for any of the following boolean flags in the tractor-i catalogs?
DONOTFIT, ISBRIGHT, POINTSOURCE, DUPLICATED_SOURCE, ISGAIA, PHOT_VARIABLE_FLAG, FREEZEPARAMS, IN_BOUNDS, ITERATIVE, FORCE_KEEP_SOURCE, REGULAR

There are also the flags for ISCLUSTER, ISMEDIUM, ISLARGEGALAXY. These are almost the same as the information in the MASKBITS, but not exactly for the cases where the center of the objects has walked across those geometric boundaries. So we may want to set these as bit masks also.

Finally, we could set a bit to indicate if the position of the object has walked more than 1 arcsec in the fitting, by comparing BX,BY to the BX0,BY0. We would not set this flag for ITERATIVE or DUP sources, which have IBX,IBY set to some nonsense values.

[dstndstn]

The FORCED_POINTSOURCE value in tractor-i currently is based on the MASKBITS at the source's initial position; this bit says whether this source was forced to be a point source on account of being in a mask.

For HIT_LIMIT -- currently, we have optimizer limits on the Sersic index (0.5 and 6.0) and the Radius: Rex 0.01" to min(10", 0.8 * max(sw,sh)) where sw,sh is the size of the subset of the blob the source was fit on. For Dev, Exp, and Ser, it's 30" rather than 10" as the max size. (Except for SGA galaxies, which get the max radius set to twice the initial radius.)

I could break out the individual limits, which might be more useful!

The ISCLUSTER, ISMEDIUM, ISLARGEGALAXY, ISGAIA, ISBRIGHT are flagging whether that particular source is itself in those categories, NOT whether it's inside the mask of another source. That information isn't 100% readily available (obviously it's computable from the other columns, but not always trivially -- eg since the BRIGHT and MEDIUM cuts for Gaia stars are based on the predicted z-band, it's a polynomial of G, BP, RP.

I like the idea of the WALKER flag!

[djschlegel]

Dustin has run tests with DR9.5.4-32-g97ec3d4b in /global/cscratch1/sd/dstn/segsize that has the following definitions for a new FITBITS column:
FB_FPSF = 1 / fitbits: forced to be PSF
FB_FITBG= 2 / fitbits: background levels fit
FB_RLIM = 4 / fitbits: hit radius limit during fit
FB_SLIM = 8 / fitbits: hit Sersic index limit during fit
FB_FROZE= 16 / fitbits: parameters were not fit
FB_BRITE= 32 / fitbits: bright star
FB_MED = 64 / fitbits: medium-bright star
FB_GAIA = 128 / fitbits: Gaia source
FB_TYCHO= 256 / fitbits: Tycho-2 star
FB_LGAL = 512 / fitbits: SGA large galaxy

Four remaining issues:
(1) The FB_RLIM doesn't appear to be properly tracking those sources that hit the radius limits, but instead is labelling the identical objects as FB_SLIM.
(2) The header docs for FITBIT first has the definitions listed above, but then for FBIT_0, ..., mis-prints the info from another mask. So the header card:
FBIT_0 = 'NPRIMARY' / fitbits bit 0 (0x1):
should be something like:
FBIT_0 = 'FORCE_PSF' / fitbits bit 0 (0x1):
(3) Do we still want WALKER/RUNNER (small/big offsets from initial BX0,BY0)?
(4) Let's pack the ITERATIVE column into this bit-mask

[dstndstn]

Hi David, Good catch on the header errors in FBIT -- I've just put in a fix for that. About FB_RLIM and SLIM -- I'm finding that they DO make sense -- Counter(T.sersic[T.fitbits & 0x8 > 0]).most_common() ==> [(0.5, 147), (6.0, 125)] so the ones marked SLIM have Sersic values 0.5 or 6.0, as expected. Meanwhile, for RLIM, Counter(T.shape_r[T.fitbits & 0x4 > 0] / 0.262 / 0.8).most_common() ==> [(11.0, 40), (9.0, 35), (13.000001, 16), (11.999999, 15), (15.0, 12), (14.000001, 9), (0.047709923, 9), ...] so the ones marked RLIM are at integer values in the (weird) scaled sizes, or tiny (shape_r ~= 0.01). I do still want to add WALKER/RUNNER, but I need to add in correct position initialization for iterative sources, and I didn't have time to plumb that in. The set of bricks I queued is nearly done (still waiting for one CLUSTER brick to finish), so if there are more you'd like to see, I can fire them in. Thanks for the checking!! --dustin

…

On Sat, Jul 4, 2020 at 12:22 PM David Schlegel ***@***.***> wrote: Dustin has run tests with DR9.5.4-32-g97ec3d4b in /global/cscratch1/sd/dstn/segsize that has the following definitions for a new FITBIT column: FB_FPSF = 1 / fitbits: forced to be PSF FB_FITBG= 2 / fitbits: background levels fit FB_RLIM = 4 / fitbits: hit radius limit during fit FB_SLIM = 8 / fitbits: hit Sersic index limit during fit FB_FROZE= 16 / fitbits: parameters were not fit FB_BRITE= 32 / fitbits: bright star FB_MED = 64 / fitbits: medium-bright star FB_GAIA = 128 / fitbits: Gaia source FB_TYCHO= 256 / fitbits: Tycho-2 star FB_LGAL = 512 / fitbits: SGA large galaxy Three remaining issues: (1) The FB_RLIM doesn't appear to be properly tracking those sources that hit the radius limits, but instead is labelling the identical objects as FB_SLIM. (2) The header docs for FITBIT first has the definitions listed above, but then for FBIT_0, ..., mis-prints the info from another mask. So the header card: FBIT_0 = 'NPRIMARY' / fitbits bit 0 (0x1): should be something like: FBIT_0 = 'FORCE_PSF' / fitbits bit 0 (0x1): (3) Do we still want WALKER/RUNNER (small/big offsets from initial BX0,BY0)? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEH7LO3NEAORAWQDYK2M3RZ5JN5ANCNFSM4OG62NFA> .

[moustakas]

Done in #618.