SNR flag

[paigemkelly]

I added in the SNR ratio flag to cut data <9. 840 clusters are being removed. I am trying to figure out why the RuntimeWarning is happening.

Removed 9 clusters due to bad_mode flag of <class 'bool'>
Removed 15 clusters due to overlap_r500 flag of <class 'bool'>
Removed 7 clusters due to overlap_r2500 flag of <class 'bool'>
Removed 41 clusters due to edge_r2500 flag of <class 'bool'>
Removed 27 clusters due to overlap_bkgd flag of <class 'bool'>
Removed 52 clusters due to edge_bkgd flag of <class 'bool'>
Removed 39 clusters due to masked flag of <class 'bool'>
/home/paige/anaconda3/lib/python3.7/site-packages/astropy/table/column.py:984: RuntimeWarning: invalid value encountered in less
result = getattr(super(), op)(other)
Removed 840 clusters due to 500_kiloparsecs_SNR flag of <class 'str'>
Removed 0 clusters due to Redshift flag of <class 'str'>

NOTE: Removed counts may be redundant, as some data fail multiple flags.
Accepted 191 data out of 1092

mean x error: 8.974993467153285e+42
mean y error: 4.850821240875912

[sweverett]

There may be NaN's or empty values in the SNR column. I'd print out the column to inspect it.

[paigemkelly]

It does, I have found a few. Jose suggested removing the NaNs before making the flag cuts, but I am having some issues getting that to work. This is the error I'm getting for that:
IndexError: boolean index did not match indexed array along dimension 0; dimension is 205 but corresponding boolean dimension is 1092
I'm trying to figure out if I should try to change the NaNs in the catalog, or keep trying to find a way to remove them before the SNR cut.

[sweverett]

Ok I just checked - the NaN check was only in place for get_data(): https://github.com/sweverett/CluStR/blob/master/clustr.py#L267

The flag cuts don't currently have this check. So if you are creating a "mask" which contains 1 for things that you want to cut, then it would look something like this:

mask |= (flag_condition) | (np.isnan(catalog[flag_column]))

What this does is take the existing mask and add new mask values for this specific flag (mask |= corresponds to mask = mask | ...) . Instead of just having the flag condition that you usually have, you also look to see if there are NaN's for flag values. This says a cluster should be flagged if the condition is met or if there is a NaN.

[paigemkelly]

I don't make the mask until after the greater than statement, so I get the same error when I put that in. I believe it is working and just ignoring the NaNs. Should I just suppress the error and move on for now? or try to figure it out?

[jjobel]

Works for version 2c3f335 clustr.py.

The issue came from using NaNs with comparison operators.

NaNs are dealt with by equating them to dummy variables then comparing them to cut_off_value.