Satellite/Airplane streak detected as > 200 sources in single sub-image

[simonegarrappa]

Catalog: /marvin/last/data/temp/LAST.01.05.01/2024/03/05/proc/021913v0/LAST.01.05.01_20240306.022013.877_clear_OP313_004_001_016_sci_proc_Cat_1.fits

SubImage:
/marvin/last/data/temp/LAST.01.05.01/2024/03/05/proc/021913v0/LAST.01.05.01_20240306.022013.877_clear_OP313_004_001_016_sci_proc_Image_1.fits

Maybe a known issue, but giving some info about it. The satellite streak in the image is detected as 264 single sources in this case, all with MAG ~ 16 (see histogram here)

I'm adding a plot here of RA and Dec of all sources in the catalog

The flags raised for some of these sources are:
'BiasFlaring', 'DarkHighVal', 'DarkLowVal', 'LowRN', 'NonLinear', 'Saturated', 'Xtalk'

45% of these sources have no flags in the catalog.

[EastEriq]

Is the current pipeline using anything at all to detect streaks? I myself made imProc.mask.maskTracks for PTF. It is far from perfect, but good to some extent.

[EranOfek]

Currently we are not searching for streaks - too expendive. If you have an idea on how to fo it in in less than 40ms per 1700x1700 image then let me know. Howver, i have a code that runs on the merged dources and flag orphan detections on a line… This is for post processing. Eran

…

On Wed, 10 Apr 2024 at 20:49 EastEriq ***@***.***> wrote: Is the current pipeline using anything at all to detect streaks? I myself made imProc.mask.maskTracks for PTF. It is far from perfect, but good to some extent. — Reply to this email directly, view it on GitHub <#433 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJUQ4POJXG5CQZBYRVAJQ3Y4V3QZAVCNFSM6AAAAABGA4CHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYGEZTOOJWGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[EastEriq]

Seems I'm not far from that, at least on random data:

>> AI.Image=20*imUtil.art.createSegments([1700,1700],[322,233;98,0],[54,11;145,211],'width',0.5) + 50*rand(1700,1700);
>> imProc.background.background(AI);
>> tic;imProc.mask.maskTracks(AI);toc
Elapsed time is 0.606075 seconds.
>> tic;imProc.mask.maskTracks(AI);toc
Elapsed time is 0.057231 seconds.
>> tic;imProc.mask.maskTracks(AI);toc
Elapsed time is 0.049659 seconds.

on last11e. Maybe the first call takes longer because the mask is created anew, I could do a more serious analysis.

[EranOfek]

I tried it on real image and it is too slow - why? Also returning not real lines... ``` cd /marvin/LAST.01.01.01/2024/03/12/proc/190730v0 AI=AstroImage.readFileNamesObj('LAST.01.01.01_20240312.190720.919_clear_918_000_001_015_sci_coadd_Image_1.fits'); AI=imProc.background.background(AI) tic;imProc.mask.maskTracks(AI);toc Elapsed time is 0.964165 seconds. AI.MaskData.bitStat ans = 32x3 table BitMask Name Number Fraction _____________________ __________ __________ {'Streak' } 7250 0.0024336 ```

…

On Wed, Apr 10, 2024 at 11:10 PM EastEriq ***@***.***> wrote: Seems I'm not far from that, at least on random data: >> AI.Image=20*imUtil.art.createSegments([1700,1700],[322,233;98,0],[54,11;145,211],'width',0.5) + 50*rand(1700,1700); >> imProc.background.background(AI); >> tic;imProc.mask.maskTracks(AI);toc Elapsed time is 0.606075 seconds. >> tic;imProc.mask.maskTracks(AI);toc Elapsed time is 0.057231 seconds. >> tic;imProc.mask.maskTracks(AI);toc Elapsed time is 0.049659 seconds. on last11e. Maybe the first call takes longer because the mask is created anew, I could do a more serious analysis. — Reply to this email directly, view it on GitHub <#433 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJUQ4MODQAJNYVYYBQNPITY4WMCVAVCNFSM6AAAAABGA4CHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBYGM2TKNZZGM> . You are receiving this because you commented.Message ID: ***@***.***>

[EastEriq]

It may be data dependent & a matter of tuning parameters. I said it is not perfect, YMMV.
On Simone's image I get timings of the order of 300ms on calls beyond the second. The longer time for the first call may be because of JIT compilation.

>> cd /marvin/LAST.01.05.01/2024/03/05/proc/021913v0
>> AI=AstroImage.readFileNamesObj('LAST.01.05.01_20240306.022013.877_clear_OP313_004_001_016_sci_proc_Image_1.fits');
>> AI=imProc.background.background(AI);
>> tic;imProc.mask.maskTracks(AI);toc
Elapsed time is 0.287869 seconds.

It more or less catches the track, I quite believe that it would improve if tuning the arguments:
imagesc(AI.MaskData.findBit('Streak'));axis xy

[EastEriq]

In fact I suspect that a considerable time in the function goes not for the detection of the streak but for the generation of the pixellated image of the segments in the Mask matrix, which can perhaps be made more efficient. I would profile it, but as of now euler is clogged by your 23 matlab processes...

[EranOfek]

euler has 128 processors and 256 threads...

[EastEriq]

I don't know what is with euler but I'm not holding my breath to get responsiveness.
Actually profiling on my desktop and on random data shows that 1/3 of the time is spent only in parsing arguments for hough and houghpeaks which is entirely superfluous, so there is certainly room for improvement.