Firelfy-838: Extraction of data from a FITS image by robyww · Pull Request #1136 · Caltech-IPAC/firefly

robyww · 2021-10-06T23:30:06Z

Firelfy-838: Extraction of data from a FITS image

Cube z-axis extraction
Line extraction across an imagae
Points extraction by click on points
Direct extraction from fits file, no extra data read into memory
UI syncing the images, charts and tables together after extraction
New icons for all three extractions types
All images are now uncompressed
Create a spectral data model for cube extraction with wavelength data
If there is a HDU name that begin with ERR use that for spectrum error
Improved Catalog drawing layer to handle catalogs of image pt

ticket: https://jira.ipac.caltech.edu/browse/FIREFLY-838

Testing

url: https://fireflydev.ipac.caltech.edu/firefly-838-cube-extractor/firefly/
reading in a fits cube or plain FITS image.
Us tool pull down to find 3 types of extraction.
Line and point work on any fits
Z-axis works on cubes
Results immediately
Use Keep to create a table
Use save to save the table right away.
Spectral cubes will create a spectral chart when you keep

- Cube z-axis extraction - Line extraction across an imagae - Points extraction by click on points - Direct extraction from fits file, no extra data read into memory - UI syncing the images, charts and tables together after extraction - New icons for all three extractions types - All images are now uncompressed - Create a spectral data model for cube extraction with wavelength data - If there is a HDU name that begin with ERR use that for spectrum error - Improved Catalog drawing layer to handle catalogs of image pt

lrebull · 2021-10-06T23:32:10Z

Do you have sample FITS files for us to try?

robyww · 2021-10-06T23:38:51Z

Sent

ejoliet · 2021-10-07T16:59:30Z

Vandana is collecting CCB-D samples here https://confluence.ipac.caltech.edu/display/FIREFLY/Example+MEF%2C+Cube%2C+Spectral+Image+FITS+Files

lrebull · 2021-10-07T21:44:53Z

Wow, this is really cool.
trying the "drill" icon tool.

I know you probably don't want detailed comments like this, but a possible ambiguity in wording -- "download", empirically, refers to downloading the data table, but because this is a plot, i thought initially that it meant to download the plot. Maybe that can be a choice in the pop-up? or one of the choices in the download file format?
After I click "keep", and the pop-up window is closed, the main image pane jumps back to the first plane, as opposed to the plane I left it on (15).
If you try to make it go to a plane by typing in a number larger than the number of planes, it just drops you back in the first plane. (obviously not a deal breaker, but had me confused for a bit.)
After messing around with it for a while, i have two target markers on my image, and i can't make them go away. Were they supposed to be the locations of the drill down? I've now deleted all the tables and they are still there. I tried again, and now those target markers sure seem like they are supposed to be the locations of the drilldown, but they are offset. (see movie attached next item)
when scrolling through the rows in my "kept" drilldown, the image jumps, or rather the markers jump? maybe both? see movie: https://user-images.githubusercontent.com/23038600/136466471-468b5fd8-9422-419a-99bf-898a77c27701.mp4

I haven't tried anything other than the "drill" icon tool yet. More later.

robyww · 2021-10-07T23:19:13Z

I know you probably don't want detailed comments like this, but a possible ambiguity in wording -- "download", empirically, refers to downloading the data table, but because this is a plot, i thought initially that it meant to download the plot. Maybe that can be a choice in the pop-up? or one of the choices in the download file format?

Good Suggestion: I will make it Download Table and add a Download Chart.

After I click "keep", and the pop-up window is closed, the main image pane jumps back to the first plane, as opposed to the plane I left it on (15).

FIXED: I was aware of that but your comment helped me think of a solution.

If you try to make it go to a plane by typing in a number larger than the number of planes, it just drops you back in the first plane. (obviously not a deal breaker, but had me confused for a bit.)

I would have to see this one, I though it gave you the error indicator. It does for me, maybe you are doing something different.

UPDATE: After watching @lrebull 's movie... It is an existing behavior, so it is of scope for this PR.

After messing around with it for a while, i have two target markers on my image, and i can't make them go away.

They go away for me. I will study your movie.

when scrolling through the rows in my "kept" drilldown, the image jumps, or rather the markers jump?

FIXED: I am going to center the marker points.

Details:

The markings are the point that was extracted. They are on the bottom left of the pixel. I will make them center on the pixel. You will see this again at other times like line extraction. This is pretty standard for fits viewers. If I plot the point 5,5 on a image with a zoom level of like 20 you will see them on the bottom left corner. To center them I have to make the point 5.5,5.5. That is fine for the markers. However, line extraction that shows up in the table will plot them on the bottom left, because it is plotting the table data.

The jumping is the image repositioning itself. Sometime it is slow enough to see. I think it ends up in the right place. So, for now at least, I am not going to do anything about it.

lrebull · 2021-10-07T23:34:54Z

video demonstrating my #3 which you said you couldn't reproduce
https://user-images.githubusercontent.com/23038600/136476082-e5cc515e-fee9-4cdd-bf72-2e139322644c.mp4

lrebull · 2021-10-07T23:42:07Z

checking new version from slack. markers still seem to me to be offset but maybe it's just the big pixels ?

Ooh, a help anchor has appeared! I need to spend some time figuring out what the docs framework needs to be - i may very have to reorganize substantially to accomodate the rework (not just the new toolbar stuff but the reorg may mean the docs need to be comprehensively changed). I'm not sure I'll really be able to "see" the new org until i see more of the new tool, but i can try.

robyww · 2021-10-08T16:18:32Z

@lrebull thanks for the video. This is out of scope for this ticket. It has been the existing behavior for over a year, maybe two. Right now I am not inclined make a ticket to change it. If you feel very strongly that the behavior is wrong I suppose you can make a ticket. It just seems very low priority for something that is not really a bug.

loitly

Great PR. Lots of features added. I do like you to reconsider using the existing
defaultChartDef(MetaConst.DEFAULT_CHART_DEF) feature instead of creating the new method. It's allow you to do what you want to do, plus more without much complexity.

loitly · 2021-10-08T19:27:56Z

+                                                List<FitsReadUtil.ExtractionResults> results,
+                                                String wavelengthColName, String fluxColName) {
+
+        dataGroup.getTableMeta().addKeyword("utype", "spec:Spectrum");


Use constant TableMeta.UTYPE instead.

loitly · 2021-10-08T19:40:55Z

+        TableMeta meta= dataGroup.getTableMeta();
+        meta.addKeyword(MetaConst.DEFAULT_CHART_X_COL, xColName);
+        meta.addKeyword(MetaConst.DEFAULT_CHART_Y_COL, defYCol);
+        dataGroup.trimToSize();


I don't think trimToSize is needed. You did not touch the data.

loitly · 2021-10-08T19:58:38Z

+    isTableLoaded,
+    stripColumnNameQuotes,
+    watchTableChanges
+} from '../tables/TableUtil.js';


You mentioned in the past that you would revisit this auto style's option.

loitly · 2021-10-08T20:15:14Z

+    // test to see if meta set the default x and y coloumns
+    const additionalChartDef= getMetaEntry(tblModel,MetaConst.ADDITIONAL_CHART_DEF);
+    let xCol= getMetaEntry(tblModel,MetaConst.DEFAULT_CHART_X_COL);
+    let yCol= getMetaEntry(tblModel,MetaConst.DEFAULT_CHART_Y_COL);
+    if (xCol) {
+        return genericXYChart({tbl_id, xCol, yCol:yCol||xCol, additionalChartDef});
+    }


I am not sure this is the best approach. It's very specific to scatter plot and yet genericXYChart can create a Histogram. It also relies on additionalChartDef which is also JSON based.

Have you looked into using defaultChartDef(MetaConst.DEFAULT_CHART_DEF) instead?
It can be as simple as this javacript equivalent, yet allow you to make it as complex as you like.
META_INFO: {defaultChartDef: JSON.stringify({data: [{x: 'tables::w1mpro', y: 'tables::w2mpro', mode: 'markers'}]})}

I have thought about this this a lot. I think we need to specify the columns. Now maybe we could use one instead of two such as wmpro1;wmpro2 using DEFAULT_CHART_DEF is very complex to every use case. The ADDITIONAL_CHART_DEF was an experiment that I am not using, I meant to take out. I will do that. But we need a way to just specify the columns.

…esponded to feedback - added a download chart - made cube not reposition when extraction table is loaded - make z-axis markers center on image point - removed additionalChartsDef

ejoliet · 2021-10-13T19:23:51Z

I know it was merged, but i'd like to give my notes here for the records i couldn't comment before as i was on vacation.

Layer settings?

Any better way to give the information ? Extraction Point
(6, 20) / space coordinate instead if exists?
“Selected point” - maybe a better wording that refer to the drill feature? “Drilled point”?

Pop up dialog

Improvement: why 5x5 and not custom N x M?
when selected 5x5 pixels, it resets after closing the drill feature, does it matter? Should we keep this next time the user open it?
“send to time series”?

Note: Found very useful that the dot in the chart is connected to the plane index.

robyww added enhancement Image FITS images Charts Anything related to charts labels Oct 6, 2021

robyww requested review from ejoliet, gpdf, loitly, lrebull and vandesai1 October 6, 2021 23:30

robyww self-assigned this Oct 6, 2021

loitly approved these changes Oct 8, 2021

View reviewed changes

Firefly-838: Added support for "keep" in the MultiProductViewer and r…

870b3f5

…esponded to feedback - added a download chart - made cube not reposition when extraction table is loaded - make z-axis markers center on image point - removed additionalChartsDef

robyww force-pushed the firefly-838-cube-extractor branch from 1304c8b to 870b3f5 Compare October 11, 2021 19:50

robyww merged commit e720d79 into dev Oct 11, 2021

robyww deleted the firefly-838-cube-extractor branch October 11, 2021 19:58

robyww added this to the 2021.4 milestone Oct 12, 2021

Conversation

robyww commented Oct 6, 2021

Firelfy-838: Extraction of data from a FITS image

Testing

Uh oh!

lrebull commented Oct 6, 2021

Uh oh!

robyww commented Oct 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ejoliet commented Oct 7, 2021

Uh oh!

lrebull commented Oct 7, 2021

Uh oh!

robyww commented Oct 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrebull commented Oct 7, 2021

Uh oh!

lrebull commented Oct 7, 2021

Uh oh!

robyww commented Oct 8, 2021

Uh oh!

loitly left a comment

Choose a reason for hiding this comment

Uh oh!

loitly Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

loitly Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

loitly Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

loitly Oct 8, 2021

Choose a reason for hiding this comment

Uh oh!

robyww Oct 11, 2021

Choose a reason for hiding this comment

Uh oh!

ejoliet commented Oct 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robyww commented Oct 6, 2021 •

edited

Loading

robyww commented Oct 7, 2021 •

edited

Loading