Firefly-1537: implement hierarchical catalogs

[robyww]

Firefly-1537: Hierarchical catalogs

• Load any side table with full coverage support

Testing

Firefly

• https://fireflydev.ipac.caltech.edu/firefly-1537-catalog/firefly
• load a big table, I have tested up to one million rows but it could go higher
• all functions are present
• try zooming in and out
• panning around
• drawing layers has a min group setting
• clicking on a group expands it until you scroll around or zoom
• highlight a row from table or images
• select and area and click on the mark data as selected (check) button
• click on a group and see it expand

FinderChart

• https://firefly-1537-catalog.irsakudev.ipac.caltech.edu/applications/finderchart
• do a big area

[loitly]

Here are the results of my testing.

Test: Load Big Table

• Uploaded a 1 million-row zipped parquet file.
• Resulted in varying sizes and shapes of objects overlaying an image.
• Visually surprising; I expected a heat map-like appearance to visualize object density.
• However, it works as intended.

All tests below are based on this table.

Test: Zoom and Pan

• Works as intended.

Test: Min Group

• Initially not intuitive, but after some fiddling, I understand it now.
• Works as intended.

Test: Highlight a Row from Table or Images

• Click on a group: all 100k points are drawn.
• Click on a point in the image: the corresponding table row is highlighted.
• Click on another point in the image: all points disappear.
  • This behavior is unexpected.
• I also don’t understand the reasoning behind expanding all the points at this level.
  • I see a blob on the marks, but as soon as I zoom in, it’s gone.

Test: Clicking on a Group Expands It Until You Scroll or Zoom

• Clicked on a group to expand it.
• The application crashed on the 5th click.
  • Error: Uncaught (in promise) RangeError: Maximum call stack size exceeded.

Test: Select an Area and Click the "Mark Data as Selected" Button

• Selected a circle that overlapped 4 groups, then clicked "Mark Selected."
  • The browser froze, and I was prompted to exit the page multiple times.
  • After about 5 minutes, the results appeared: two yellow circles with yellow/red numbers.
  • I tried to verify if the rows were selected, but table performance had severely degraded.
  • Expanded the table, hoping for better performance, but it did not improve.
  • Closed the expanded table, and the browser froze again, prompting to exit the page.
  • Eventually, it came back after roughly the same amount of time it took to recover the first time it froze.

[lrebull]

Ok, just trying things out here, so just noting what I'm seeing, sort of stream-of-consciousness, for better or worse...

On the whole, I really, really like this a lot, and I can't wait for this to be widely available.

Uploaded three massive catalogs (well, two HUGE and one smaller). They loaded pretty quickly, all things considered, but all of them have come up with red squares, so the first thing I have to do is pull up layers and change colors. (see screenshot 1)

Even the smaller catalog is represented the same way as the big catalogs, which I initially didn't expect, but that does make sense. I just need to remember to highlight this to users who may be conditioned now to making catalogs under the 5000 source limit.

I didn't expect some ellipses (as opposed to all circles) as I zoomed in, but I guess that makes sense given the non-uniform tiles? screenshot 2. Why are some individual sources shown here? is it because those cells are sparse enough that it has decided to show the individual sources?

Empirically, I'm trying the "min group" thing, and I'm thinking this must be the minimum number of sources in a 'cell' before it turns into a cell, but I'm not 100% sure of that. Is that what it is? We need a tooltip; I can help with wording if I understand what it's doing.

When you have multiple catalogs overlaid, the numbers of sources in a 'cell' are overwritten right on top of each other. I'm not sure there is a solution to this. (screenshot 3) It doesn't look like tooltips work here (e.g. like plots do when you mouseover a point).

Ooh, THIS feature is really cool. (screenshot 4) I didn't expect this! I don't know what causes it to redraw it as a cell though. I first thought it was once I clicked in a different healpix cell, but empirically that's not the case. It just seems to randomly decide, oh, now i'll re-render it as a bin rather than individual points. And it only works if you click within the circle, not outside a circle, which is unexpected, especially if I have an individually rendered source from one catalog outside a circle, but everything else from the other two catalogs shown as a circle.

This really is cool! I'm sure the others will have feedback too.

[lrebull]

I have a hastily scribbled note to myself that you wanted me to test this again but i can't tell from the stuff here if you are ready for me to test it again or not ...?

[robyww]

@lrebull try:

• multiple catalogs
• selection and (checking) of any amount
• different forms for grouping (in layer dialog) (ellipse,point,healpix)
• The ellipse drawing is improved
• clicking on an area (anywhere in the healpix area, like you requested)
• min group changes
• expanding groups by clicking, unlimiited when you zoom in deep
• expanded group do regroup when you scroll, only when you zoom

[lrebull]

WOW the multiple catalog handling looks REALLY good now!
I really like the different choices for grouping too. (in the layer dialog)
i verify taht the clicking anywhere (not just in the circle) works.
i verify that the expanded groups only regroup when you zoom, not scroll.

SO COOL when do we get it on ops??

i would like to test this in finder chart because i bet that will break all sorts of things ...

[kpuriIpac]

@lrebull try:

• multiple catalogs
• selection and (checking) of any amount
• different forms for grouping (in layer dialog) (ellipse,point,healpix)
• The ellipse drawing is improved
• clicking on an area (anywhere in the healpix area, like you requested)
• min group changes
• expanding groups by clicking, unlimiited when you zoom in deep
• expanded group do regroup when you scroll, only when you zoom

Only just got around to testing this as I was occupied with Euclid, this looks great! I think most (if not all) of the bugs @loitly encountered have since been resolved, I tested mostly according to his testing flow above and this latest update. I was also confused by Min Group (and I think I still am - that's the only one I didn't properly understand, but I think that's on me).

Played around for a while, zooming/panning, expanding groups, etc. Didn't crash/freeze for me. Tried out the different grouping types, clicked anywhere in the Healpix grid to expand (neat that this works also when you've Box or Ellipse selected). Zooming out regroups the previously expanded groups. Clicked on a point to see the corresponding row highlighted in the table, and vice versa. Selected areas via cone/rectangular select and saw the corresponding rows highlighted the table. Didn't get a chance to look at the code, but this works great.

[lrebull]

OOh! OK, trying FC. I did the thing where I upload enormous catalogs again, because that's my most stressful use case. NEVERMIND I TAKE IT BACK THIS IS THE REAL EDGE OF THE CATALOG. It does.. this. which is weird. it looks like i've zoomed in enough to get the individual sources from one cell but not an adjacent cell? i don't know, it's ...strange. catalog files attached as a tarfile. i uploaded "ysolist.tbl" as the multiple position search with everything set as the defaults, and then overlaid the two other catalogs (entire*tbl) and zoomed in on the first source in the list until I could see individual sources. UPDATE: still happens for me in 'new private window.'

bigcatalogs.tar.gz

[lrebull]

This is really cool. It seems to work pretty well for me!

[lrebull]

The color swatches on the catalog tabs are missing, FYI.

[loitly]

If performance matters, you should use BigInteger.valueOf(v) instead.

[loitly]

Here, you extract ra/dec from the includeColumns field with assumption that ra is in index 0, then dec at 1 later down the road, accounting for quoted syntax used by SQL.
You could have made 'ra' / 'dec' parameters, then setInclColumns with those values. This make the request more clear, making it easier to validate as well.