Identical catalog indices cause nest_lists to duplicate rows #334
Description
Bug report
I'm using ZTF DR22, which has some duplicate indices (_healpix_29). One object has observations in two different bands, which might have the same _healpix_29 indices. When this occurs, the nest_lists function nests the list into a copy of each pre-existing row, yielding a duplicate of each one. See the attached screenshot, in which this object starts out with 2 rows and has 4 after nest_lists. Also notice the mismatch between nepochs and the length of the nested lists indicating which row is "fake".
Here is the code snippet in the screenshot:
cone = lsdb.ConeSearch(ra=285, dec=-15, radius_arcsec=1*3600) ztf_22_scattered = lsdb.open_catalog('/epyc/data3/hats/catalogs/ztf_dr22/ztf_lc', margin_cache='https://data.lsdb.io/hats/ztf_dr22/ztf_lc_10arcs', search_filter = cone) ztf_22 = ztf_22_scattered.nest_lists(list_columns=["hmjd", "mag", "magerr", "catflags", "clrcoeff"], name="lc") ztf = ztf_22.compute() ztf.loc[2050364164511734616]
Before submitting
Please check the following:
- [✓ ] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [ ✓] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.