Support list-column aggregation into nested columns

[hombit]

Feature request

With HATS, we plan to serialize nested columns as a set of list-columns and include metadata for reconstruction: astronomy-commons/lsdb#466

To support this, we could modify our custom to/read_parquet implementations accordingly. It is not yet clear where HATS will store metadata that maps list-columns back to their respective nested columns. Here, we can explore the possibility of storing this information within the parquet metadata.

Before submitting
Please check the following:

• I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
• I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
• If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

[dougbrn]

@hombit is this resolved by the serialization PR, or is there some amount of work still to do on this?

[hombit]

I think there is nothing left on the nested-pandas side here. I still want to see the support of list-column aggregation on HATS/LSDB side