count_nested should correctly handle the empty NestedFrame case, to produce meta

[gitosaurus]

Feature request

When a function that accepts DataFrame or NestedFrame is able to handle the zero-size input case, returning a correctly structure zero-size output, then lsdb.catalog.Catalog.map_partitions is able to correctly deduce the meta= for that function, without obliging the user to do so.

Consider the case of count_nested:

from nested_pandas.utils import count_nested

def count_points(pts):
    if len(pts) == 0:
        return pts.assign(n_lc=0)
    return count_nested(pts, "lc")


ztf_objects.map_partitions(count_points).compute()

The above is relatively compact, but still requires the user to know that the additional column added will be called n_lc. If count_nested made that same test within itself, the interface would become even cleaner:

from nested_pandas.utils import count_nested

def count_points(pts):
    return count_nested(pts, "lc")


ztf_objects.map_partitions(count_points).compute()

Before submitting
Please check the following:

• I have described the purpose of the suggested change, specifying what I need the enhancement to accomplish, i.e. what problem it solves.
• I have included any relevant links, screenshots, environment information, and data relevant to implementing the requested feature, as well as pseudocode for how I want to access the new functionality.
• If I have ideas for how the new feature could be implemented, I have provided explanations and/or pseudocode and/or task lists for the steps.

[hombit]

Could you please show a minimal failing example? This code works for me:

from nested_pandas.utils import count_nested
from nested_pandas.datasets import generate_data

count_nested(generate_data(10, 3).iloc[0:0], "nested")

Empty NestedFrame
Columns: [a, b, nested, n_nested]
Index: []

by != None cannot work without meta or some eager computations, because in the Dask world we cannot derive all unique values of a sub-column. See how it is currently implemented in lsdb.nested:
https://github.com/astronomy-commons/lsdb/blob/61f5aa9e4cec67e30bc22841f1a09e9773c1d193/src/lsdb/nested/utils/utils.py#L35

[gitosaurus]

Sure enough, this seems to have been fixed by nested-pandas-0.4.6.dev2+g94621d7!

When I tried your example above on version nested-pandas==0.4.4.dev2+gc136206, I got something different:

from nested_pandas.utils import count_nested
from nested_pandas.datasets import generate_data

count_nested(generate_data(10, 3).iloc[0:0], "nested")

Empty NestedFrame
Columns: [a, b, nested, t, flux, band]
Index: []

count_nested(generate_data(10, 3).iloc[0:1], "nested")

          a         b                                             nested  n_nested
0  0.737367  0.712907  [{t: 0.826597, flux: 50.826447, band: 'g'}; …]...         3

This is the minimal example which obliged me to add the handling of an empty dataframe, before:

import lsdb
from nested_pandas.utils import count_nested

ztf_objects = lsdb.open_catalog(
    "https://data.lsdb.io/hats/ztf_dr22",
    search_filter=lsdb.ConeSearch(ra=254.5, dec=35.3, radius_arcsec=1800),
)
ztf_objects = ztf_objects.nest_lists(
    list_columns=["hmjd", "mag", "magerr", "clrcoeff", "catflags"],
    name="lc",  # name to give the resulting nested column
)

def count_points(pts):
    return count_nested(pts, "lc")

catalog_w_features = ztf_objects.map_partitions(count_points)
catalog_w_features.head(10)

When this ran before, I got this error:

ValueError: The columns in the computed data do not match the columns in the provided metadata.
  Extra:   ['n_lc']
  Missing: ['catflags', 'clrcoeff', 'hmjd', 'mag', 'magerr']

But now it succeeds! Closing as fixed.