Redundancy between ibis and ibis-ml

[vspinu]

Thanks for the amazing initiative!

I am a bit taken aback by the redundancy of abstractions between ibis-ml and native ibis. I would expect ibis-ml to be a lightweight extension of ibis as much as possible, but that doesn't seem to be the case. Ibis-ml does its own stuff which is not compatible with the core ibis.

Here are a few examples which I came accross.

Selectors

Ibis-ml has its own abstraction for selectors. For example the following cast

ml.Cast(ml.has_type("boolean"), "int8"),

could have been:

ml.Cast(s.of_type("boolean"), "int8"),

Casing

Ibis-ml uses CamelCase. Ibs uses snake_case.

Ibis Pipelines

Most importantly, Ibis pipelines are already lazy and backend independent. So why not reuse those as ML recipes directly?

Ibis-ml could simply either

1. enrich the existing backend transforms with the ML functionality, or
2. provide its own proxy backend which would be dispatched to backends depending on the input data to the fit method

For example:

## 1. Ibis Table is already a deferred recipe, so use it as such:
rcp = (
    df
    .drop(["approved", "day"])
    .mutate(day=_.cast("string"))
    .mutate(s.across(s.endswith("_id"), _.cast("string")))
    .fill_na(s.of_type("string"))
    .mutate(s.of_type("boolean"), _.cast("int8"))
    .ordinal_encode(s.of_type("string"), min_frequency=0.01)
)

tr = rcp.fit() # or rcp.fit(df), or rcp.fit(df_from_other_backend)


## Option 2:
# Start with a ml.recipe pseudo backend 
rcp = (
    ml.reicipe
    .drop(["approved", "day"])
    .mutate(day=_.cast("string"))
    .mutate(s.across(s.endswith("_id"), _.cast("string")))
    .fill_na(s.of_type("string"))
    .mutate(s.of_type("boolean"), _.cast("int8"))
    .ordinal_encode(s.of_type("string"), min_frequency=0.01)
)

tr = rcp.fit(df)

Does this make sense?

[deepyaman]

Hi @vspinu! Thank you for raising this and other issues, and I apologize for not seeing them earlier. I'll try to make some time over the weekend to go through the other issues; I suspect some things may not be working since IbisML has not get been updated for Ibis 10.

Ibis Pipelines

Most importantly, Ibis pipelines are already lazy and backend independent. So why not reuse those as ML recipes directly?

Ibis-ml could simply either

1. enrich the existing backend transforms with the ML functionality, or
2. provide its own proxy backend which would be dispatched to backends depending on the input data to the fit method

For example:

1. Ibis Table is already a deferred recipe, so use it as such:

rcp = (
df
.drop(["approved", "day"])
.mutate(day=_.cast("string"))
.mutate(s.across(s.endswith("_id"), _.cast("string")))
.fill_na(s.of_type("string"))
.mutate(s.of_type("boolean"), _.cast("int8"))
.ordinal_encode(s.of_type("string"), min_frequency=0.01)
)

tr = rcp.fit() # or rcp.fit(df), or rcp.fit(df_from_other_backend)

If I understand correctly, the main benefit here would be that you could mix and match Ibis functions with extensions for ML preprocessing? I think the part that feels unclear to me is you're saving the result as rcp—some Recipe-like object—which happened to be returned because you're calling ML extensions, whereas if you'd done "normal" Ibis transformations, it would have still been a table. That function-dependent behavior seems like it could be confusing.

Option 2:

Start with a ml.recipe pseudo backend

rcp = (
ml.reicipe
.drop(["approved", "day"])
.mutate(day=_.cast("string"))
.mutate(s.across(s.endswith("_id"), _.cast("string")))
.fill_na(s.of_type("string"))
.mutate(s.of_type("boolean"), _.cast("int8"))
.ordinal_encode(s.of_type("string"), min_frequency=0.01)
)

tr = rcp.fit(df)
Does this make sense?

This definitely would be possible! The way I see it, this is essentially syntactic sugar that uses more of a builder pattern for the recipe, whereas the existing declarative construction is more akin to what scikit-learn or tidymodels recipes does.

I would expect ibis-ml to be a lightweight extension of ibis as much as possible, but that doesn't seem to be the case. Ibis-ml does its own stuff which is not compatible with the core ibis.

Coming back to this question, I would say IbisML extends Ibis to add the fit-transform paradigm that's very common in ML pipelines, and, in this sense, it's a layer on top. What you're interested in/proposing is making this less of a separate layer, but I'm not sure right now how this would work while cleanly adding the additional capabilities (like storing state, doing things based on a target column, maintaining order where necessary, etc.).

I'd be curious if some of the other people who have worked on this (@jcrist @gforsyth @cpcloud @adatta92) may have thoughts, but this is my immediate reaction.

[vspinu]

If I understand correctly, the main benefit here would be that you could mix and match Ibis functions with extensions for ML preprocessing?

Essentially yes - all custom data transformations are then naturally intermixed with the ML framework transformations and the other way around. On the dev side, there would be no need to re-invent the wheel and re-implement data transformations of selectors on the ML side. Everything is 100% composable!

Yet 3rd option (and maybe I am over simplifying) - one potential solution in order to bridge the composability in the current implementation is to allow arbitrary stateless ibis transformations as part of the ibisML pipeline by having a dedicated .pipe step which would take an argument ibis lazy statement. This way IbisML potentially would not need to bother with stuff like select, drop, mutate etc. Still, I think there might be a much bigger opportunity here to do the ML stuff differently from what everyone else does.

I think the part that feels unclear to me is you're saving the result as rcp—some Recipe-like object—which happened to be returned because you're calling ML extensions, whereas if you'd done "normal" Ibis transformations, it would have still been a table. That function-dependent behavior seems like it could be confusing.

To my mind recipe-like in ibisML and Defered/Table constructs in core Ibis are not that conceptually different. Both are lazy constructs which are essentially a sequence of computations. In case of ML pipelines those are often relying on some state precomputed from the train steps (state which can includes target column designation).

So the .fit() enhancement to the Table would run through the data and update the state of the ML transforms accordingly, still returning the core ibis Table (or a sub-class).

The way I see it, this is essentially syntactic sugar that uses more of a builder pattern for the recipe, whereas the existing declarative construction is more akin to what scikit-learn or tidymodels recipes does.

To my eye it's actually 100% declarative because it's a lazy sequence of computations which could be inspected upon and optimized at the compute time.

For tidymodels this is a missed opportunity IMHO, they could have specialized dplyr constructs on recipe objects instead of re-implementing them from scratch. Or maybe even better, do the other way around and blend recipes into the dplyr world by making recipes dply transforms. They already had lazy ibis-style DB constructs in place. But of course it's way more tricky for them as they already start with eager data.frame-only pipelines which need to be supported. With ibis there is no such baggage AFAICJ.