comparison to ibis? consider pep 0827 support?

[NickCrews]

Hi! Thanks for linking here [from this duckdb-python](duckdb/duckdb-python#370 (comment) discussion)! This is a cool project, nice job so far!

two things:
Have you seen ibis? It would be useful to put in your readme how this is different from ibis. Or, we at ibis would love to hear why ibis isn't good enough for your needs, so that we can improve. Finally, the biggest stretch: If we could convince you that ibis was a better investment of your energy, we would love to have you contribute there and unify our efforts!

second thing:
The real missing feature that in MY opinion ibis is terrible at is typing. If you do keep going with this, the reason that I might switch to a different dataframe library is for excellent typing support. For example, tracking the schema of individual columns as a relations goes through the transofrmations. This isn't currently possible with python's type system, but https://peps.python.org/pep-0827/ tries to fix that. Ibis is way to big and mature at this point for me to really want to take on migrating to this typing feature, but you might havehope here! So I guess I'm just suggesting you keep that PEP in mind, maybe experiment with it here, and keep your architecture of this lib open for adding this typing support in the future if that PEP is accepted.

[OutSquareCapital]

Thanks for the kind words! Really appreciate it :).

About the first thing

Honestly I heard about Ibis, but saw an API that had nothing who looked like Polars when I looked at it and so instantly stopped considering it.
But it 's very weird, because I just took a look at this https://ibis-project.org/tutorials/basics and it is in fact very similar to polars minus a few minor things (mutate vs with columns for ex).
Maybe I was VERY bad at reading that day, or on the wrong website, or Ibis had API changes since the last year, IDK.
Anyways, I will mention it in the README, add it in my test suite and the comparison file

This way I will be able to answer the second point more precisely in a future follow-up :). I already updated the README to mention it. I relied on AI slop for the ibis section, will update it once I have my aforementioned follow-up conclusion.

But what I can say right now is that "no other backend compatibility limitation" will surely end up being the key point.
For example, I dynamically generate 8K LOC in this file and plan to expose them all in the public API in the future.
Idk if those DuckDB functions are supported in other databases, and by extension supported by Ibis.

On the third point about contribution, If I'm convinced sure!
The biggest barrier to this is not linked to this library but by my code style.
You will see that I love method chaining, and handle all iterations/null/result values with my library.
Idk why but that's my nerdy special interest with coding haha and I'm not sure this would work that well in a big library with dozens other contributors who might not be accustomed with this (+ additional dependency).

About the second thing

Excellent typing support is indeed a hard requirement for me. Idk if I can go more pedantic than with basedpyright ALL and Ruff ALL compliance.
BTW I have a PR improving a lot of things typing-wise waiting for review in the DuckDB repo.
If it lands, Ibis will potentially benefit from it. I invite you to keep an eye on this!

I'm not finished yet with the reading of the PEP you mentionned, but so far I'm very convinced by it. If it is accepted, I will for sure support it.
FYI, the schema tracking thing is something that dataframely does very well.
You can make your dataframe generic, with a lot of custom rules, etc... it's really great.
Unfortunately, it's polars specific, and don't plan on expanding to other libraries when I asked about it

dataframely is what inspired me to start writing framelib

For the full origin story of framelib and pql, I always wanted to try DuckDB, so I started using SQLAlchemy first for it, but had a few weird bugs, crashes, and when I saw that I was forced to use a primary key when it's not a requirement in DuckDB I concluded that I was kinda meh for it.

Then I stumbled upon narhwals and dataframely and was already addicted by the polars API, so why not trying to combine them? Thus I started working on framelib.
Then I realized that narwhals was too limited for complex queries for DuckDB , and I started writing pql.

End goal is to connect both when pql is more fleshed out and I can start working again on framelib.
pql would be specifically for duckdb as a Ibis/SQLFrame/narwhals on steroids for the expression/relations side,
and framelib would be the backend agnostic connexion/file/schema/validation side.

[NickCrews]

This is great to hear your story and what you liked and didn't like about each thing, thanks for the detailed response!

Maybe I was VERY bad at reading that day,

I know what you mean, I have been hearing about polars forever and always think I misunderstood what it was until very recently. Funny how easy it is for that to happen!

You will see that I love method chaining, and handle all iterations/null/result values with

This is cool. I think the thing I am most compelled by is that I don't like the lack of typesafety with vanilla exceptions in python. Does Result[T, E] actually give me typesafety, like if I don't check for all error types then it typechecks?

Yeah, Ibis isn't going to adopt that though :) We have too many things implemented already. Still, I'd love to hear any bits of feedback on what ibis should improve on!

BTW I have a PR improving a lot of things typing-wise waiting for review in the duckdb/duckdb-python#352.

Nice, this is going to be awesome! If you want help shephering that in, give me a tag. I'm motivated for better typing there as well.

FYI, the schema tracking thing is something that dataframely does very well.

I think the one thing that NONE of the existing DF libraries do is to actually track schema changes through modifications. eg t.select("foo", "bar") now should use type inference to extract only the "foo" and "bar" columns from the source table. All of the existing libraries only allow you to define the schema at the start or the end of the transformation pipeline. For the THAT is going to be the killer feature that would convince me to switch df libraries.

If you're interested in what that could look like and the developer experience it could provide, check out my experiments at https://github.com/NickCrews/tybis, where I'm trying to do this in typescript.