-
Notifications
You must be signed in to change notification settings - Fork 996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add Mariadb offline store #4158
base: master
Are you sure you want to change the base?
Conversation
@tmihalac @jeremyary my 2 cents here, if I may 😄 I've written this elsewhere a couple times and of course I have nothing against this particular implementation, I just think that we will eventually run out of maintenance effort if we keep reimplementing similar offline store engines from scratch every time. I think it will take too much effort ensuring they work and have parity with one another, also adding a new feature in offline store logic means changes in too many places. As you might have guessed, I'd rather see all new offline stores (and ideally existing ones as well eventually) to be ibis-based to avoid all that, but that's just me... |
@tokoko |
@tmihalac python static methods and inheritance just don't get along, that has beaten me before as well :). Anyway, I didn't mean to ask you to come up with a generic implementation, just pointing out that we already have one here. it's written with ibis which has a mysql backend (I'm assuming it will work with mariadb as well). It's right now used only by duckdb implementation and figuring out how to use it from mariadb will still be quite a bit of work, but I'm still pushing for that approach. P.S. you can take a look at this RFC as well. |
@jeremyary and me talked before to try and use Sqlalchemy ORM for all the Offline Store DB support. |
It's very much like a ORM, most of it's backends used to be powered by sqlalchemy actually in previous versions, but they recenly switched to sqlglot. You can think of it as an abstraction above that. it exposes a DataFrame-like API and executes these dataframes on the selected backend, in most cases it is actually generating sql first to do that. Having both ibis and sqlalchemy-based "abstract" offline stores sounds wasteful to me. |
3a9e233
to
8d0112b
Compare
Added MariaDB offline store Added MariaDB template Signed-off-by: Theodor Mihalache <tmihalac@redhat.com>
8d0112b
to
82cb260
Compare
Signed-off-by: Theodor Mihalache <tmihalac@redhat.com>
@tmihalac do you still plan on doing this? |
What this PR does / why we need it:
Adds option to use MariaDB as an offline store