dev: ensure that downstream dependencies remain compatible with ibis 4.x

[kszucs]

We should ensure projects depending on ibis remain functional with all the breaking changes we merged with #4512

Pay attention to:

• the ibis adopters we know
• we have a couple of 3rdparty backends, like ibis-bigquery or ibis-heavyai
• popular projects we may not be aware of (@jcrist I know you have an ibis query and a datasource to check this out, could we use that here?)

[kszucs]

@jreback @icexelloss @timothydijamco I'm not sure where are you at the migration, but please let me know if I can help you with anything. I created a 3.x.x branch which is identical with the previous HEAD before any of the breaking changes.

[icexelloss]

Thanks! In term of migration status, we haven't started working on it. It's close to year-end so it's hard to squeeze in a major development effort like the ibis 3 -> 4 migration on our side (most people are busy with finishing up their year-end goal).

What are you guys thoughts around:
(1) How long do we plan to maintain 3.x branch. Is it going to be mostly in maintenance mode or do we plan to back port features to it?
(2) How long do we plan to maintain ibis 3.x compatibility on the ibis-substrait side? We just started porting ibis-substrait internally and our plan is to target ibis-substrait with ibis 3 for the next few month (3-4) at least. So curious to hear that you guys think about maintaining that and if we need to budget eng resources to keep the compatibility ourselves.

[cpcloud]

@icexelloss Thanks for chiming in!

The plan is to port nothing back to 3.x for now. Regarding ibis-substrait, how does 3-4 months sound for a cutover to 4.0? In the meantime, I can try to look into adding a shim layer for ibis-substrait that is backwards compatible so that 3.x and 4.x can live alongside each other for that period. After 3-4 months though, we'll set a lower bound to ibis 4.

How does that sound?

[icexelloss]

Sorry for the delayed reply but yeah I think it is reasonable but I cannot promise we will be 100% ready to go with ibis 4.x in 3-4 month time (it depends on how much work is needed for bumping to ibis 4 internally and when we can schedule the work) but I think it's a reasonable target.

[kszucs]

Just as a sidenote: the migration may seem like a big effort, but the majority of the incompatibilities can be resolved by removing the .op() calls.

[icexelloss]

Hopefully..(finger crossed)

[kszucs]

What I mean it may worth a try sooner than later. I'll be harder for us too to recall how to resolve the more complicated scenarios.

[saulpw]

@kszucs Can we add a def op(self): return self just for backwards compatibility? We can deprecate it straightaway, just seems like it might ease the migration substantially if this will cause a majority of the issues.

[kszucs]

Definitely, that's a pretty good idea actually! Would you mind putting up a PR for it?

[cpcloud]

Closing this out, the op method exists for compatibility now.