Persisting a database session?

[dpavlic]

Hello,

In my project, I have a need to persist the same database connection through different nodes.

In traditional Python code, I have a db connection that persists for the entire stage of the "pipeline": the temporary tables I write to in the beginning are then referenced through all the latter stages in the pipeline. Refactoring this into kedro, I want essentially the same thing, have one persistent db connection that is initialized, and then all of the nodes are referencing this same connection, without exiting it.* How could I go about getting this accomplished?

• This is necessary because I only have write permissions for temporary tables on this database. As soon as the connection is closed, the temporary tables are dropped.

[datajoely]

Hi @dpavlic this is a good question and something that I'm not sure is possible today. I think you may be able to hack this together with a Kedro hook - but this isn't going to be pretty.

Better SQL support in general is something I want to tackle later this year so I'll add this to our key requirements for that piece of work.

[datajoely]

I also wonder if this can be applied/extended to the current dataset implementation
https://docs.sqlalchemy.org/en/14/core/pooling.html