Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

working with unserializable objects #10

Open
edublancas opened this issue Nov 17, 2021 · 3 comments
Open

working with unserializable objects #10

edublancas opened this issue Nov 17, 2021 · 3 comments

Comments

@edublancas
Copy link
Contributor

e.g., the notebook uses a db connection:

conn = open_db_connection()

We should not serialize this object, but rather embed this line in any
of the tasks that use it. How do we know if serialize it or not? Easiest way
is to have the user add a "preparation" cell, and then add it if any of the tasks
use the variables as inputs

@idomic
Copy link
Contributor

idomic commented Jun 16, 2022

In short, what happens today is that soorgeon throws an error on global variables.
We want to:

  • Check if there are any (code exist already).
  • Check if the vars are serialzible (as mentioned above).
  • Instead of throwing an error embed those vars where they're being used.
  • Adding the necessary tests and doc updates

@edublancas clarification on the 2nd point, we don't serialize/reinstantiate the objects to maintain the state? (i.e in the db_conn we don't really care what happened before). Ideally the users won't need to take this extra step - it should happen for them.

@edublancas edublancas changed the title Global variables working with unserializable objects Jun 17, 2022
@edublancas
Copy link
Contributor Author

I changed the title since it wasn't accurate. this isn't about all types of global variables. but about variables defined in the notebook that do not support serialization.

A typical example is connections to a database or logging objects. The problem is that we won't know if the notebook contains non-serializable objects because soorgeon only performs static analysis, it never runs the code.

@idomic
Copy link
Contributor

idomic commented Jun 17, 2022

Keeping this for a later stage, until we'll have a run function that can run the user's code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants