[BUG] cannot pickle 'SSLContext' with background callback

[fxstempfelals]

Describe your context

Windows 11, Python 3.9

dash                      2.16.1                                                                                               
dash-bootstrap-components 1.4.1                                                                                                
dash-core-components      2.0.0                                                                                                
dash-html-components      2.0.0                                                                                                
dash-table                5.0.0

Occurs with Edge & Chrome.

Describe the bug

I use a background callback in a multi-page app that throws an error when called:

• TypeError: cannot pickle 'SSLContext' object in the browser (without more details, see screenshot below)
• in the standard output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\201021305\.conda\envs\track\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\201021305\.conda\envs\track\lib\site-packages\multiprocess\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Users\201021305\.conda\envs\track\lib\site-packages\dill\_dill.py", line 289, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "C:\Users\201021305\.conda\envs\track\lib\site-packages\dill\_dill.py", line 444, in load
    obj = StockUnpickler.load(self)
EOFError: Ran out of input

Here's how the callback is defined:

BACKGROUND_CALLBACK_MANAGER = dash.DiskcacheManager(diskcache.Cache("./cache"))
@dash.callback(
    [...]
    prevent_initial_call=True,
    background=True,
    manager=BACKGROUND_CALLBACK_MANAGER,
)

The callback involves an object that has a SQLAlchemy engine as an attribute. The connection is made through SSL, so I guess this is the object that fails to be pickled. However, I can serialize this object successfully with dill.dumps, so I'm not sure...

Maybe related to uqfoundation/dill#308, but until the issue is fixed, there might be a workaround?

Expected behavior

I expect the callback to run without error.

Screenshots

[T4rk1n]

Are you using the SQLAlchemy engine defined globally elsewhere?
The multiprocess will try to transfer it when used but that is not valid pickle/multiprocess object. May need to establish the engine inside the callback for that to work or switch to celery.

[fxstempfelals]

The engine is not a global object but a property of an object created in the callback. Roughly:

class DBHandle:
    def __init__(self, url, schema):
        connect_args = {...}
        self.engine = sqlalchemy.create_engine(
            url,
            echo=True,
            pool_pre_ping=True,
            connect_args=connect_args,
            execution_options={"schema_translate_map": {None: schema}},
        )

@dash.callback(
    [...]
    prevent_initial_call=True,
    background=True,
    manager=BACKGROUND_CALLBACK_MANAGER,
)
def _callback():
    dbh = DBHandle(url, schema)

I'll try with Celery if I can't find a solution but it would be nice and make development easier if it could work with a local cache.

[T4rk1n]

Could you try running your app with:

app.run(debug=True, dev_tools_prune_errors=False)

With that option the stacktrace in the screenshot should have more info to help identify which variable is causing the issue.

[fxstempfelals]

Thanks for the suggestion. After some digging, I realized the SSLContext instance is a nested property of botocore.client.S3. This S3 client is not used in the background callback that is the source of the error, but elsewhere in the app. I guess dash serializes the whole execution environment?

[T4rk1n]

I guess dash serializes the whole execution environment?

Yes, we use multiprocess to spawn the background process and that will try to transfer all the global variables.