Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pickle and threading conflict #1968

Open
Beuc opened this issue Aug 4, 2019 · 6 comments

Comments

@Beuc
Copy link
Contributor

commented Aug 4, 2019

Hi,

I'm experimenting with fetching URLs in the background so as not to block the game on network error.

I'm getting some success but I get all kind of weird issues on saving as soon as I start a thread, even if I exclude it from pickling.

Here's a reasonably simple test case that occurs on second save; I have a more complex download example where the same error happens on first save. Seems like starting a thread somewhat taints the environment?

init python:
    import threading, time
    class Test:
        def __init__(self):
            self.thread = None
        def __getstate__(self):
            self.thread = None
            return self.__dict__
        def send(self, ignored):
            self.thread = threading.Thread(target=lambda:time.sleep(1))
            self.thread.start()
        def isAlive(self):
            return self.thread and self.thread.isAlive()


label start:
    $ config.use_cpickle = False
    "Start"
    $ t = Test()
    $ t.send('')
    show text "request in progress"
    while t.isAlive():
        pause 0.1
    hide text
    "Save here (OK)"
    "Then save here (KO)"
    "End"
  File "/home/tom/ab/x64lucid-deps/install/lib/python2.7/pickle.py", line 817, in whichmodule
  File "/home/tom/ab/x64lucid-deps/install/python/pygame_sdl2/__init__.py", line 33, in __getattr__
NotImplementedError: Could not import pygame_sdl2.mixer: No module named mixer

Adding self.thread=None in isAlive can also trigger the issue (while it should lessen its probability).

(also sadly this technique doesn't help when the user rollsback, ren'py becomes unresponsive)

Any clue on what is causing the exception?

@jsfehler

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2019

How about:

    class Test:
        def __init__(self):
            self.thread = None
        def __getstate__(self):
            self.thread = None
            return self.__dict__
        def send(self, ignored):
            renpy.invoke_in_thread(lambda:time.sleep(1))
        def isAlive(self):
            return self.thread and self.thread.isAlive()
@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2019

That sounds interesting.
Question is: how do we know when the thread is finished? With this Test.isAlive is broken.

@renpytom

This comment has been minimized.

Copy link
Member

commented Aug 4, 2019

You're not allowed to store a thread (or anything else unpickleable, like iterators) in the Ren'Py store. I'd suggest not storing the thread - use invoke_in_thread to call a function or method, and have that
update some piece of data when th callable finishes.

@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2019

I'm getting some success but I get all kind of weird issues on saving as soon as I start a thread, even if I exclude it from pickling.

@renpytom

This comment has been minimized.

Copy link
Member

commented Aug 4, 2019

Okay, how are you excluding it from pickling? Since in your example, you create a Test object with a thread field, and then refer to that from t. So that will cause it to be pickled. It's pretty hard to safely create a thread from the Ren'Py context, but something like:

init python:
     class MyThread(object):
           def __init__(self)
                 self.done = False
                 t = threading.Thread(target=self.run)
                 t.start()
           def run(self):
                time.sleep(10)
                self.done = True
@Beuc

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2019

With __getstate__ (doc), sorry if that wasn't obvious.

        def __getstate__(self):
            self.thread = None
            return self.__dict__

which incidentally fixed some occurrences of the error, but not all.

Also the error is quite different from what you get when pickling a thread (which would be "can't pickle lock objects").
(As for my project I implemented some work-around with invoke_in_thread meanwhile)

I smell a bug of sorts, hence this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.