Deepcopy failed on _thread.lock

[gjelsas]

I'm trying to build a little board game with a reasonably smart computer opponent. For that, I need deepcopy to find the best next move. Using copy.deepcopy gives me the following error. Any hints for getting this to work would be great!

Python 3.10 on Ubuntu is in use here...

Exception in thread Thread-9 (handler):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/georg/.virtualenvs/SteinchenSpiel/lib/python3.10/site-packages/flet/event_handler.py", line 16, in handler
h(r)
File "/home/.../gui/MuldenButton.py", line 45, in spiele
self.board.lasse_computer_ziehen()
File "/home/.../gui/Spielbrett.py", line 37, in lasse_computer_ziehen
self.computer_spieler.spiele_zug_mit_meisten_steinen_am_ende()
File "/home/.../Spieler/AutomatischerSpieler.py", line 17, in spiele_zug_mit_meisten_steinen_am_ende
sf = deepcopy(self.get_spielfeld())
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
.
.
.
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object

I skipped some reoccurring lines

[ndonkoHenri]

Have you tried googling the error?
https://www.google.com/search?q=cannot%20pickle%20%27_thread.lock%27%20object

[gjelsas]

Have you tried googling the error? https://www.google.com/search?q=cannot%20pickle%20%27_thread.lock%27%20object

Yes, in fact I did google it, but I'm not sure if it's a general thing caused by the multiprocess nature of flet.

When I run the Program without GUI, the code is working. When the method with utilizes the deepcopy is called from a flet GUI Button, it crashes with the error mentioned above.

I guess I need to find a way to omit _thread.lock from being copied or run the whole method single threaded. But I have no clue so far, where to start this endeavor.

[FeodorFitsner]

Could you do a simple repro, so we could investigate?

[gjelsas]

Here a little working example. The problem is the deepcopy of the ListElementClass object, which is a flet.Text object, I think.

import flet as flet
from flet import Page, AnimatedSwitcher, Text, Container


class ListElementClass(Text):  # Element to be altered inside copied list
    def __init__(self):
        super().__init__(value="Push me!")


list = []
for _ in range(10):
    list.append(ListElementClass())  # Mache Elemente flet Object...


class OtherClass(AnimatedSwitcher):
    class InnerClass(Container):
        def __init__(self, methode):  # Constructor of InnerClass
            super().__init__(content=Text(value=list[0].value), on_click=methode)

    def __init__(self):  # Constructor of OuterClass
        super().__init__(content=OtherClass.InnerClass(methode=self.deepcopy_methode))

    def deepcopy_methode(self, e):
        from copy import deepcopy
        print("Methode invoked")
        copy_of_list = deepcopy(list)


def main(page: Page):
    oc = OtherClass()  # Creating the construct
    page.add(oc)  # adding to the site


flet.app(target=main)

By producing this example code, I thought that maybe I should keep the values outside any flet object and store them in a different datastructure to do the deepcopy on that datastructure...

@FeodorFitsner And a word of Thank You for this nice project! I really appreciate your work here.

[gjelsas]

So I came up with a solution to this issue. There is a class which is subject to the deepcopy process. This class itself has an attribute self.board which is a flet object. By excluding board from deepcopy, things work like a charm.

def __deepcopy__(self, memo):
        from copy import deepcopy
        cls = self.__class__
        result = cls.__new__(cls)
        memo[id(self)] = result
        for k, v in self.__dict__.items():
            if k != 'board': # here is the trick
                setattr(result, k, deepcopy(v, memo))
        return result

I hope my failure will be of help to anybody else. Thanks again for the project and the fast response!

[ndonkoHenri]

Issue seems closed. Did you manage to get it fixed?

[gjelsas]

Issue seems closed. Did you manage to get it fixed?

Well, I exclude objects of flet classes from the deepcopy by overriding the __deepcopy__ methode (see #565 (comment)) . As I was only interested in other objects which happened to be connected to flet objects, this was a solution to my personal problem.

Deepcopy of flet Elements is still not possible. I don't know where to start investigating this issue, so I just excluded flet-objects in my project. I created an even simpler Example to show the issue:

import flet as flet
from flet import Page, AnimatedSwitcher, Text, Container


class SomeClass(AnimatedSwitcher):

    def __init__(self):
        super().__init__(content=Container(content=Text(value="Push me for Error!"), on_click=self.deepcopy_methode))

    def deepcopy_methode(self, e):
        from copy import deepcopy
        print("Methode invoked")
        list2 = [Text("A"), Text("B")]
        copy_of_list = deepcopy(list2)
        print("This is not printed...")


def main(page: Page):
    page.add(SomeClass())  # adding to the site


flet.app(target=main)

I don't know if reopening the issue would be any good, as flet objects maybe shouldn't be deepcopied at all?

[ndonkoHenri]

So, I just faced this same issue, and I actually confirm that flet controls can't be deepcopied. Please @FeodorFitsner investigate on this.
I made this basic code sample to help ease investigations:

import copy
import flet as ft

def main(page: ft.Page):
    text = ft.Text("Hello")
    mycopy = copy.deepcopy(text)

    page.add(
        ft.Text("Hello")
    )


ft.app(target=main)

Error:

TypeError: cannot pickle '_thread.lock' object

This issue should be reopened. If it can't then I can maybe create another one.

[FeodorFitsner]

We could provide a custom ___deepcopy__() implementation and bypass non-cloneable attributes: https://stackoverflow.com/a/50352368/1435891 - is anyone willing to help with that?

[ndonkoHenri]

I will give it a try.
Please what do you mean by "bypass non-cloneable attributes"?

[FeodorFitsner]

I mean when you implement Control.___deepcopy__() you choose what object fields to clone and you don't clone self.__lock and other such things.

[ndonkoHenri]

So, I am now on this issue and understand exactly from where the error comes.
I also now understand what you meant by "bypass non-cloneable attributes".
I found the non-cloneable attribute causing all the problems: '_lock': <unlocked _thread.lock object at 0x000002AEDF27A340>
This key, value pair is found in self.__dict__
I am willing to "bypass" it, but will like to know it's use, or if it won't cause any issue?

[FeodorFitsner]

those locks are needed for proper functioning.

[ndonkoHenri]

After investigations I happened to come out with this bit of code(which is to be added into the Control class - so every control inherits it):

    def __deepcopy__(self, memo):
        """
        It creates a new instance of the class, and then copies all the attributes of the old instance into the new one,
        except for those attributes that can't be deepcopied(ex: _lock).

        :param memo: A dictionary of objects already copied during the current copying pass
        :return: A deep copy of the object.
        """
        cls = self.__class__()
        memo[id(self)] = cls
        for k, v in self.__dict__.items():
            try:
                cls.__dict__[k] = copy.deepcopy(v, memo)
            except TypeError:
                pass
        return cls

I tested it on the code below and it worked like charm:

import copy
import flet as ft


def main(page: ft.Page):
    text = ft.Text(
        "Hello from Text",
        size=50,
        color=ft.colors.WHITE,
        bgcolor=ft.colors.GREEN_700,
        weight=ft.FontWeight.BOLD,
        italic=True,
    )

    deepcopy = copy.deepcopy(text)  # make a deepcopy

    print(f"{text=}, \n{deepcopy=}\n")
    print(f"{text.value=},\n{deepcopy.value=}\n")

    def modify(e):
        text.value = "Bye from Text"
        deepcopy.value = "Bye from DEEP Copy"
        page.update()
        print(f"{text.value=}, \n{deepcopy.value=}\n")

    page.add(
        ft.ElevatedButton("Modify", on_click=modify),
        text,
        deepcopy
    )

ft.app(target=main)

But the above solution I made doesn't seem to work as expected with complex controls (the DataTable control precisely, which is the only control I need to deepcopy in my usecase - PaginatedDT). I prepared a little code sample below:

import copy
import flet as ft


def main(page: ft.Page):
    page.theme_mode = "light"

    dt = ft.DataTable(
        width=700,
        bgcolor="yellow",
        border=ft.border.all(2, "red"),
        border_radius=10,
        vertical_lines=ft.border.BorderSide(3, "blue"),
        horizontal_lines=ft.border.BorderSide(1, "green"),
        sort_column_index=0,
        sort_ascending=True,
        heading_row_color=ft.colors.BLACK12,
        heading_row_height=100,
        data_row_color={ft.MaterialState.HOVERED: "0x30FF0000"},
        show_checkbox_column=True,
        divider_thickness=0,
        column_spacing=200,
        columns=[
            ft.DataColumn(
                ft.Text("Column 1"),
                on_sort=lambda e: print(f"{e.column_index}, {e.ascending}"),
            ),
            ft.DataColumn(
                ft.Text("Column 2"),
                tooltip="This is a second column",
                numeric=True,
                on_sort=lambda e: print(f"{e.column_index}, {e.ascending}"),
            ),
        ],
        rows=[
            ft.DataRow(
                [ft.DataCell(ft.Text("A")), ft.DataCell(ft.Text("1"))],
                selected=True,
                on_select_changed=lambda e: print(f"row select changed: {e.data}"),
            ),
            ft.DataRow([ft.DataCell(ft.Text("B")), ft.DataCell(ft.Text("2"))]),
        ],
    )

    mycopy = copy.deepcopy(dt)

    print(f"{dt=}, \n{mycopy=}\n")

    page.add(
        dt,
        mycopy
    )


ft.app(target=main)

The result could be seen below. The original on the top, and the deepcopied down (the line + checkbox in the middle)

I might keep investigating, but don't know what I can do to properly 'deepcopy' flet controls. I will appreciate some help.

[FeodorFitsner]

Implementation looks good to me - I'd probably start with that :)
I guess (just a guess) there are some non covered cases with DataTable, like inner collections? Don't know if copy.deepcopy() works agains them automatically. You can investigate on some simpler controls with collections like Row, etc.

[ndonkoHenri]

So, I found where the issue is from. I added a print in the except clause, so as to see the key value pairs that couldn't be deepcopied:

            except TypeError:
                print(k, v)
                # pass

And had this:

_lock <unlocked _thread.lock object at 0x00000142C1017D80>
_DataTable__columns [<flet.datatable.DataColumn object at 0x00000142C0FFEFD0>, <flet.datatable.DataColumn object at 0x00000142C10169D0>]
_lock <unlocked _thread.lock object at 0x00000142C1017440>
_DataRow__cells [<flet.datatable.DataCell object at 0x00000142C1016E90>, <flet.datatable.DataCell object at 0x00000142C10171D0>]
_lock <unlocked _thread.lock object at 0x00000142C1017C40>
_DataRow__cells [<flet.datatable.DataCell object at 0x00000142C1017710>, <flet.datatable.DataCell object at 0x00000142C1017A50>]

The building blocks of the DataTable apparently also raise a type error, and hence result in the image i sent above.

[ndonkoHenri]

I also went forward modifying the for loop as follows (in order to see the message in the TypeError raised by the Data** stuffs);

        for k, v in self.__dict__.items():
            try:
                cls.__dict__[k] = copy.deepcopy(v, memo)
            except TypeError as error:
                print(f"{error=}\n{k=}\n{v=}\n")

And had the below error:

error=TypeError("cannot pickle '_thread.lock' object")
k='_lock'
v=<unlocked _thread.lock object at 0x000001579E147C40>

error=TypeError("DataColumn.__init__() missing 1 required positional argument: 'label'")
k='_DataTable__columns'
v=[<flet.datatable.DataColumn object at 0x000001579E12F590>, <flet.datatable.DataColumn object at 0x000001579E146890>]

error=TypeError("cannot pickle '_thread.lock' object")
k='_lock'
v=<unlocked _thread.lock object at 0x000001579E147300>

error=TypeError("DataCell.__init__() missing 1 required positional argument: 'content'")
k='_DataRow__cells'
v=[<flet.datatable.DataCell object at 0x000001579E146D50>, <flet.datatable.DataCell object at 0x000001579E147090>]

error=TypeError("cannot pickle '_thread.lock' object")
k='_lock'
v=<unlocked _thread.lock object at 0x000001579E147B00>

error=TypeError("DataCell.__init__() missing 1 required positional argument: 'content'")
k='_DataRow__cells'
v=[<flet.datatable.DataCell object at 0x000001579E1475D0>, <flet.datatable.DataCell object at 0x000001579E147910>]

The error could be solved by modifying this line, so the content or label of is respectively added:

cls = self.__class__()

I need some help please.

[FeodorFitsner]

My thinking is that you have to override __deepcopy__() in DataColumn and DataCell classes as well.
And maybe filter copying of an attribute with _lock name, but instead create a new Lock explicitly.

[ndonkoHenri]

I think I have to make a PR so you see how things move.
If creating a new lock won't disturb, How can I create a new lock please?