TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

[pmatsibekker]

Click to expand!

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

binary

Tensorflow Version

v1.12.1-88869-g80170ee25b4 2.12.0-rc0

Custom Code

No

OS Platform and Distribution

WIndows 11

Mobile device

No response

Python version

3.11.2

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

I am running the keras training module for mnist from tensorflow_datasets and want to save the model.
this is failing with error message below. I looked at the file thats triggering data_structures.py and noticed the super classes don't have the __getattribute__ method. I do see __getattr__.
Is this a version difference. My site-packages identified wrapt as wrapt-1.15.0-dist-info. Is there a version incompatability? If so, which version do I use?

Standalone code to reproduce the issue

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

import tensorflow_datasets as tfds
import keras

(ds_train, ds_test), ds_info = tfds.load(
    'mnist',
    split=['train', 'test'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True,
)

def normalize_img(image, label):
  """Normalizes images: `uint8` -> `float32`."""
  return tf.cast(image, tf.float32) / 255., label

ds_train = ds_train.map(
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
ds_train = ds_train.cache()
ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
ds_train = ds_train.batch(128)
ds_train = ds_train.prefetch(tf.data.AUTOTUNE)

ds_test = ds_test.map(
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
ds_test = ds_test.batch(128)
ds_test = ds_test.cache()
ds_test = ds_test.prefetch(tf.data.AUTOTUNE)


model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10)
])
model.compile(
    optimizer=tf.keras.optimizers.Adam(0.001),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
)


model.fit(
    ds_train,
    epochs=6,
    validation_data=ds_test,
)

tf.saved_model.save(
    model,
    'saved/save_files'
)

Relevant log output

WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
Traceback (most recent call last):
  File "C:\Users\pmats\PycharmProjects\pythonProject\minst.py", line 52, in <module>
    tf.saved_model.save(
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 1240, in save
    save_and_return_nodes(obj, export_dir, signatures, options)
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 1276, in save_and_return_nodes
    _build_meta_graph(obj, signatures, options, meta_graph_def))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 1455, in _build_meta_graph
    return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 1408, in _build_meta_graph_impl        
    saveable_view = _SaveableView(augmented_graph_view, options)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 281, in __init__
    self._initialize_save_and_restore_functions()
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\saved_model\save.py", line 303, in _initialize_save_and_restore_functions
    save_util_v1.get_checkpoint_factories_and_keys(self.object_names))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\checkpoint\save_util_v1.py", line 75, in get_checkpoint_factories_and_keys
    saveable_object_util.saveable_objects_from_trackable(
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py", line 614, in saveable_objects_from_trackable
    if trackable_has_serialize_to_tensor(obj):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py", line 745, in trackable_has_serialize_to_tensor
    if "_serialize_to_tensors" in obj.__dict__:
                                  ^^^^^^^^^^^^
  File "C:\Users\pmats\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorflow\python\trackable\data_structures.py", line 823, in __getattribute__      
    return super().__getattribute__(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

[pjpratik]

@pmatsibekker Thanks for reporting the issue.

We see that you are using 1.x version which is not supported anymore.

I was able to run the code without any error on TF 2.11. Please find the gist here and let us know if it helps.

Thank you.

[NiharJani2002]

@pmatsibekker I guess this might work. Does it help to you, or still it persist.

In Python, objects have a special attribute called dict, which is a dictionary that stores the object's attributes. However, there are some special objects called _DictWrapper that don't support the dict attribute. If you try to access dict on a _DictWrapper object, you'll get a TypeError.
To fix this error, you can try to access the wrapped attribute of the _DictWrapper object, which will give you access to the underlying object's dictionary. Or, you can try to avoid using dict on _DictWrapper objects and instead use the methods provided by the wrapper to access and manipulate its attributes.

Code:
wrapped_obj = my_dict_wrapper.wrapped
wrapped_obj_dict = wrapped_obj.dict

[treddis]

Solved after removing saving model callback. It seems that error consists in model preservation procedure, which handled via transparent dictionary wrapper incorrectly.
As temporary solution (for those who faced this problem) saving model data can be done through model.save_weights

[lithium0003]

This issue also happens in version 2.12.0-rc0 on ubuntu
This code produces error.
tf.keras.callbacks.BackupAndRestore may be broken.

import tensorflow as tf

mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

callbacks = [
    tf.keras.callbacks.BackupAndRestore('backup'),
]

model.fit(x_train, y_train, epochs=5, callbacks=callbacks)

Epoch 1/5
1854/1875 [============================>.] - ETA: 0s - loss: 0.2973 - accuracy: 0.9132Traceback (most recent call last):
  File "/home/ubuntu/mnist.py", line 23, in <module>
    model.fit(x_train, y_train, epochs=5, callbacks=callbacks)
  File "/home/ubuntu/venv/tensorflow2.12.0rc0/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/ubuntu/venv/tensorflow2.12.0rc0/lib/python3.10/site-packages/tensorflow/python/trackable/data_structures.py", line 823, in __getattribute__
    return super().__getattribute__(name)
TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

[lithium0003]

tf_nightly-2.13.0.dev20230302 works fine same code.

[lithium0003]

b016eb2

wrapt >= 1.11.0, <1.15 works, so tf_nightly works fine

[pmatsibekker]

upgrade worked. python had a hard time letting go of the version 1.x tensorflow library but a wipe and rebuild worked. thank you!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[julianstrietzel]

In the solving commit it is mentioned that the error has to be understood and solved more in depth. Maybe the issue and discussion on the wrapt repo can be helpful.

[GrahamDumpleton]

At this point I believe a more appropriate fix is to change the __getattribute__() override to:

  def __getattribute__(self, name):
    import types

    #if (hasattr(type(self), name)
    #    and (isinstance(getattr(type(self), name), property)):

    if hasattr(type(self), name):
        obj = getattr(type(self), name)

        if isinstance(obj, property):
            return object.__getattribute__(self, name)
        elif type(obj) == types.MappingProxyType and name == "__dict__":
            raise AttributeError("'mappingproxy' object has no attribute '__dict__'")
        else:
            return super().__getattribute__(name)
    else:
      return super().__getattribute__(name)

The change in wrapt 1.15.0 was to fix a bug whereby exceptions were suppressed by a wrapper and not raised when doing attribute access. That these exceptions are now raised has subtly changed how __gettattribute__() in Python may see things, having implications for tensorflow as a result. Best guess I can make, not really understanding what tensorflow wrapper does, is to have it explicitly check for mappingproxy type and raise an explicit AttributeError so that higher levels of Python internal __getattribute__() flow through and do what else it has to do properly rather than raising an exception. Can't see any other options at this point as overriding __getattribute__() can get quite gnarly.

Really need people in the tensorflow community to check this change with all your test suites and verify things still work as expected. I can't really do much else as have no idea what those wrappers in tensorflow do.

For more details see GrahamDumpleton/wrapt#231.

[GrahamDumpleton]

BTW, forgot to highlight this, but the trigger for this is that the exception that tensorflow raises:

TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

uses TypeError when it arguably should be raising AttributeError.

If it changed to AttributeError, this likely would all work without needing to change __getattribute__() to add a special check for __dict__ access on a mappingproxy.

[jobsZhao99]

b016eb2

wrapt >= 1.11.0, <1.15 works, so tf_nightly works fine

Thanks!

It works for me when I download to 1.14.1 in conda using:
conda install -c conda-forge wrapt=1.14.1