Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache_hash can give the wrong hash code for deserialized objects #482

Closed
rgabbard opened this Issue Jan 11, 2019 · 6 comments

Comments

Projects
None yet
3 participants
@rgabbard
Copy link
Contributor

rgabbard commented Jan 11, 2019

I just realized a bug in the code I wrote for hash code caching (#426). Because the hash code cache field gets serialized and deserialized by Pickle, when you deserialize a cache_hash=True attrs object, the hashcode will be the hashcode the object had at serialization-time. However, if your object has fields with hash codes which are not deterministic between interpreter runs, then on a new interpreter run your deserialized object will have a hash code which differs from a newly created identical object.

We can fix this for pickle by recomputing the hash code in __setstate__. Other serialization libraries which don't respect __setstate__ will still have a problem, but I don't think we can do anything about that. If the __setstate__ solution sounds acceptable I will implement it next week.

@hynek

This comment has been minimized.

Copy link
Member

hynek commented Jan 12, 2019

I think it would be more elegant to clear the cache – would that be feasible?

@wsanchez wsanchez added the Bug label Jan 12, 2019

@rgabbard

This comment has been minimized.

Copy link
Contributor Author

rgabbard commented Jan 14, 2019

Good point - I will just clear the cache

@hynek hynek added this to the 19.1.0 milestone Jan 17, 2019

@hynek

This comment has been minimized.

Copy link
Member

hynek commented Jan 20, 2019

bump? :)

@rgabbard

This comment has been minimized.

Copy link
Contributor Author

rgabbard commented Jan 22, 2019

Last week was busier than expected, but I hope to complete this tomorrow

rgabbard added a commit to rgabbard/attrs that referenced this issue Jan 22, 2019

Don't cache hash codes across deserialization.
 Because the hash code cache field gets serialized and deserialized by
 Pickle, previously when you deserialize a cache_hash=True attrs object, the
 hashcode will be the hashcode the object had at serialization-time.
 However, if your object had fields with hash codes which were not
 deterministic between interpreter runs, then on a new interpreter run
 your deserialized object would have a hash code which differs from a
 newly created identical object.

 This commit fixes that by clearing the cache on deserialization. It
 needs to override the __setstate__ method to do so, so this commit
 also forbids using a custom __setstate__ on a cache_hash=True object.

Closes python-attrs#482 .

@rgabbard rgabbard referenced this issue Jan 22, 2019

Merged

Clear cache hash on de-serialization #489

10 of 10 tasks complete

rgabbard added a commit to rgabbard/attrs that referenced this issue Jan 22, 2019

Don't cache hash codes across deserialization.
 Because the hash code cache field gets serialized and deserialized by
 Pickle, previously when you deserialize a cache_hash=True attrs object, the
 hashcode will be the hashcode the object had at serialization-time.
 However, if your object had fields with hash codes which were not
 deterministic between interpreter runs, then on a new interpreter run
 your deserialized object would have a hash code which differs from a
 newly created identical object.

 This commit fixes that by clearing the cache on deserialization. It
 needs to override the __setstate__ method to do so, so this commit
 also forbids using a custom __setstate__ on a cache_hash=True object.

Closes python-attrs#482 .
@rgabbard

This comment has been minimized.

Copy link
Contributor Author

rgabbard commented Jan 22, 2019

@hynek : This is ready in #489

@hynek hynek closed this in d2be76c Feb 2, 2019

bors bot added a commit to mozilla/normandy that referenced this issue Mar 4, 2019

Merge #1784
1784: Scheduled weekly dependency update for week 09 r=peterbe a=pyup-bot






### Update [attrs](https://pypi.org/project/attrs) from **18.2.0** to **19.1.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 19.1.0
   ```
   -------------------

Backward-incompatible Changes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Fixed a bug where deserialized objects with ``cache_hash=True`` could have incorrect hash code values.
  This change breaks classes with ``cache_hash=True`` when a custom ``__setstate__`` is present.
  An exception will be thrown when applying the ``attrs`` annotation to such a class.
  This limitation is tracked in issue `494 &lt;https://github.com/python-attrs/attrs/issues/494&gt;`_.
  `482 &lt;https://github.com/python-attrs/attrs/issues/482&gt;`_


Changes
^^^^^^^

- Add ``is_callable``, ``deep_iterable``, and ``deep_mapping`` validators.

  * ``is_callable``: validates that a value is callable
  * ``deep_iterable``: Allows recursion down into an iterable,
    applying another validator to every member in the iterable
    as well as applying an optional validator to the iterable itself.
  * ``deep_mapping``: Allows recursion down into the items in a mapping object,
    applying a key validator and a value validator to the key and value in every item.
    Also applies an optional validator to the mapping object itself.

  You can find them in the ``attr.validators`` package.
  `425 &lt;https://github.com/python-attrs/attrs/issues/425&gt;`_
- Fixed stub files to prevent errors raised by mypy&#39;s ``disallow_any_generics = True`` option.
  `443 &lt;https://github.com/python-attrs/attrs/issues/443&gt;`_
- Attributes with ``init=False`` now can follow after ``kw_only=True`` attributes.
  `450 &lt;https://github.com/python-attrs/attrs/issues/450&gt;`_
- ``attrs`` now has first class support for defining exception classes.

  If you define a class using ``attr.s(auto_exc=True)`` and subclass an exception, the class will behave like a well-behaved exception class including an appropriate ``__str__`` method, and all attributes additionally available in an ``args`` attribute.
  `500 &lt;https://github.com/python-attrs/attrs/issues/500&gt;`_
- Clarified documentation for hashing to warn that hashable objects should be deeply immutable (in their usage, even if this is not enforced).
  `503 &lt;https://github.com/python-attrs/attrs/issues/503&gt;`_


----
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/attrs
  - Changelog: https://pyup.io/changelogs/attrs/
  - Homepage: https://www.attrs.org/
</details>





### Update [botocore](https://pypi.org/project/botocore) from **1.12.101** to **1.12.106**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.12.106
   ```
   ========

* api-change:``ec2``: Update ec2 client to latest version
* api-change:``autoscaling-plans``: Update autoscaling-plans client to latest version
   ```
   
  
  
   ### 1.12.105
   ```
   ========

* api-change:``ssm``: Update ssm client to latest version
* api-change:``apigatewayv2``: Update apigatewayv2 client to latest version
* api-change:``alexaforbusiness``: Update alexaforbusiness client to latest version
* api-change:``application-autoscaling``: Update application-autoscaling client to latest version
   ```
   
  
  
   ### 1.12.104
   ```
   ========

* api-change:``waf-regional``: Update waf-regional client to latest version
* api-change:``waf``: Update waf client to latest version
   ```
   
  
  
   ### 1.12.103
   ```
   ========

* api-change:``discovery``: Update discovery client to latest version
* api-change:``organizations``: Update organizations client to latest version
* api-change:``resource-groups``: Update resource-groups client to latest version
* api-change:``opsworkscm``: Update opsworkscm client to latest version
* api-change:``pinpoint``: Update pinpoint client to latest version
* api-change:``mediaconvert``: Update mediaconvert client to latest version
* api-change:``cur``: Update cur client to latest version
   ```
   
  
  
   ### 1.12.102
   ```
   ========

* api-change:``elbv2``: Update elbv2 client to latest version
* api-change:``mediastore``: Update mediastore client to latest version
* api-change:``ce``: Update ce client to latest version
* api-change:``autoscaling``: Update autoscaling client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/botocore
  - Changelog: https://pyup.io/changelogs/botocore/
  - Repo: https://github.com/boto/botocore
</details>





### Update [cffi](https://pypi.org/project/cffi) from **1.12.1** to **1.12.2**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/cffi
  - Changelog: https://pyup.io/changelogs/cffi/
  - Docs: http://cffi.readthedocs.org
</details>





### Update [idna](https://pypi.org/project/idna) from **2.6** to **2.8**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.8
   ```
   ++++++++++++++++

- Update to Unicode 11.0.0.
- Provide more specific exceptions for some malformed labels.
   ```
   
  
  
   ### 2.7
   ```
   ++++++++++++++++

- Update to Unicode 10.0.0.
- No longer accepts dot-prefixed domains (e.g. &quot;.example&quot;) as valid.
  This is to be more conformant with the UTS 46 spec. Users should
  strip dot prefixes from domains before processing.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/idna
  - Changelog: https://pyup.io/changelogs/idna/
  - Repo: https://github.com/kjd/idna
</details>





### Update [pyflakes](https://pypi.org/project/pyflakes) from **2.1.0** to **2.1.1**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pyflakes
  - Changelog: https://pyup.io/changelogs/pyflakes/
  - Repo: https://github.com/PyCQA/pyflakes
</details>





### Update [google-api-core](https://pypi.org/project/google-api-core) from **1.7.0** to **1.8.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/google-api-core
  - Repo: https://github.com/GoogleCloudPlatform/google-cloud-python
</details>





### Update [google-cloud-core](https://pypi.org/project/google-cloud-core) from **0.28.1** to **0.29.1**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/google-cloud-core
  - Repo: https://github.com/GoogleCloudPlatform/google-cloud-python
</details>





### Update [protobuf](https://pypi.org/project/protobuf) from **3.6.1** to **3.7.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/protobuf
  - Changelog: https://pyup.io/changelogs/protobuf/
  - Repo: https://github.com/protocolbuffers/protobuf/releases
  - Homepage: https://developers.google.com/protocol-buffers/
</details>





### Update [boto3](https://pypi.org/project/boto3) from **1.9.101** to **1.9.106**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.9.106
   ```
   =======

* api-change:``ec2``: [``botocore``] Update ec2 client to latest version
* api-change:``autoscaling-plans``: [``botocore``] Update autoscaling-plans client to latest version
   ```
   
  
  
   ### 1.9.105
   ```
   =======

* api-change:``ssm``: [``botocore``] Update ssm client to latest version
* api-change:``apigatewayv2``: [``botocore``] Update apigatewayv2 client to latest version
* api-change:``alexaforbusiness``: [``botocore``] Update alexaforbusiness client to latest version
* api-change:``application-autoscaling``: [``botocore``] Update application-autoscaling client to latest version
   ```
   
  
  
   ### 1.9.104
   ```
   =======

* api-change:``waf-regional``: [``botocore``] Update waf-regional client to latest version
* api-change:``waf``: [``botocore``] Update waf client to latest version
   ```
   
  
  
   ### 1.9.103
   ```
   =======

* api-change:``discovery``: [``botocore``] Update discovery client to latest version
* api-change:``organizations``: [``botocore``] Update organizations client to latest version
* api-change:``resource-groups``: [``botocore``] Update resource-groups client to latest version
* api-change:``opsworkscm``: [``botocore``] Update opsworkscm client to latest version
* api-change:``pinpoint``: [``botocore``] Update pinpoint client to latest version
* api-change:``mediaconvert``: [``botocore``] Update mediaconvert client to latest version
* api-change:``cur``: [``botocore``] Update cur client to latest version
   ```
   
  
  
   ### 1.9.102
   ```
   =======

* api-change:``elbv2``: [``botocore``] Update elbv2 client to latest version
* api-change:``mediastore``: [``botocore``] Update mediastore client to latest version
* api-change:``ce``: [``botocore``] Update ce client to latest version
* api-change:``autoscaling``: [``botocore``] Update autoscaling client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/boto3
  - Changelog: https://pyup.io/changelogs/boto3/
  - Repo: https://github.com/boto/boto3
</details>





### Update [django-cors-headers](https://pypi.org/project/django-cors-headers) from **2.4.0** to **2.4.1**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.4.1
   ```
   ------------------

* Fix ``DeprecationWarning`` from importing ``collections.abc.Sequence`` on
  Python 3.7.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-cors-headers
  - Changelog: https://pyup.io/changelogs/django-cors-headers/
  - Repo: https://github.com/ottoyiu/django-cors-headers
</details>





### Update [django-mozilla-product-details](https://pypi.org/project/django-mozilla-product-details) from **0.13** to **0.13.1**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-mozilla-product-details
  - Repo: https://github.com/mozilla/django-product-details/
</details>





### Update [djangorestframework](https://pypi.org/project/djangorestframework) from **3.9.1** to **3.9.2**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/djangorestframework
  - Changelog: https://pyup.io/changelogs/djangorestframework/
  - Homepage: https://www.django-rest-framework.org/
</details>





### Update [jsonschema](https://pypi.org/project/jsonschema) from **3.0.0** to **3.0.1**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/jsonschema
  - Changelog: https://pyup.io/changelogs/jsonschema/
  - Repo: https://github.com/Julian/jsonschema
</details>





### Update [pytest-django](https://pypi.org/project/pytest-django) from **3.4.7** to **3.4.8**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.4.8
   ```
   ------------------

Bugfixes
^^^^^^^^

* Fix DB renaming fixture for Multi-DB environment with SQLite (679)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-django
  - Changelog: https://pyup.io/changelogs/pytest-django/
  - Docs: https://pytest-django.readthedocs.io/
</details>





### Update [google-cloud-storage](https://pypi.org/project/google-cloud-storage) from **1.13.0** to **1.14.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/google-cloud-storage
  - Repo: https://github.com/GoogleCloudPlatform/google-cloud-python
</details>







Co-authored-by: pyup-bot <github-bot@pyup.io>
@rgabbard

This comment has been minimized.

Copy link
Contributor Author

rgabbard commented Mar 7, 2019

I feel slightly better that I am not the only one to screw this up. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.