PY3: use MutableMapping for python 3 #810
Conversation
|
so, in py3.4 it pass the "metaclass" associated tests that fails on py2.7 when using MutableMapping, right? side note about |
|
Yes, in 3.4 it works fine, and I also see other projects using this import routine to gain compatibility across Python 2/3. |
|
What about UserDict methods that are not implemented in MutableMapping? See http://stackoverflow.com/questions/11165188/how-to-achieve-the-functionality-of-userdict-dictmixin-in-python-3 By the way, it was me who broke Travis with scrapy/w3lib#23, disregard the failure. |
|
Hrm, I see. I didn't find any clue that the two affected methods were used anywhere in scrapy itself, so this looks like breaking backward compatibility for someone else when he migrates to Python 3. I would suggest to keep it this way for now, and if people are complaining about this, we can append the two methods directly to DictItem. |
|
I'm fine with that. |
|
I'm not convinced about using different base classes the following is debug information about running tests with MutableMapping in Python2 |
|
I think this PR doesn't add any value if we are only merging an optional import for python3 that actually doesn't pass Assigning metaclasses in python3 is different than python2 so I think it is not failing for MutableMapping because |
|
Hrm, I think I got the point. I'll look into it further. |
# metatest.py
import six
from collections import MutableMapping
class M(type):
pass
class A(MutableMapping):
pass
class B(object):
pass
class C(six.with_metaclass(M, A, B)):
pass
C()in Python3: $ python --version
Python 3.4.1
$ python metatest.py
Traceback (most recent call last):
File "metatest.py", line 13, in <module>
class C(six.with_metaclass(M, A, B)):
File "/usr/lib/python3.4/site-packages/six.py", line 706, in __new__
return meta(name, bases, d)
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its basesin python2: $ python --version
Python 2.7.8
$ python metatest.py
Traceback (most recent call last):
File "metatest.py", line 13, in <module>
class C(six.with_metaclass(M, A, B)):
File "/home/daniel/envs/sh/lib/python2.7/site-packages/six.py", line 631, in with_metaclass
return meta("NewBase", bases, {})
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases |
|
Hi, I've re-submitted a commit, to actually implement the missing methods from MutableMapping, and make all test cases in test_item passed. However, since I'm not familiar with this part, I could do this wrong. For example, I noticed that |
| for i in getattr(self, "_values", ()): | ||
| yield i | ||
|
|
||
| def __hash__(self): |
kmike
Jul 24, 2014
Member
Providing __hash__ for mutable objects can lead to strange issues. Why is it needed?
Providing __hash__ for mutable objects can lead to strange issues. Why is it needed?
|
I see how what is calling |
|
@kmike, it is needed because of scrapy.utils.trackref#L30
|
|
Is it for weakrefs? WeakKeyDictionary should use references as keys, not objects directly, so there shouldn't be a requirement for object to be hashable. |
|
weakref docs says that "Several built-in types such as list and dict do not directly support weak references but can add support through subclassing". Did it stop working because DictItem stopped being a subclass of dict? |
|
happens that MutableMapping does define a we can keep old behavior with: def __hash__(self):
return BaseItem.__hash__(self) |
|
ah, right, it was not even a subclass of UserDict (and UserDict is also not a subclass of dict :) Another way to keep old behaviour: |
|
The following patch pass tests: $ git diff
diff --git a/scrapy/item.py b/scrapy/item.py
index ec41b21..1f1e8ac 100644
--- a/scrapy/item.py
+++ b/scrapy/item.py
@@ -76,14 +76,10 @@ class DictItem(MutableMapping, BaseItem):
return len(self._values)
def __iter__(self):
- for i in getattr(self, "_values", ()):
- yield i
+ return iter(self._values)
def __hash__(self):
- if hasattr(self, "_values"):
- return hash(frozenset(self._values.items()))
- else:
- return 1
+ return BaseItem.__hash__(self)
def keys(self):
return self._values.keys()
diff --git a/tox.ini b/tox.ini
index 2bd9732..002a778 100644
--- a/tox.ini
+++ b/tox.ini
@@ -54,6 +54,10 @@ deps =
:HPK:pytest>2.5.2
pytest-twisted
+[testenv:py34]
+basepython = python3.4
+deps = {[testenv:py33]deps}
+
[testenv:windows]
commands =
bin/runtests.bat []$ tox -e py34 scrapy/tests/test_item.py
GLOB sdist-make: /home/daniel/src/scrapy/setup.py
py34 create: /home/daniel/src/scrapy/.tox/py34
py34 installdeps: twisted >= 14.0.0, lxml>=3.2.4, pyOpenSSL>=0.13.1, cssselect>=0.9, queuelib>=1.1.1, w3lib>=1.5, mock, :HPK:pytest>2.5.2, pytest-twisted
py34 inst: /home/daniel/src/scrapy/.tox/dist/Scrapy-0.25.1.zip
py34 runtests: PYTHONHASHSEED='1975113793'
py34 runtests: commands[0] | py.test --twisted scrapy/tests/test_item.py
======================================= test session starts =======================================
platform linux -- Python 3.4.1 -- py-1.4.22 -- pytest-2.6.0
plugins: twisted
collected 12 items
scrapy/tests/test_item.py ............
==================================== 12 passed in 0.11 seconds ====================================
_____________________________________________ summary _____________________________________________
py34: commands succeeded
congratulations :) |
|
@kmike alternative to |
|
@felixonmars : what do you think if we use |
|
@dangra i thought it was fixed in six 1.7 (https://bitbucket.org/gutworth/six/issue/66/replace-the-implementation-of). Not sure why aren't docs on pythonhosted.org up to date (people, use RTFD instead). |
|
.. but we require six >= 1.5.2, so it is better to either update six version or to use add_metaclass as Daniel suggested. |
|
I merged it with the changes we discussed here. |
|
That was great! I just woke up and see these :D |
While Python 2.7 also has
collections.MutableMapping, it's not compatible with current code (raises some metaclass errors).This PR works in both Python 2.7 & 3.4 (3.3 not tested, but should work too).