Expose dict_proxy internal type as types.MappingProxyType #58594
Comments
|
Attached patch makes the dictproxy type public. Example: $ ./python
Python 3.3.0a1+ (default:059489cec7b9+, Mar 22 2012, 02:45:36)
>>> d=dictproxy({1:2})
>>> d
dict_proxy({1: 2})
>>> d[1]
2
>>> d[1]=3
TypeError: 'dictproxy' object does not support item assignment
>>> del d[1]
TypeError: 'dictproxy' object doesn't support item deletion
>>> d.copy()
{1: 2}
>>> dir(d)
[..., '__getitem__', 'copy', 'get', 'items', 'keys', 'values']The patch doesn't have any test or documentation yet. See also the (now rejected) PEP-416 (frozendict). |
|
dictproxy.patch: proxy_new() should check that dict is a dict (PyDict_Check). |
|
Patch version 2:
I added a section "Immutable Mapping" in the doc, this title is maybe wrong because a dictproxy is not truly immutable. If the dict referenced by the dictproxy is modified, the dictproxy is also modified. dictproxy implementation supports any mapping, it is not specific to dict. It would be interesting any mapping, collections.ChainMap for example. The problem is to reject sequence in dictproxy constructor. The "PyMapping_Check(dict) && !PyMapping_Check(dict)" fails on ChainMap. type.__new__(ChainMap) fills tp_as_sequence->sq_item slot is defined because ChainMap has a __getitem__ method. Accepting sequences would be a bad idea because dictproxy(sequence)[missing_key] raises IndexError instead of KeyError. Note: issubclass(collections.ChainMap, collections.abc.Sequence) is False. |
|
The dictproxy is useful to implement a Python sandbox: see the related issue bpo-14385. |
Why? Just because you can't delegate in quite the same way? A sequence *does* meet the (immutable) Mapping interface; it just won't happen to have any non-integer keys. Or are you worried about IndexError vs KeyError? Personally, I would just document it as returning LookupError, but if you're really worried, you could always catch the IndexError and raise a KeyError from it.
I'm not seeing how anything could sanely pass that ... was there a typo when pasting? |
Ah? Let's try: >>> x=[1, 2, 3]
>>> x.get(2)
AttributeError: 'list' object has no attribute 'get'
>>> x.keys()
AttributeError: 'list' object has no attribute 'keys'list doesn't implement the collections.abc.Mapping ABC. |
Oh, I mean "PyMapping_Check(dict) && !PySequence_Check(dict)" |
|
I'm -1 on exposing this API. Victor's sandboxing project doesn't warrant cluttering the public toolset. As a consultant who gets to see many large codebases in different companies and I see zero need for this type to be exposed. It is not hard to build proxies around any existing type. Victor's use case is unique in that he needs the type to be implemented in C. To that end, it would be okay to expose _dictproxy but not to make it a documented type. |
|
I believe it was actually Guido who suggested exposing dict_proxy, in response to Victor (but independent of Victor's needs, if I understood the message correctly). It has always seemed odd to me ("always" referring my time working with Python3, of course) that a class that implements the mapping protocol has no way to return the same type of objects that real dict methods do. |
mxProxy, zope.proxy and zope.security implement object proxies in C (for security purpose). These proxies are not specific to dict, but are generic. dictproxy is already used in Python by the type.__dict__ descriptor getter of new-style classes since Python 2.2, since this changeset: changeset: 18933:09df3254b49d Guido van Rossum wrote in its rejection notice of the PEP-416: "On the other hand, exposing the existing read-only dict proxy as a built-in type sounds good to me. (It would need to be changed to allow calling the constructor.) GvR." --
My sandbox only needs the issue bpo-14385, pysandbox can implement its own dictproxy class. If the issue bpo-14385 is accepted, pysandbox can use any mapping, even a mapping implemented in Python! It's just simpler to implement a secure proxy in C. So this issue is not directly related to sandboxing. |
|
It's clutter and we should show some restraint before adding new types. Enthought's Python Distribution isn't shy about adding tools when they have a demonstrated need. Likewise, Django adds any reusable tools they need. Neither have a dict proxy. I've never run into a use case for it and haven't seen any feature requests for it. PyPi doesn't have package for it (at least not one with any uptake). The languages doesn't *need* this, nor do the other implementations which are trying to keep up with CPython. Guido may let you put in it, but that doesn't mean you should. Again, I ask you to show some restraint. The language has already exploded beyond what fits into most developer's heads (even core devs don't even know about many the newer features). |
dictproxy() doesn't need to be a public builtin type. We can just implement its constructor without documenting the type. Or it may be exposed in a module like collections. |
|
Given that dictproxy is needed to implement the proper interface for classes I don't think it is an implementation detail, and so I think it ought to be constructable. We have more esoteric types constructable (e.g. function). I also happen to think that its use to implement classes is a pretty darn good use case, and I am sure there are other use cases. I don't really care if we make it a builtin but there should be a standard name for it somewhere in the stdlib. |
|
[haypo]
+1
+1
Please don't use collections as a dumping ground for this sort of thing. |
|
New patch replacing dictproxy() builtin type with collections.mappingview() type:
I don't understand how collections types are called: should it be MappingView or mappingview? The code of mappingview is just moved to _collectionsmodule.c to avoid the creation of two short files. PyDictProxy_Type and PyDictProxy_New() are kept and still declared in descrobject.h for backward compatibility. I fixed the doc to indicate that mappingview methods operate on the *underlying mapping*. I also added test_views() test. TODO:
|
|
I would very much like this to go somewhere other than the collections module. As the maintainer of that module, I've worked hard to keep it clutter free (people frequently want to stick things in that module when they can't think of some place else to put it). Perhaps consider the existing types module, the builtin namespace, or a new module for proxies (if dict_proxy goes in, other proxy types won't be far behind). |
|
Raymond, that sounds like a very irrational way of deciding where it should go. |
|
I wonder about the name: There is already a collections.abc.MappingView. |
|
The weakref collections objects (sets, dicts, etc) were all put in a separate module and that has worked out well. I suggest doing the same for proxy objects. In maintaining the itertools module, I learned long ago that adding new tools to the module made the whole module more difficult to use (increasing the number of choices increases the complexity of choosing the correct tool). I believe that same also applies to the collections module and have been very reserved about adding new types there. ISTM that exposing internal types such as dict_proxy or bound/unbound methods could be collected together or that various proxy types and tools could be collected together. I don't feel that dict_proxy belongs with namedtuple, deques, or OrderedDicts. As the module maintainer, I request that dict_proxy be put elsewhere. Of course, as BDFL you are free to override that request. |
|
I like the suggestion of a library module containing proxy classes. |
|
There's already a weak proxy class in the weakref module. Not sure a separate module for the other proxies is a good idea. |
|
If we do create a proxy module I'd like to suggest adding all or part of Phillip Eby's ProxyTypes: http://pypi.python.org/pypi/ProxyTypes I haven't used it directly, but did read through it a while ago and have ended up re-implementing chunks of it myself several times. |
|
weakref.mappingview sounds nice to me. |
|
But it has nothing to do with weakrefs, so... |
|
Since it has to go *somewhere*, let's put it in the types module. It has a variety of other types that are used in the implementation of classes and functions and related things. |
|
[GvR]
+1 |
|
It is exposed as types.DictProxyType in Python 2... |
|
"It is exposed as types.DictProxyType in Python 2..." Yes, but the purpose of the issue is to enable its constructor. You cannot instanciate a DictProxy in Python 2: >>> import types
>>> types.DictProxyType
<type 'dictproxy'>
>>> types.DictProxyType({})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create 'dictproxy' instances |
|
Patch version 4, it is ready for a review. Summary of the patch:
I added the new tests to test_descr. Is test_types a better place for these types (the type is exposed in the types module, but implemented in descrobject.c)? I leaved the object implementation in descrobject.c to not have to add a new short file and to keep the Mercurial history. The file does still contain this funny comment: /* This has no reason to be in this file except that adding new files is a MappingViewType constructor accepts any mapping: dict, collections.ChainMap, ... Sequences are rejected: tuple, list, dict_view, etc. |
|
I'd like to bikeshed a little on the name. I think it should be Also make sure there's no way for someone who has access to the proxy |
You are right, it's closer to a proxy because it has the same methods than a mapping (except of some methods used to modify a mapping), whereas the API of keys/values/items views is very different from the mapping API.
MappingView is a strange ABC: it should probably declare __contains__ as an abstract method. I don't see why it doesn't inherit from Set (which would avoid the need of declaring __contains__, Set is a Container) whereas KeysView, ItemsView and ValuesView do inherit from Set. I suppose that MappingView is also present to factorize code but it should not be used directly. Anyway, you are not the first one who remarks that we already use "view" to define something else, so I wrote a new patch to use the "mappingproxy" name (exposed as types.MappingProxyType).
I don't think that we can write unit tests for such property. I suppose that the comment below is enough.
Done in my last patch: /* WARNING: mappingproxy methods must not give access to the underlying mapping */ |
|
Oh, collections.abc contains also the mappingview type exposed with the name "dict_proxy": It was exposed as _abcoll.dict_proxy in Python 3.2. It is not documented. Should we keep it for backward compatibility, or can it be removed? _abcoll was a private module and I didn't find dict_proxy in Python 3.2 doc. |
|
vstinner
changed the title
|
mappingproxy-5.patch is ready for a review. msg157036 contains a summary except that types.MappingViewType is now called types.MappingProxyType. |
vstinner
changed the title
|
[Victor] It was exposed as _abcoll.dict_proxy in Python 3.2. You can ignore those. As you saw, these are undocumented and not listed in __all__. If we ever expose these, it will likely be through the types module (or less likely through the collections.abc module which is public in 3.3). Probably, they won't get exposed at all because they are awkward to access directly and because they have implementation specific details (i.e. other implementations have their choice of how to implement various iterators and whatnot). |
You mean that I can remove it? It's a little bit surprising to find a concrete class in collections.abc. So I think that it's better to expose dict_proxy in types than in collections.abc. |
|
[Victor]
No, it needs to stay there. Summary: The collections.abc module is fine as-is. Go ahead and add dictproxy to the types module. |
|
New changeset c3a0197256ee by Victor Stinner in branch 'default': |
Ok, but there is still an issue: issubclass(types.MappingProxyType, collections.abc.Mapping) is False. Attached registers MappingProxyType as a Mapping. Is it correct? The patch also renames dict_proxy to mappingproxy. Is it backward incompatible? (collections.abc module was added to Python 3.3) |
|
New changeset 34af3e74292d by Victor Stinner in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: