Move all public methods on a model into a single namespace

[samuelcolvin]

There's currently a tension in pydantic: understandably there's a desire to add more public methods to models, for example iter in #997. But there's also a concern with adding more public methods since they might conflict with fields people want to add, e.g. in the above example it would cause annoyance if someone wanted an iter field (there's an issue about this somewhere, but I can't find it right now, pointer welcome).

Of course this can be partially solve with aliases, but it's still frustrating and confusing.

Potential solutions:

• make all the methods on models "private" (this is what's done I think on named tuples) so my_model.dict() would become my_model._dict(). This would upset pycharm and perhaps linting tools a lot and would look ugly.
• move all public methods into a single namespace, e.g. my_model.dict() would become my_model.<whatever>.dict(). Slightly more typing but I think clearest.

Obviously if we do this is would need to be in version 2.

I think the second solution would be the best approach. Questions:

• do you agree?
• would it have a performance impact, I guess; yes but small?
• most importantly: if this is agreed with, what name should we use?

I think something short, preferably one character.

How about m, standing for either "methods" or "model"?

Example of this usage:

from datetime import datetime
from typing import List
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

user = User(id=123)
print(user.m.dict())
print(user.m.schema())

@dmontagu, @MrMrRobat, anyone else; thoughts?

[dmontagu]

In general, I like the idea of more flexibility around public methods.

---

I'm against making all methods private due to linter weirdness, and confusion about the safety of overriding the methods.

I could be convinced to use sunder naming for all of them though (i.e., _method_), since I think PyCharm and most linters treat those as public. But it's a little rare to see and might be confusing for users, and it's a little annoying to type extra underscores. Also, I think in PyCharm you have to type ._ before they will show up in the autocompletion list, rather than just ..

But it would probably(?) make migration as simple as possible, and would result in zero overhead. And maybe the autocompletion has some benefits -- namely making it easier to separate field names from methods.

---

Blocking the use of a field named m without an alias seems somewhat unfortunate. At the same time, it's probably not generally a good field name in practice, so I don't mind forcing the use of an alias. And I guess it would allow any of the current public method names to become field names, e.g., json, dict, etc., which might be more likely than m to cause conflicts anyway.

Using _m_ would prevent PyCharm/linting issues, and would result in zero public attribute name conflicts, but is admittedly uglier and more annoying to type.

---

I'd be curious to see benchmarks of the performance implications, at least when cythonized. Presumably they would be small, and could mostly be mitigated in internal calls by assigning a local variable to model.m outside of loops. But it would be good to confirm this.

[Bobronium]

Personally I don't like the idea of using _private and _enum_like_ approaches — as already mentioned, first one will upset PyCharm, the second seems uglier and both forces user to type leading underscore.

I see solution in adding underscore in the end of name, like it's done when you need to name variable with some built-in name: e. g.type_. Also it's commonly used in many libraries, for example SQLAlchemy with its or_, and_, etc.

Upd. actually this naming mentioned in PEP-8 (search for single_trailing_underscore_)

Also, i see another solution. What if we had an aliases to all public methods with underscore and original names available unless they were shadowed by some field?

class SomeModel(BaseModel):
    id: int
    name: str
    dict: dict  # type: ignore


m = SomeModel(id=1, name='John', dict={'Hello': 'World'})
print(m.dict)  # {'Hello': 'World'}
print(m.dict_())  # {'id': 1, 'name': 'John', 'dict': {'Hello': 'World'}}

At the same time it will allow to have the same performance in runtime and will not force to type some extra symbols if you don't have any fields which shadow public methods, so 100% backward compatibility and no any breaking changes here.

It may not seem so obvious, but actually this is how things in pure python classes would work:

class Base:

    def dict(self):
        return self.__dict__

    dict_ = dict

class Shadows(Base):
    dict: dict

    def __init__(self):
        self.dict = {1: 2, 3: 4}


s = Shadows()
print(s.dict)  # {1: 2, 3: 4}
print(s.dict_())  # {'dict': {1: 2, 3: 4}}

class Model(Base):
    def __init__(self):
        self.field = 'Hello!'

m = Model()
print(m.dict())  # {'field': 'Hello!'}

[dmontagu]

@MrMrRobat I think there's something potentially something interesting with the masking approach, but I see a few issues:

1. (Minor) You'll see duplicates of each method during autocomplete
2. (Less minor) I think it would probably be a pain to get mypy/PyCharm to play nice with the overrides
3. (Major) It would become dangerous to use public methods anywhere where you weren't 100% sure the name wouldn't be masked. So, for example, FastAPI couldn't make use of public methods on models that were not defined within the FastAPI codebase.

• While we might be able to assume that well-tested frameworks like FastAPI were safe, this seems like it would be a bigger issue in user-code where people might not realize/remember to handle this possible error case.

I suppose we could transition toward always having methods end with a trailing underscore (though I find it a little ugly), and never allowing those methods to be masked, but then 1) we are still blocking the use of certain names and so wouldn't be able to make changes without breaking backwards compatibility, and 2) I think this might ultimately result in a confusing migration path.

---

Another possible mitigation: adopt a second convention that field names are not allowed to follow, and reserve it for methods. This would allow adding arbitrary public methods over time, would only introduce new disallowed-field-names logic once, and would ensure there was no danger of calling possibly-masked methods.

Unfortunately, I think trailing underscore is not ideal here because that is useful in many cases due to PEP8 as you mentioned above, and leading underscore is undesirable for the reasons above. So I'm not sure there is a great option here either.

[samuelcolvin]

My proposed is that once we agree a new naming convention/namespace we implement all the new methods and mark the old methods deprecated in v2.

I'm not apposed to .dict_(), .json_(), .construct_(), .parse_obj_() etc. it's still a little ugly but most concise. It would also make auto-completion work better than .m.dict() and be more conventional.

We would just ban use of trailing underscores in all field names so no future conflict was possible.

[thebigmunch]

Alternatively, these methods could be top-level functions. attrs does this with its asdict, astuple, etc functions, for example.

[samuelcolvin]

We're going to use that approach for some methods e.g. that relate to dataclasses where we can't really add methods to the class. See #934 and #481.

But for the common model methods there are numerous reason to keep them on the class.

[dmontagu]

Yeah, I much prefer methods, regardless of the "masking" scheme. It's a lot easier to get autocompletion and also results in a lot fewer import statements to write/generate.

I'm okay with the trailing _ naming scheme for methods, but I think it would be good to have a suggested naming scheme for fields that currently have a trailing underscore (and don't want to just drop it, e.g., to prevent shadowing builtins). E.g., type_ -> type_field (with aliases as desired). Obviously it wouldn't be enforced or anything, but I think proposing a convention would help lower the barrier to migration.

[layday]

I feel like the simplest way (and more discoverable way, as opposed to top-level functions) to avoid collisions is to move methods up to the metaclass. Model.dict(model) isn't as pretty as model.dict() but you'll never have to worry about overwriting instance methods again.

[mbello]

I arrived here after converting a model to BaseModel failed due to 'fields' being an invalid name for a BaseModel field (already taken by the library).
Then I used name "fields_" and added an alias to 'fields' only to find out that the dict() was being produced with a key 'fields_' rather than 'fields' so, at least for producing the dict, the alias is not a good solution.

I think adding a trailing underscore to field and method names is the best solution.

[Bobronium]

@mbello, you can use m.dict(by_alias=True) to construct dict with aliases. However, I agree that it’s not as convenient as a plain m.dict() call