add exclude_private parameter

[H4dr1en]

Change Summary

Add exclude_private parameter to BaseModel.dict and BaseModel.json, allow BaseModel.__setattr__ to register properties if extra == Extra.ignore.

This PR would solve the problem described in the issue linked below: User can add private properties at runtime and exclude them from export.

Related issue number

#655

Checklist

• Unit tests for the changes exist
• Tests pass on CI and coverage remains at 100%
• Documentation reflects the changes where applicable
• changes/<pull request or issue id>-<github username>.md file added describing change
  (see changes/README.md for details)

[samuelcolvin]

This is failing, but even if it was passing I'm not sure I agree with it.

It adds even more logic and arguments to .dict(), I think you can either use exclude or implement your own logic (perhaps on _iter, see #997).

[samuelcolvin]

Upon reflection, I'm prepared to accept this.

Please:

• make the changes as requested
• add tests
• add a change description
• add a section to the docs (I guess in models.md) documenting this feature

[samuelcolvin]

Suggested change

	if self.__config__.extra is Extra.forbid and name not in self.__fields__:
	if name.startswith('_'):
	self.__dict__[name] = value
	return
	if self.__config__.extra is not Extra.allow and name not in self.__fields__:

[samuelcolvin]

remove this, I think it's safe to always exclude anything in __dict__ where the key startswith with _.

[Bobronium]

But this would break concept when we have only parsed values in __dict__ and will force additional checks, etc.

[Bobronium]

from pydantic import BaseModel, Extra

class Record(BaseModel):
    class Config:
        extra = Extra.allow

print(Record(_a='b'))  # _a='b'

[Bobronium]

Personally, I think that the best solution for this problem, when you want to set your own private attrs which must not be included in fields is to use __slots__, as proposed in #655 (comment)

Maybe it should be documented as a standard way of doing this to avoid confusion in future. I don't see any disadvantages of __slots__ approach, it solves the problem, it keeps __dict__ only for fields and makes things simpler with no excuses.

[samuelcolvin]

I agree with @MrMrRobat and also see the performance concerns, let's discuss other solutions on #655.

[maxnoe]

@MrMrRobat

I don't see any disadvantages of slots approach

Maybe a disadvantage is, that it does not really work?

In [1]: from pydantic import BaseModel                                                                                                        

In [2]: class Component(BaseModel): 
   ...:     __slots__ = ('log', ) 
   ...:     foo: int 
   ...:     bar: float 
   ...:                                                                                                                                       

In [3]: c = Component(foo=1, bar=3.5, log='test')                                                                                             

In [4]: c.log                                                                                                                                 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-706395a84d01> in <module>
----> 1 c.log

AttributeError: log

In [5]: c.log = 'test'                                                                                                                        
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-8fce80748249> in <module>
----> 1 c.log = 'test'

~/.local/lib/python3.8/site-packages/pydantic/main.cpython-38-x86_64-linux-gnu.so in pydantic.main.BaseModel.__setattr__()

ValueError: "Component" object has no field "log"

[samuelcolvin]

Maybe a disadvantage is, that it does not really work?

No, not good enough. Please reword your comments to be more polite.

You might also think about crafting a more articulate description of your point here - I'm not really sure what you point is.

[maxnoe]

Rereading the comment mentioning slots, it also adds object.__setattr__ to actually set the field, sorry I missed that in the first read.

But that does not really address private variables to be used like normal members, which was the point of this PR right?

At least for us, the slots approach does not really solve our problem.
We'd reallly like to use pydantic for configuration but need to have additional properties or members not handled by pydantic. For this, there is currently not really a way to get this working, right?

This PR would have been a great step into this direction.

[trallnag]

@maxnoe, maybe it's an option to convert the Pydantic model to a Box tree (see this project https://github.com/cdgriffith/Box) after the initial setup?

But in general I agree with you, my use case is a slight modification of data send to an API endpoint. What I currently do is create a second version of my whole model that inherits from the base model and only includes the additional fields. Then I use construct() to create the updated model. The biggest disadvantage with this is that construct() does not handle nesting that well