API semantics for deep nested evolve calls

[jacobg]

When trying to evolve a frozen object at a deep level, there is a lot of boilerplate, e.g.,:

foo = attr.evolve(
   foo,
   bar=attr.evolve(
      foo.bar,
      baz=attr.evolve(
         foo.bar.baz,
         qux='whew!'
       )
   )
)

Scala has a similar problem, being that immutable data classes is a normal thing in that language. There is a Scala library developed helped to solve that problem: https://www.optics.dev/Monocle/

Maybe it would be useful to have those sort of semantics for updating frozen attr objects,e.g.:

foo = attr.focus(foo, Foo.bar.baz.qux).modify('yay!')

Any thoughts?

[Tinche]

Definitely an interesting idea, but Foo.bar.baz.qux cannot work (since Foo.bar is unset for dict classes, and occupied for slot classes). And nested fields() get ugly.

[jacobg]

Thanks @Tinche . So maybe a string path?

foo = attr.focus(foo,'bar.baz.qux').modify('yay!')

[hynek]

This might be an alternative approach to #634 and #861.

It shouldn't need any special support within attrs?

Modify seems to be a callable thing (replace is for setting), but that should be easy enough with Python.

cc @sscherfke

[jacobg]

I just made a separate function based on #634 (comment), and it seems to work fine:

def evolve_recursive(inst, **changes):
    """ Recursive attr.evolve() method, where any attr-based attributes
        will be evolved too.
    """
    cls = inst.__class__
    attrs = attr.fields(cls)
    for a in attrs:
        if not a.init:
            continue
        attr_name = a.name  # To deal with private attributes.
        init_name = attr_name if attr_name[0] != "_" else attr_name[1:]
        value = getattr(inst, attr_name)
        if init_name not in changes:
            # Add original value to changes
            changes[init_name] = value
        elif attr.has(value):
            # Evolve nested attrs classes
            changes[init_name] = attr.evolve(value, **changes[init_name])
    return cls(**changes)

def test_set_deep_sparse():

    @attr.s(auto_attribs=True, frozen=True, slots=True)
    class Bar:
        a: str
        b: str

    @attr.s(auto_attribs=True, frozen=True, slots=True)
    class Baz:
        c: str
        d: str

    @attr.s(auto_attribs=True, frozen=True, slots=True)
    class Foo:
        bar: Bar
        baz: Baz

    foo = Foo(bar=Bar(a='a1', b='b1'), baz=Baz(c='c1', d='d1'))

    assert (evolve_recursive(foo, bar={'a': 'a2'}, baz={'c': 'c2'}) ==
            Foo(bar=Bar(a='a2', b='b1'), baz=Baz(c='c2', d='d1')))

[sscherfke]

Typed Settings contains a recursive version of evolve(): https://gitlab.com/sscherfke/typed-settings/-/blob/main/src/typed_settings/attrs/__init__.py#L328-368 For some cases (with old, untyped coded) it has slightly different behavior than attrs.evolve() (which is why we removed it from attrs).

Typed Settings also defines operations for working with dicts and dotted paths which is a bit similar to @jacobg 's original idea.

Updating nested instances with a dotted path may be easier if you just want to update single attributes. The recursive evolve is a bit more convenient when you want to update multiple attributes (on different nesting levels). So maybe there's room for both variants.

[td-anne]

If it's syntactically nicer, one can use the dict constructor in evolve_recursive:

evolve_recursive(foo, bar=dict(a="a2"), baz=dict(c="c2"))

In fact one could define a class Evolve that just wrapped its arguments, and then it could be used backward-compatibly in evolve:

evolve(foo, bar=Evolve(a="a2"), baz=Evolve(c="c2"))

That is, evolve could check specifically whether the value being passed was an instance of Evolve, and if so, know that recursive evolution was being requested. In all other cases evolve would act as now. Since attrs.Evolve would be a new class, this should not break any existing code.

Unfortunately, this won't allow static type checking of the arguments to Evolve. But then I don't really think it's feasible to statically type evolve arguments anyway?

This would look something like:

class Evolve:
    def __init__(self, **changes):
        self.changes = changes

def evolve(inst, **changes):
    cls = inst.__class__
    attrs = fields(cls)
    for a in attrs:
        if not a.init:
            continue
        attr_name = a.name  # To deal with private attributes.
        init_name = a.alias
        if init_name not in changes:
            changes[init_name] = getattr(inst, attr_name)
        elif isinstance(changes[init_name], Evolve):
            changes[init_name] = evolve(getattr(inst, attr_name), changes[init_name].changes)

    return cls(**changes)

If one wanted to be more ambitious, one could make Evolve private (_Evolve) and if attrs.evolve was called without a positional argument, it would return an _Evolve instance; then this sort of implementation would allow one to write:

evolve(foo, bar=evolve(a="a2"), baz=evolve(c="c2"))

Without additional work this would be a problem for error handling, because if one forgot the positional argument the effect would be very peculiar. It might make more sense to give Evolve an modify method, as Scala does:

new_thing = Evolve(bar=Evolve(a="a2"), baz=Evolve(c="c2")).modify(old_thing)

This would coexist fairly well with static type checking, in that an Evolve object would be an obviously different type from the instance we were modifying. In principle it might be possible to make Evolve generic in a way that type inference could work for it.

[hynek]

I don't think given the increasingly narrow guardrails that are forced on us by Pyright/dataclass transform, thinking about type-safety is a waste of time. :(

The builder approach sure looks interesting, so if someone wants to tackle it, why not add a third way to copy-and-modify instances. ;)

[loehnertj]

If one wanted to be more ambitious, one could make Evolve private (_Evolve) and if attrs.evolve was called without a positional argument, it would return an _Evolve instance; then this sort of implementation would allow one to write:

Incidentally, I implemented exactly this approach some months ago. Here is the code:

https://gist.github.com/loehnertj/4cf864d98054d7a54749e99bd4ae28b8

Documentation and some testing included. Use it as you like. I'll gladly help with integration if desired (however I don't know about attrs' internal structure + standards).

One particular challenge was how to handle list-of-object fields, which occur rather often. This is solved by means of a second "evols" (Evolve-list) function giving you a "builder"-ish interface to modify lists. I.e. you chain method calls to evols in order to change one element, all elements, etc.

Quoting the docstrings:

def evo(_inst: T = None, **kwargs) -> Union[T, Evolution]:
def evols(rtype: type = list) -> ListEvolution:

evo: Create a new instance, based on the first positional argument with changes applied.

This is very similar to attr.evolve, with two major extensions.

The first argument may be omitted. In this case, returns a callable that
applies the changes to its single argument, a so called .Evolution.

Secondly, if any keyword argument-value is an Evolution instance, it will be
applied to the object's property as identified by the keyword.

This means, that you can save a lot of boilerplate repetition when evolving
nested objects. The resulting syntax is very similar to nested construction.

Example: Suppose that object a has subobject b with field x that
you want to copy-and-modify.

Classical attr.evolve would read like::

a_mod = evolve(a, b=evolve(a.b, x=1))

With evo, this simplifies to::

a_mod = evo(a, b=evo(x=1))

evols: Frequently, properties are lists of objects. Such properties can be evolved
using .evols. Allows both content and structural modification.

Example: suppose that the b property is now list-valued::

a_mod = evo(a, b=evols().at(-1, x=1))
a_mod2 = evo(a_mod, b=evols().pop(0))

To actually specify the operations to do, chain-call the appropriate methods
of .ListEvolution. For example, you can mimic sorted(l) by doing::

evolution = evols().sort()
sorted_l = evolution(l)

Other possible operations are adding, removing, reordering items as well as
evolving or replacing individual items. See .ListEvolution.

Optionally, return type of the modification can be specified by means of
rtype parameter.

List evolution methods include at (modify item), set (replace item), all
(modify all items), pop, remove, insert, append, select (permutation),
sort.