Allow `new` to accept keyword arguments for more control over incomplete initialization

[FedericoStra]

The Problem

Currently, when defining inner constructors, we can decide to incompletely initialize a structure simply by passing to new(...) fewer values than the number of fields:

struct Incomplete
    x1
    x2
    x3
    ...
    x719
    Incomplete(x1, x2, x3, x4) = new(x1, x2, x3, x4) # x5, ..., x719 are uninitialized
end

Since the call to new(...) accepts only positional arguments, this has the obvious limitation that we can only initialize a prefix of all the fields and leave the remaining ones uninitialized. It is impossible to initialize only x1, x42, x470 and x666.

The Proposal

My proposal is to allow new(...) to accept keyword arguments, so that the following code becomes legal:

struct Incomplete
    x1
    x2
    x3
    ...
    x719
    Incomplete(a, b, c, d) = new(x1=a, x42=b, x470=c, x666=d) # the other fields are uninitialized
end

Remarks

Motivation

It is just useful sometimes to be able to do this. See for instance this thread where I asked on the forum if there was a way to achieve this with the current state of the language. There are plenty of workarounds, with more or less significant downsides (complexity-wise and performance-wise), but no direct solution.

Backward compatibility

This should be 💯.

Downsides

I honestly don't see any.

Labels

design, feature, keyword arguments, types and dispatch

[tpapp]

Orthogonally to incomplete initialization, just having a form of new with keyword arguments would be kind of nice for composite types with a nontrivial amount of fields.

[StefanKarpinski]

Downsides

I honestly don't see any.

• Complicates the parser and lowering passes of the front-end;
• Compiler needs to be able to handle arbitrary subsets of fields being uninitialized.

[FedericoStra]

• Complicates the parser and lowering passes of the front-end

How does it complicate the parser? The parser is already capable of parsing keyword arguments elsewhere. Is the new function special-cased in this regard? I'll try to dig into the code as soon as I find the time. I'm not familiar at all with the internals, but it seems strange to me that this proposal would require a big change to the parser.

• Compiler needs to be able to handle arbitrary subsets of fields being uninitialized.

Does this mean that currently the compiler relies on the fact that only a suffix of the fields can be uninitialized? I don't see why the compiler should care about this. The implementation should be pretty straightforward:

• keep what is already in memory for the fields which are "plain data",
• possibly initialize to 0, NULL or whatever the fields which are references, or skip initialization altogether if possible.

Then the logic for accessing fields doesn't need to change at all. We don't have to keep track of which fields are initialized or not.

How does the compiler currently "handle" which fields are initialized? I think it just emits the asm to read them whenever they are accessed, right? There shouldn't be anything new to do w.r.t. the current implementation.

[StefanKarpinski]

It's probably not a huge complication, but it will certainly complicate some of the front-end-code. The new constructor isn't really a function, so it's handled a bit specially.

Regarding the compiler, it needs to defensively emit code that checks any ref fields that can be uninitialized to make sure they're not NULL pointers before accessing them. When accessing fields that are always initialized, this doesn't need to be done. Currently, which fields need to be checked and which don't can be tracked with a single number; if it becomes an arbitrary subset, a bitfield will be required instead.

I'm not arguing against this feature, just pointing out that it isn't without implementation and maintenance costs.

[JeffBezanson]

I do think this feature makes sense. It's easy to implement for mutable structs, since we can just lower it to a series of setfield!s. For immutable we would probably need the "mutating immutables" feature (#11902) before we could do this.

[FedericoStra]

@StefanKarpinski

Regarding the compiler, it needs to defensively emit code that checks any ref fields that can be uninitialized to make sure they're not NULL pointers before accessing them. When accessing fields that are always initialized, this doesn't need to be done. Currently, which fields need to be checked and which don't can be tracked with a single number; if it becomes an arbitrary subset, a bitfield will be required instead.

Now I completely get what you mean. Sorry :) Of course I agree with what you say.

I believe the counter you mention is ninitialized in julia.h

typedef struct _jl_datatype_t {
    /* ... */
    int32_t ninitialized;
    /* ... */
} jl_datatype_t;

right?

@JeffBezanson

For immutable we would probably need the "mutating immutables" feature (#11902) before we could do this.

I also thought about having temporary mutability for immutable structs inside inner constructors, but that seemed somewhat a bigger change to the language, therefore I limited myself to this proposal, which could potentially be implemented in an easier way than the much bigger rework of immutability.

It's easy to implement for mutable structs, since we can just lower it to a series of setfield!s.

That's also the case that I consider less interesting, because I could just do it myself by using an emptynew() in the inner constructor and then explicitly setting the fields that I want. The only downside is that those fields would probably not be marked as "always initialized", so accessing them requires a NULL check. Of course having the support for this feature would still be handy, but not as indispensable as it is for immutable structs, for which there is no possible workaround in userland.

[StefanKarpinski]

FWIW, I've always questioned the need for partially initialized immutable structs. I guess that it can be useful in some situations, but since you can't finish initializing them, it seems like a bit of a dicey feature to me.

[FedericoStra]

Well, maybe you don't need to "finish initializing" them; they are already finished as they are. They just carry redundant fields for homogeneity with other instances of the same type.

I'll reproduce here the example I gave on the forum. Imagine that you want to represent bounded/unbounded open/closed intervals with endpoints of type T. In C you would do

typedef struct _interval_t {
    T left;
    T right;
    uint8_t kind;
} interval_t;

where kind is a bitmask that indicates the kind of endpoints: there are 3 possibilities (closed, open, unbounded) for each endpoint, so you can use 2 bits for each.

When you need to represent the interval [42, ∞), you just don't assign to the member right (and of course never read it afterwards!). The produced struct has an uninitialized member, but it is in no way "unfinished". It's like this just to blend in with her friends (-∞, 13] and [0, 1].

[JeffBezanson]

In that case, if T is a reference type you're better off setting the unused field to some value anyway, so we don't have to check fields on access.

[FedericoStra]

True, but it isn't straightforward to produce "some value" of type T in a generic fashion. One would need to force library users to define a

default(::Type{T}) = ...?...

for their types if they want to use them in the structure I provide them.

Also, either you always return exactly the same instance, or you'll have some allocations.

default(::Type{BigInt}) = BigInt(0)

would be bad. One has to do

const BIGINT_DEFAULT = BigInt(0)
default(::Type{BigInt}) = BIGINT_DEFAULT

---

This could be seen as a possible tradeoff once the two approaches are available:

• either create incompletely initialized structs, which require no cooperation on the part of the library user, but require NULL checks;
• or arbitrarily initialize non plain data fields, possibly requiring some cooperation on the part of the library user, but leading to more efficient generated code because it avoids NULL checks.