Fix Issue 9636: null initialization for std.typecons.Nullable

[MetaLang]

https://issues.dlang.org/show_bug.cgi?id=9636

For now, it's only allowed to use DefaultNullable on initialization. Nullable already has a Nullify function, so it's not as useful to also enable assigning DefaultNullable to a Nullable to nullify it.

If possible, this should be merged before #2587, so I can then amend that PR to print "DefaultNullable" when Nullable is internally "null", so as not to cause confusion if the wrapped type also has a null state.

[monarchdodra]

The spirit of the pull looks good to me. But now, for some bike shedding:

enum DefaultNullable = DefaultNullableImpl.init;

That's not a type, so lower case please: defaultNullable

private alias DefaultNullableImpl = Typedef!(byte, 0, "std.typecons.DefaultNullable");

Why the whole Typedef? Just: private struct DefaultNullableImpl{}. No need to bloat.
Also, I don't think it should be private, so users can provide their own function that can special case the null state. Also, no need for Impl.

Finally, I don't like the word "DefaultNullable". There's nothing "default" about it. I'd go for NullNullable, or (if possible) simply Nullable.NullState and Nullable.nullState. Something along those lines.

[JakobOvrum]

Nullable.init is already in the null state, and it's also default-constructable. So why all this?

[monarchdodra]

Nullable.init is already in the null state, and it's also default-constructable. So why all this?

Thats... actually a good point.

[Geod24]

If possible, this should be merged before #2587, so I can then amend that PR to print "DefaultNullable" when Nullable is internally "null", so as not to cause confusion if the wrapped type also has a null state.

(Citing @monarchdodra from bugzilla)

Arguably, you won't see that very often, but it is plausible for someone to want to be able to have a nullable pointer, whose "non-null" value can itself be null.

If the wrapped type has a null state, it's pointless to use Nullable with it. I agree that it's possible, however the point of Nullable is to allow one to have a value in an "invalid"/"uninitialized" state which is exactly what null is for pointers.

Nullable.init is already in the null state, and it's also default-constructable. So why all this?

Looking at the issue (9636), nothing more than a syntactic sugar for default parameter.

I think we should rather disallow pointers for Nullable (unless someone can cite a valid use case) and resurrect something like #1356 .

[bearophile]

@Geod24: >If the wrapped type has a null state, it's pointless to use Nullable with it.

This is not true. Sometimes you want to avoid raw null pointers with a Nullable!(int*, null). It uses the same memory as a raw pointer, but it's guarded by asserts.

@JakobOvrum: >Nullable.init is already in the null state, and it's also default-constructable. So why all this?

Do you mean I should write code like this?

void foo(Nullable!(immutable int[4]) items = Nullable!(immutable int[4]).init) {}

I find it not acceptable and I'd like some shorter way to write it.

[JakobOvrum]

We could make Nullable(T) alias to T when T is already nullable, add a constructor and assignment overload for typeof(null) (fixing the default argument problem), and make nullify a UFCS function. That should be fairly backwards-compatible and fix both problems.

[bearophile]

@JakobOvrum: >We could make Nullable(T) alias to T when T is already nullable,

Please let's keep the semantics of Nullable as much as possible clean. A clean semantics avoids tons of problems later.

[JakobOvrum]

Please let's keep the semantics of Nullable as much as possible clean. A clean semantics avoids tons of problems later.

Semantics are cleaner without the nasty double null situation.

[monarchdodra]

I find it not acceptable and I'd like some shorter way to write it.

Just use an alias. Are you seriously using Nullable!(immutable int[4]) all over the place, without an alias? You are basically saying that you find writing Type t = Type.init unacceptable... It's a pain yes (maybe), but hardly "unacceptable".

This is not true. Sometimes you want to avoid raw null pointers with a Nullable!(int*, null). It uses the same memory as a raw pointer, but it's guarded by asserts.

Furthermore, Nullable can be also be to differentiate between "default state" and "not yet specified state". If my function taking a Nullable!(int*) encounters a non-null nullable that holds a null pointer, then I know that someone, somewhere, took the time to explicitly place that null in my nullable, and it was no accident.

We could make Nullable(T) alias to T when T is already nullable

Do we even have a isNullable trait?

[monarchdodra]

Semantics are cleaner without the nasty double null situation.

If somebody writes Nullable!(int*), then surely, they know what they are doing. If they are operating on T, then surely, it's better to give Nullable consistent behavior regardless of T.

[JakobOvrum]

Furthermore, Nullable can be also be to differentiate between "default state" and "not yet specified state". If my function taking a Nullable!(int*) encounters a non-null nullable that holds a null pointer, then I know that someone, somewhere, took the time to explicitly place that null in my nullable, and it was no accident.

Attaching an arbitrary boolean state variable should not be the scope of Nullable.

Do we even have a isNullable trait?

Just is(typeof(T.init is null)) should be fine.

If somebody writes Nullable!(int*), then surely, they know what they are doing. If they are operating on T, then surely, it's better to give Nullable consistent behavior regardless of T.

What if they write Nullable!T in generic code?

[bearophile]

@JakobOvrum> Semantics are cleaner without the nasty double null situation.

It's not very different than using Nullable!(Nullable!int). If you introduce implicit type conversions for reference types, you are introducing more complexity in Nullable, so you are making its semantics more complex (and indeed you have to add more documentation to explain its behavour in ddoc comments). For me more complexity and special cases mean a less clear semantics.

[monarchdodra]

What if they write Nullable!T in generic code?

I already said in what I quoted: Then it's better to have consistent state, regardless of what T is.

[JakobOvrum]

I already said in what I quoted: Then it's better to have consistent state, regardless of what T is.

It does have consistent state regardless of T.

[bearophile]

@monarchdodra >Just use an alias.

As you see in Issue 9636 I am currently using an alias. But I'd like a bit of syntax sugar to avoid the alias (also because the alias needs to be in the outer scope, like at module level). If you think such usage is not worth the added complexity, than this is a judgement call :-)

@monarchdodra> Type t = Type.init unacceptable...

When "Type" is something longer like "immutable int[4]" or even longer, then it's not DRY and it becomes quite long.

[bearophile]

It seems I have to fight "tooths and nails" for every little thing I'd like added to Phobos :-)

[monarchdodra]

When "Type" is something longer like "immutable int[4]" or even longer, then it's not DRY and it becomes quite long.

OK, I agree. But how is that specific to Nullable ?

[MetaLang]

@monarchdodra

private alias DefaultNullableImpl = Typedef!(byte, 0, "std.typecons.DefaultNullable");
Why the whole Typedef? Just: private struct DefaultNullableImpl{}.

That's probably a better idea. I'll change it to a struct.

Also, I don't think it should be private, so users can provide their own function that can special case the null state.

How would that work?

Finally, I don't like the word "DefaultNullable". There's nothing "default" about it. I'd go for NullNullable, or (if possible) simply Nullable.NullState and Nullable.nullState. Something along those lines.

Well, if you want to be technical, it's the very definition of a default Nullable, as the constructor that takes defaultNullable doesn't perform any initialization and is simply "default constructed".
Nullable.(anything) is a no-go because it conflicts with Nullable(T). nullState isn't bad, though.

[MetaLang]

@JakobOvrum

Nullable.init is already in the null state, and it's also default-constructable. So why all this?

As Bearophile said, code like the following:
void foo(Nullable!(immutable int[4]) items = Nullable!(immutable int[4]).init) {}

It would be annoying to introduce an alias for a type that you're only going to use once, and often the type wrapped by Nullable does not have its own null state (which is probably why you're using Nullable in the first place), so there is no way to provide default initialization to null like above. Changing Nullable now to use typeof(null) is the obvious solution, but that may break code now. I agree that it is a design mistake that Nullable does not alias itself to T if T is a nullable type itself.

[bearophile]

@monarchdodra >But how is that specific to Nullable ?

For the use case explained in Issue 9636 the problems are specific for Nullable. I haven't found similar problems with other structs/typecons.

[monarchdodra]

For the use case explained in Issue 9636 the problems are specific for Nullable. I haven't found similar problems with other structs/typecons.

Anything with a "not yet initialized" state comes to mind. RefCounted, random generators, or pretty much any container.

And even then, for everything else, you are still just writing a variation of T t = U.init. You may not be typing T twice, but you are still relying on an implicit constructor to transform your "u" into "t".

Frankly, if it's not auto t = someThing, then it just isn't DRY.

[Geod24]

@bearophile

This is not true. Sometimes you want to avoid raw null pointers with a Nullable!(int*, null). It uses the same memory as a raw pointer, but it's guarded by asserts.

That's a nice trick to know. I also note it doesn't require a separate null state. So you're right, we shouldn't forbid nullable types for Nullable.

We could make Nullable(T) alias to T when T is already nullable, add a constructor and assignment overload for typeof(null) (fixing the default argument problem), and make nullify a UFCS function. That should be fairly backwards-compatible and fix both problems.

Agree. Although why should we keep x.nullify, if x = null performs the same action ?

Furthermore, Nullable can be also be to differentiate between "default state" and "not yet specified state". If my function taking a Nullable!(int*) encounters a non-null nullable that holds a null pointer, then I know that someone, somewhere, took the time to explicitly place that null in my nullable, and it was no accident.

The default state for pointer is "not yet specified", as null can never be a valid value for a pointer. The default state for non-nullable values (i.e. integer), is a value within the range of valid values. That's the gap Nullable fill.

[monarchdodra]

The default state for pointer is "not yet specified", as null can never be a valid value for a pointer.

Um what? The C standard library would disagree with you.

The default state for non-nullable values (i.e. integer), is a value within the range of valid values. That's the gap Nullable fill.

Pointers are but one way to implement nullable on a type. Both int* and Nullable!int achieve the same goal. The pointer nulls the value associated, not itself. null is the state of the pointer. It is not the absence of the pointer's state, which can only truly be achieve either with Nullable!(int*) or int**.

[monarchdodra]

It does have consistent state regardless of T.

auto getNullable!T(T t)
{
    auto a = Nullable!T(t);
    assert(!a.isNull); //or is it...?
    return a; 
}

[JakobOvrum]

@monarchdodra, indeed that demonstrates that in the current version one can always rely on the isNull property being false after parameterized construction (confusingly so, because assert(!a.isNull && a == null) can hold), which is not the case in the Nullable that I proposed. That's an argument against backwards-compatibility, not an inconsistency or complexity of the proposed Nullable.

Again, attaching an arbitrary boolean state to a type should not be within the scope of Nullable. Nullable is about null.

[MetaLang]

Again, attaching an arbitrary boolean state to a type should not be within the scope of Nullable. Nullable is about null.

That ship has sailed. Changing it now will potentially break code.

[JakobOvrum]

That ship has sailed. Changing it now will potentially break code.

Phobos has a number of shoddy or incomplete types or modules, and I think Nullable is one of them. assert(!a.isNull && a == null) is just terrible (it is also telling that C# disallows reference types for T in its own Nullable type). The standard library should provide top quality interfaces, and if we have to break code to provide that, then in my opinion, so be it.

Anyway, I'm fine with judging this PR against the current (broken) design, I admit there is no need to expand the scope of this pull request.

Are there any other situations apart from default arguments where defaultNullable is useful?

One thing to note is that while it doesn't save on typing, one can opt to use overloading to achieve the same effect but with arguably nicer documentation than the Nullable!ComplicatedTypeHere.init default argument:

void foo(Nullable!(immutable int[4]) items);
void foo();

Secondly, although this one will certainly vary, needing to use a complicated type for T that isn't already aliased and needs to be used in a default argument is probably a very rare situation. I'd guess that most uses of Nullable would be with simple primitive types or with structs. Do we really want to add two new public symbols just for this extreme edge case, when the only problem it solves is a bit of typing?

[monarchdodra]

assert(!a.isNull && a == null)

Arguably, I think the issue you are showing here is abusive use of alias this for implicit conversion, when it was actually designed to emulate structure inheritance.

assert(!a.isNull && a != null && a.value == null); makes more sense.

[bearophile]

@monarchdodra: >Anything with a "not yet initialized" state comes to mind. RefCounted, random generators, or pretty much any container.

With the current design of std.random module, the random generators can't be passed by value, but only by ref (or by pointer), and you can't add a default value to a ref argument. So this is an invalid example.

If the problem is really general as you say, then are you suggesting a general solution, some "defaultInitialized" that works with them all (including Nullable)?

[bearophile]

@JakobOvrum: >One thing to note is that while it doesn't save on typing, one can opt to use overloading to achieve the same effect

In my use case the function had one nullable argument, and four nullable arguments that should default to null. You can't use overloading here, it makes the code awful. But I agree this is an uncommon use case.

[monarchdodra]

If the problem is really general as you say, then are you suggesting a general solution

I was not actually suggesting a general solution, as I don't really think there is a problem here. At least, not a problem that warrants the current change suggested here.

some "defaultInitialized" that works with them all (including Nullable)?

That's called T.init. That said, do you think it is possible to have a (library) keyword called defaultInitialized that works in a fashion similar to nullptr? As in, it's a unique type defaultInitialized_t, and can be cast to anything. EG:

T t = defaultInit;

That's indeed something I think would be a more generic "solution". But even then, I wouldn't be sure an extra concept is warranted here :/

[MetaLang]

This is a simple change. Can we either merge it or close it? There's no good alternative to the problem described issue 9636, when you have a long type name and don't want to introduce an external alias.

Or, we can close this, and I'll create another pull making typeof(null) do the same thing as defaultNullable, which will be a breaking change (but the best possible solution IMO).

Or, I can change defaultNullable to defaultInit, and any type can make use of it by adding a constructor, opAssign, etc. that accepts typeof(defaultInit).

[JakobOvrum]

Or, we can close this, and I'll create another pull making typeof(null) do the same thing as defaultNullable, which will be a breaking change (but the best possible solution IMO).

Nullable!(int*) a = null;
assert(a.isNull);

int* p = null;
Nullable!(int*) b = p;
assert(!b.isNull);

Let's not do that. It causes null and a null T.init to behave subtly differently because of the double null semantics.

Nullable should never have allowed for this double null in the first place..

[MetaLang]

@JakobOvrum

Let's not do that.

What I mean is, removing double-null from Nullable and allowing assignment of null or construction from null of Nullable to do what the proposed defaultNullable does.

[JakobOvrum]

What I mean is, removing double-null from Nullable and allowing assignment of null or construction from null of Nullable to do what the proposed defaultNullable does.

Well that's exactly what I've been suggesting.

Once you remove the double null from Nullable!T, it can just alias itself away when T is already a nullable type.

[MetaLang]

Yes, and that will be a breaking change.

[MetaLang]

Ping

[Geod24]

I would be in favor of closing it until we come to a decision here.
As @JakobOvrum said, having separate null state should have never been possible in the first place.
Sorry for the wasted effort though!

[MetaLang]

This is always going to be a problem unless we want to break Nullable, but I guess I'll have to close this for now.