Proposal: Nullable reference types and nullability checking

[MadsTorgersen]

Null reference exceptions are rampant in languages like C#, where any reference type can reference a null value. Some type systems separate types into a T that cannot be null and an Option<T> (or similar) that can be null but cannot be dereferenced without an explicit null check that unpacks the non-null value.

This approach is attractive, but is difficult to add to a language where every type has a default value. For instance, a newly created array will contain all nulls. Also, such systems are notorious for problems around initialization and cyclic data structures, unless non-null types are permitted to at least temporarily contain null.

On top of these issues come challenges stemming from the fact that C# already has null-unsafe types, that allow both null values and dereferencing. How can a safer approach be added to the language without breaking changes, without leaving the language more complex than necessary and with a natural experience for hardening existing code against null errors in your own time?

Approach: Nullable reference types plus nullability warnings

The approach suggested here consists of a couple of elements:

• Add a notion of safe nullable reference types T? to the language, in addition to the current ones T which are from now on non-nullable.
• Detect and warn on cases where nullable types are dereferenced, or when null is values are assigned to (or left in) non-nullable variables.
• Offer opt-in means to deal with breaking behavior and compat across versions and assemblies.

The feature will not provide airtight guarantees, but should help find most nullability errors in code. It can be assumed that most values are intended not to be null, so this scheme provides the minimal annotation overhead, leaving current code to be assumed non-nullable everywhere.

Nullable reference types are helpful, in that they help find code that may dereference null and help guard it with null checks. Making current reference types Non-nullable is helpful in that it helps prevent variables from inadvertently containing an null value.

Nullable and non-nullable reference types

For every existing reference type T is now "non-nullable", and there is now a corresponding nullable reference type T?.

Syntactically speaking, nullable reference types don't add much, since nullable value types have the same syntax. However, a few syntactic corner cases are new, like T[]?.

From a semantic viewpoint, T and T? are mostly equivalent: they are the same type in that they have the same domain of values. Specifically, because the system is far from watertight, non-nullable reference types can contain null. The only way in which they differ is in the warnings caused by their use.

For type inference purposes T and T? are considered the same type, except that nullability is propagated to the outcome. If a reference type T is the result of type inference, and at least one of the candidate expressions is nullable, then the result is nullable too.

If an expression is by declaration of type T?, it will still be considered to be of type T if it occurs in a context where by flow analysis we consider it known that it is not null. Thus we don't need new language features to "unpack" a nullable value; the existing idioms in the language should suffice.

string s;
string? ns = "hello";

s = ns; // warning
if (ns != null) { s = ns; } // ok

WriteLine(ns.Length); // warning
WriteLine(ns?.Length); // ok

Warnings for nullable reference types

Values of nullable reference types should not be used in a connection where a) they are not known to (probably) contain a non-null value and b) the use would require them to be non-null. Such uses will be flagged with a warning.

a) means that a flow analysis has determined that they are very likely to not be null. There will be specific rules for this flow analysis, similar to those for definite assignment. It is an open question which variables are tracked by this analysis. Just locals and parameters? All dotted names?

b) means dereferencing (e.g. with dot or invocation) or implicitly converting to a non-nullable reference type.

Warnings for non-nullable reference types

Variables of non-nullable reference types should not be assigned the literal null or default(T); nor should nullable value types be boxed to them. Such uses will result in a warning.

Additionally, fields with a non-nullable reference type must be protected by their constructor so that they are a) not used before they are assigned, and b) assigned before the constructor returns. Otherwise a warning is issued. (As an alternative to (a) we can consider allowing use before assignment, but in that case treating the variable as nullable.)

Note that there is no warning to prevent new arrays of non-nullable reference type from keeping the null elements they are initially created with. There is no good static way to ensure this. We could consider a requirement that something must be done to such an array before it can be read from, assigned or returned; e.g. there must be at least one element assignment to it, or it must be passed as an argument to something that could potentially initialize it. That would at least catch the situation where you simply forgot to initialize it. But it is doubtful that this has much value.

Generics

Constraints can be both nullable and non-nullable reference types. The default constraint for an unconstrained type parameter is object?.

A warning is issued if a type parameter with at least one non-nullable reference constraint is instantiated with a nullable reference type.

A type parameter with at least one non-nullable constraint is treated as a non-nullable type in terms of warnings given.

A type parameter with no non-nullable reference constraints is treated as both a nullable and a non-nullable reference type in terms of warnings given (since it could without warning have been instantiated with either). This means that both sets of warnings apply.

? is allowed to be applied to any type parameter T. For type parameters with the struct constraint it has the usual meaning. For all other type parameters it has this meaning, where S is the type with which T is instantiated:

• If S is a non-nullable reference type then T? refers to S?
• Otherwise, S? refers to S

Note: This rule is not elegant - in particular it is bad that the introduction of a struct constraint changes the meaning of ?. But we believe we need it to faithfully express the type of common APIs such as FirstOrDefault().

Opting in and opting out

Some of the nullability warnings warn on code that exists without warnings today. There should be a way of opting out of those nullability warnings for compatibility purposes.

When opting in, assemblies generated should contain a module-level attribute with the purpose of signaling that nullable and non-nullable types in signatures should generate appropriate warnings in consuming code.

When consuming code references an assembly that does not have such a top-level attribute, the types in that assembly should be treated as neither nullable nor non-nullable. That is, neither set of warnings should apply to those types.

This mechanism exists such that code that was not written to work with nullability warnings, e.g. code from a previous version of C#, does indeed not trigger such warnings. Only assemblies that opt in by having the compiler-produced attribute, will cause the nullability warnings to happen in consuming code accessing their signatures.

When warnings haven't been opted in to, the compiler should give some indication that there are likely bugs one would find by opting in. For instance, it could give (as an informational message, not a warning) a count of how many nullability warnings it would have given.

Even when a library has opted in, consuming code may be written with an earlier version of C#, and may not recognize the nullability annotations. Such code will work without warning. To facilitate smooth upgrade of the consuming code, it should probably be possible to opt out of the warnings from a given library that will now start to occur. Again, such per-assembly opt-out could be accompanied by an informational message reminding that nullability bugs may be going unnoticed.

Libraries and compatibility

An example: In my C# client code, I use libraries A and B:

// library A
public class A
{
  public static string M(string s1, string s2);
}

// library B
public class B
{
  public static object N(object o1, object o2);
}

// client C, referencing A and B
Console.WriteLine(A.M("a", null).Length);
Console.WriteLine(B.N("b", null).ToString());

Now library B upgrades to C# 7, and starts using nullability annotations:

// upgraded library B
public class B
{
  public static object? N(object o1, object o2); // o1 and o2 not supposed to be null
}

It is clear that my client code probably has a bug: apparently it was not supposed to pass null to B.N. However, the C# 6 compiler knows nothing of all this, and ignores the module-level attribute opting in to it.

Now I upgrade to C# 7 and start getting warnings on my call to B.N: the second argument shouldn't be null, and I shouldn't dot into the return value without checking it for null. It may not be convenient for me to look at those potential bugs right now; I just want a painless upgrade. So I can opt out of getting nullability warnings at all, or for that specific assembly. On compile, I am informed that I may have nullability bugs, so I don't forget to turn it on later.

Eventually I do, I get my warnings and I fix my bugs:

Console.WriteLine(B.N("b", "")?.ToString());

Passing the empty string instead of null, and using the null-conditional operator to test the result for null.

Now the owner of library A decides to add nullability annotations:

// library A
public class A
{
  public static string? M(string s1, string s2); // s1 and s2 shouldn't be null
}

As I compile against this new version, I get new nullability warnings in my code. Again, I may not be ready for this - I may have upgraded to the new version of the library just for bug fixes - and I may temporarily opt out for that assembly.

In my own time, I opt it in, get my warnings and fix my code. I am now completely in the new world. During the whole process I never got "broken" unless I asked for it with some explicit gesture (upgrade compiler or libraries), and was able to opt out if I wasn't ready. When I did opt in, the warnings told me that I used a library against its intentions, so fixing those places probably addressed a bug.

[MgSam]

So are the thoughts on the ! operator from the Aug 18 notes not part of the official proposal yet?

Additionally, fields with a non-nullable reference type must be protected by their constructor so that they are a) not used before they are assigned, and b) assigned before the constructor returns. Otherwise a warning is issued. (As an alternative to (a) we can consider allowing use before assignment, but in that case treating the variable as nullable.)

This rule seems like a non-starter to me, as it is impossible for the compiler to verify this if you use a helper method called from the constructor to assign fields. That is unless you provide an in-line way of disabling warnings (essentially telling the compiler that you know for certain the fields are being assigned). ReSharper has such a feature to disable warnings.

[HaloFour]

@MgSam Isn't that currently a pain-point with non-nullable fields in Apple Swift? What's worse is that the base constructor could call a virtual method overridden in the current class that could access those fields before they've been assigned so not even the constructor is good enough.

[tpetricek]

This looks nice!

Can the proposal be clarified on how are the nullability annotations going to be stored at the CLR level? (I suppose these have no runtime meaning and they would just produce a .NET attribute or some other sort of metadata?)

It would be nice if some future version of the F# compiler could read & understand these and offer some sort of "strict" mode where nullable types are automatically exposed as option<'T> (or something along those lines).

One other possible related idea would be to have an annotation not just when referring to a type, but also when defining a type - for example, types defined in F# do not allow null value (when used safely from F#) and so when the C# compiler sees them, it should (ideally) always treat them as non-nullable, but when working with a type that came from some old .NET library that has nulls everywhere, it makes sense to treat the type as nullable by default. (This would perhaps make things too complicated - but some declaration-site/use-site option might be worth considering...)

[rmschroeder]

Seconding Tomas' points above, this is a great as it stands, but it would be nice if there could be some way to enable a 'strict' mode that guaranteed no nulls and if that could be trusted by consumers of libraries, or is there some way to determine if a dependency did not ignore the warnings?
I would also like to hear about how this would be exposed via metadata, reflection, etc.

[mariusschulz]

What I love about this approach is that non-nullability would be the default for reference types, which I think is a lot better than having to opt in to it explicitly every single time (like string! or object!).

@HaloFour In Swift, you can't call a base class' constructor until all (non-nullable) fields have been initialized. That prevents exactly the problem you mentioned.

[sfiruch]

I like this idea and implementation very much. +1 vote to enable (possible) warnings for arrays of non-nullable references.

[YuvalItzchakov]

First of all, kudos to the language team for this proposal. It definitely looks like effort and time has been put to thinking about this.

Regarding the proposal, It seems to me that non-nullability in C#, given its age and maturity would be confusing to many developers, senior and junior as one. Given that the proposal is mainly around compiler time warnings and not errors, this would lead to even more confusion, as it would feel like something developers can easily ignore. Also, we are so used to nullability being around structs, that suddenly allowing them on reference types can add to that same confusion and peculiar feeling.

Think feature doesn't "feel" like it can be done properly without baking it into the CLR. I think attempts to work around not creating a breaking change wouldn't be as good as it can be.

[FlorianRappl]

One question that directly appeared was how library authors should handle parameters, which are not supposed to be null, but are given as null.

The example reads:

public static string? M(string s1, string s2); // s1 and s2 shouldn't be null

Now prior to C# 7 we could introduce a runtime check such as:

public static string M(string s1, string s2) // s1 and s2 can be null
{
  if (Object.ReferenceEquals(s1, null))
    throw new ArgumentNullException(nameOf(s1));
  if (Object.ReferenceEquals(s2, null))
    throw new ArgumentNullException(nameOf(s2));
  /* ... */ // won't crash due to s1 or s2 being null
}

This result, of course, in a runtime error if null is passed-in. In C# 7, knowing that the input should be non-null, a warning in generated. Now we could omit these checks and maybe have a crash, which is a consequence of passing in null references, however, since the user has been warned the origin should be obvious.

public static string? M(string s1, string s2) // s1 and s2 shouldn't be null
{
  /* ... */ // could crash due to s1 or s2 being null
}

Is it supposed to be used like that or would the recommendation be to have both kind of checks (a runtime guard and the compiler mechanism to have an initial warning)?

Needless to say that the runtime guard may have other benefits and provides backwards compatibility, but also adds noise that could be reduced with the new approach.

Additionally I agree to @YuvalItzchakov (especially regarding baking it into the CLR).

[sandersaares]

@FlorianRappl I imagine the null-checking operator from #5033 would be an ideal match for that scenario if it could also apply to method arguments.

public static string? M(string s1!, string s2!)
{
}

Or maybe with an alternate syntax (inconsistent with null-checking operator elsewhere but potentially more suitable for method arguments):

public static string? M(string! s1, string! s2)
{
}

Such an approach would check for null (at runtime) in addition to emitting non-nullable reference warnings at compile time.

The visuals look a bit odd at first glance but having an easy way to specify "throw if null" is very valuable, especially if we also consider that nullability analysis by the compiler would be one basis for the non-nullable reference type warnings, meaning that you would normally run into the following issue:

public static void Bar(string s) { ... }

public static void Foo(string? s)
{
    Argument.VerifyIsNotNull(s, nameof(s)); // Think green, save energy by saving 1 line of code!

    Bar(s1); // Warning: potentially null value used.
    // The compiler can't tell (without extra annotations) that a null check was done out of band.
}

With the null-checking operator applying to a method argument, this is no problem:

public static void Bar(string s) { ... }

public static void Foo(string s!)
{
    Bar(s1); // The compiler knows that a null check was done, so no problem here.
}