Null configuration elements deserialized as empty strings

[kenlyon]

Describe the bug

When a property is explicitly set a value of null in appsettings.json, I expected that to be null after the configuration was loaded, but it actually has a value of "". This led to a bug in our code that was hard to track down. Ideally, I'd like a way to change this behaviour, although it is possible to work around it, now that I know this is how it behaves.

To Reproduce

Steps to reproduce the behavior:

1. In Visual Studio 2017, create a project of type "ASP.NET Core Web Application".

2. When prompted for a template, choose the following options:

   • .NET Core
   • ASP.NET Core 2.1
   • API

3. Add a new class called "MyConfig" and add the following properties:

    public string Prop1 { get; set; }
    public string Prop2 { get; set; }  
    public string Prop3 { get; set; }  
    public string Prop4 { get; set; }
   

4. Add the following elements to the "appsettings.json" file:

    "Prop1": "Prop 1 Value",
    "Prop2": "",
    "Prop3": null
   

   (IMPORTANT: Do not add an entry for Prop4.)

5. Edit the Startup class:

   • Append this line to the ConfigureServices method:

     services.Configure<MyConfig>(Configuration);
     

   • Append the following parameter to the Configure method:

     IOptions<MyConfig> config
     

6. Add a breakpoint in the Configure method and debug the project in IIS Express.

7. When the breakpoint is hit, inspect the value of the config parameter.

Results

Property	Expected Value	Actual Value
config.Value.Prop1	"Prop 1 Value"	"Prop 1 Value"
config.Value.Prop2	""	""
config.Value.Prop3	null	""
config.Value.Prop4	null	null

Additional context

Output of dotnet --info:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional>dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.403
 Commit:    04e15494b6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.403\

Host (useful for support):
  Version: 2.1.5
  Commit:  290303f510

.NET Core SDKs installed:
  1.0.0-preview2-003131 [C:\Program Files\dotnet\sdk]
  1.0.4 [C:\Program Files\dotnet\sdk]
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.0.2 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.100 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.200 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional>

[oliverjanik]

Any update here? This is truly a violation of principle of least surprise.

[vdailly]

Hello,

Some days ago I had the same disappointed experience regarding configuration.
And as many other I spend a lof of time to understand the issue, and digging into source code.
With the number of opened issues on this subject, the IConfiguration and related, while very convenient and powerful miss something probably.

Let's consider this minimal Json example:

{
   "obj1": [ ],
   "obj2": [
      "value"
    ]
}

The IConfiguration result is as follow:
obj2:0 "value"

obj1 has completely disappeared between internal JsonParser and IConfiguration.
Nevertheless when binding on a Dictionnary<string, T> the key obj1 is meaningful, even if T get its default values.
Providing explicitly a property in the configuration of an application means this information is meaningful for the application.

In my own case I add the following configuration:

{
  "obj1": {
     "prop1" : [ ]
   },
  "obj2": {
    "prop1": [
       { "subprop" : "value" }
     ]
  }
}

class Conf {
  public List<Element> Prop1 {get; set; } = new List<Element>();
}

class Element {
  public string SubProp {get; set;} = string.Empty;
}

services.Configure<Dictionnary<string, Conf>>(Configuration);

For me "objx" keys were clearly meaningful, and I supposed to get a Dictionnary<string, Conf> with 2 keys where obj1 should be a default Conf object. But obj1 key-value pair was meaningful enough to break the application when I've updated prop1 to an empty Json array without a clear knowledge that obj1 would disappear.

This topic can be splitted into 2 issues:

• parent properties lost with a unique property is set as null Array
• distinguish an explicit null from an explicit empty Json array for the internal IConfiguration keys

Issue 1: parent properties lost on unique null Json Array

Whatever the depth of the Json, if there is only one empty Json array as leaf property, then all parent Json objet are lost.
As intermediate solution I would propose to store all the path, whatever the Json kind of property encountered on leaf node.
In the example provided, this would store obj1:prop1 in IConfiguration.

So the Binder may create the default Conf`` element of the example. Nevertheless this is only a part of the solution, because we don't exactly know what to store as a value for the obj1:prop1` configuration path. By default this would be null or empty string (I've don't checked). Here the value is not null, it is an empty Json array.

But this lead to the second issue.

An alternate solution is to update the configuration as follow (by example):

[
   { 
      "name" : "obj1",
      "prop1" : [ ] 
   },
   {
     "name" : "obj2",
     "prop1" : [
       { "subprop" : "value" }
     ]
   }
]

But this is to the caller to transform the List<Conf> to a Dictionnary<string, Conf>. And from the configuration file point of view, Json provide unique property name, while the intent is not clear from a Json array that application require to have a unique name property.

Issue 2: Distinguish explicit null from explicit empty Json array

The main question here is to distinguish an explicit null value from an explicit empty Json Array.
As mentioned in several issues, an explicit Json null must be a null.
But what is an empty Json Array when stored in the IConfiguration?

• This is not a null.
• This is not an indexed configuration path:0, because this would mean there is 1 element in the array.

Probably the best may be handle empty collection as String.Empty.
For the solution, simply use IConfiguration[path].GetChildren().Any() to distinguish a null path element with children from an empty Json array.
Meaning that if the Binder encounter by reflection a generic IEnumerable or Collection, Array or whatever collection, the behavior may be:

• if IConfiguration[path] == String.Empty || IConfiguration[path].GetChildren.Any() : Initialize the collection and recurs children.
• else : this is an explicit null && no children exist: so explicitly set the collection to null.

Propositions

I don't have a PR to submit, because I don't have the tooling for the .NET5 and so on. But Issue1 this is related to file JsonConfigurationFileParser. Something like that may solve the issue1 described here.

var arrayElements = value.EnumerateArray();
if (arrayElements.Count == 0) {
    string key = _currentPath;
    if (_data.ContainsKey(key))
    {
        throw new FormatException(SR.Format(SR.Error_KeyIsDuplicated, key));
    }
    _data[key] = value.ToString();   // or String.Empty internal JsonElement.ToString() does not already set to string.Empty.
} else {
    foreach (JsonElement arrayElement in arrayElements) {
        EnterContext(index.ToString());
        VisitValue(arrayElement);
        ExitContext();
        index++;
    }
}
break;

If I have some time I'll try to search in the binder what to eventually update.

and for issue 2 this may be ConfigurationBinder.cs

private static object BindInstance(Type type, object instance, IConfiguration config, BinderOptions options)
{
           //best update here because value is available before attempting bind on collections.
          if (convertedValue == String.Empty || config.GetChildren().Any()) {
            ... 
          }
         else {
            return convertedValue;
         }
}

[maryamariyan]

In the PR #43297 that just got closed we discussed we need to fix this by adding an API, introducing a flag

Link to guideline for formal API proposal is available in api-review-process.md