casting json result to List<List<String>>

[drpond1]

Using IntelliJ CE for a web app I have the following concern. debugging with Dart 2.16

I decode an object with json and then cast it to a List<List<String>> meta
Later, I try
for (List<String> list in meta) {...

This compiles but produces a runtime error of
List<dynamic> is not a List<String>

The type in the debugger is cast._new (I think) and the item
looks like a List<String>.

It seems to me that this is not a great behavior. I think the runtime
ought to know that this is ok, but that further runtime checks will
need to be done when the list in the for loop is read.

If this is not going to be allowed at runtime, then I think the type of meta should be
a compile time syntax error.
Some way of specifying List<List<cast.String>> might be an alternative, although List<List<dynamic>> is ok.

If needed, I can produce and test a simple example to duplicate the behavior.

[eernstg]

Dart maintains a representation of the actual type arguments of any given instance of a generic class at run time. This is needed in order to maintain soundness in various situations, including the situation where we execute things like myList.add(1), where myList is a variable of type dynamic.

This means that a List<dynamic> that contains a number of elements each of which is a List<dynamic> is not the same thing as a List<List<dynamic>>, not to mention List<List<String>>. The latter has a representation where the type argument is known to be a List<String> at run time, and each of the elements is a list whose representation knows that its type argument is String.

If you perform a type cast (like myList as List<List<String>>) then you will simply get a run-time error, because it isn't true that myList is a List<List<String>> when it is actually a List<dynamic> that contains a bunch of List<dynamic> that contain a bunch of strings.

So in order to allow the list-of-lists to be typed as List<List<String>> you have to copy the outermost list, and copy each of the contained lists, and make sure that the copy has the desired actual type arguments.

var json = <dynamic>[
  <dynamic>['Hello'],
];

void main() {
  var typedJson = <List<String>>[
    for (List innerList in json) List<String>.from(innerList)
  ];
  print(typedJson.runtimeType);
}

If you do not wish to copy the entire structure of the given nested list then you can also work on the objects that you actually have, which means that every element of a list has type dynamic, rather than List<String> respectively String, and the type casts will then occur during the traversals.

[lrhn]

To be precise, I think that when you say that you cast to List<List<String>>, you do decodedJson.cast<List<String>>().
That call is successful because the receiver is a list, and the cast method always succeeds in creating a new list which may throw on any member access.

As @eernstg says, those member accesses do throw because the members are List<dynamic>, not List<String>.
If you did .cast<List<dynamic>>(), it would probably work (assuming the outer list really contains only lists).
And as he also says, you need to create new inner lists that are List<String>, which means creating a new outer list (or destructively updating the one you have). Like any of the following:

var typedJson = [...json.map((list) => list.cast<String>())]; // aka json.map((list) => list.cast<String>()).toList()
var typedJson = [for (var list in json) <String>[...list]]; // relies on implicit downcast from dynamic.
var typedJson = json..asMap().forEach((i, v) => json[i] = v.cast<String>()).cast<List<String>>();

[drpond1]

I understand what has been said. I also have no problem producing code which produces what I want. What bothers me is that
List<List<String>> meta does not mean the meta has type List<List<String>>. There is no indication of a kind of "dynamic" in the type declaration.

Something like List<List<cast.String>> might
be a better reflection
of the actual "type" assuming the runtime type checker is going to check for String in the inner list. If not, List<List<dynamic>> would be better.

Then I can say
List<List<cast.Sring> meta =foo.cast<List<String>>() and
for (List<cast.String> list in meta) { ...
The static type checking is List<String> ... no change.
The run-time checking
is now List<cast.String> as a kind of List<dynamic>. No error message.
And ideally, a read access to an element of ````List<cast.String> is checked for typeString```.

The "breaking the language problem" is a concern for the error message, but perhaps there could be some way to at least turn on a warning for the unclear typingList<List<String>>.

[eernstg]

@drpond1, I'm not 100% sure about the intended rules around type arguments like cast.String, but it seems likely that you want it to impose the typing discipline lazily:

void main() {
  List<dynamic> xs = [1, true];
  List<cast.int> ys = xs; // This would be OK, but implies dynamic checks later on.
  for (int i in ys) {
    print(i); // Prints '1', then throws because `true` isn't an `int`.
  }
}

We could have a language mechanism like this, but we do already have a library feature which is very similar:

void main() {
  List<dynamic> xs = [1, true];
  List<int> ys = xs.cast<int>(); // Create a wrapper around `xs` that checks each element type at run time.
  for (int i in ys) {
    print(i); // Prints '1', then throws because `true` isn't an `int`.
  }
}

[lrhn]

The cast method has no effect on the type system.
The type List<List<String>> has no mention of dynamic and it does not allow a List<dynamic> or a List<List<dynamic>> at all.
The declaration List<List<String>> meta does indeed guarantee that meta has a runtime type which is a subtype of List<List<String>> (or null, pre-null safety). That static type doesn't guarantee anything about the behavior of the methods of that object, only what its runtime type is. The methods may throw.

When you do .cast<List<String>>() on a List<dynamic>, the result is a List<List<String>>. However, cast is dangerous and must be used with care. It does not guarantee type safety, and if you use it on a list where the elements are not actually List<String> objects, it will throw when used.
The cast method is a dynamic cast, its failures all happen at runtime. The warning you need is that you are using cast at all. After that call, there is nothing the static type system can do for you.

Now, if we had a feature like List<cast.String>, basically allowing cast.type as a type which is really dynamic, but which is implicitly downcast to type when needed in a covariant position ... that could potentially work.
It has issues, though.

You can't assign a List<cast.String> to List<String>, because the downcasting is on the elements, not the list itself. A List<cast.String> may very well be a List<dynamic>, and that is not a List<String>, so you cant use a List<cast.String> as an argument to a function expecting List<String>.

In that way, it's more reminiscent of a view. It's a static behavior put on top of a List<dynamic> which enforces that the elements must be of a specific type, but it's not really a List of that type.
At least the cast method does create a proper List<type>.

[drpond1]

No disagreements with either reply.

This is not an issue of the static typing... that works exactly how I expect. The problem is that the runtime ends up treating the items in the list as List<dynamic>
and I get a typing error. If that is how it is going to work,
then cast<List<String>> should not be allowed.

So either:

1. Don't allow cast to handle that kind of thing. The coder
   should specify cast<(List<dynamic>>() and
   use things likeList<List<dynamic>>meta.
2. Have the runtime checker do a more clever check,
   If useful, enhance the language to help the runtime type
   checker

The possible language enhancement if needed is
something like List<cast.String>>.

List<cast.String> is not a new type. It is a a directive similar to List<dynamic>.
The runtime checks that the items of ``List<List<cast.String>>```
are:

1. a list.
2. the elements of the inner lists when accessed are Strings.

If you think I am wrong about this, there should be an example where cast<List<String>>() is currently useful.

[lrhn]

This is not an issue of the static typing... that works exactly how I expect. The problem is that the runtime ends up treating the items in the list as List<dynamic>

It's not just treating them as List<dynamic>. They are List<dynamic> objects. It can't treat them as anything but.
You want to treat the elements of the list as List<String>, and they aren't. They might only (currently) be containing strings, but that doesn't matter. They are List<dynamic>s which contains strings. Those are not assignable to List<String>, and you have to create a new list object in order to get something which is a List<String>.

and I get a typing error. If that is how it is going to work,
then cast<List<String>> should not be allowed.

Doing .cast<List<String>>() is perfectly valid if all the elements of the list are List<String>s.
You should only use .cast<X> on a list if all the elements of the list are already of type X.
The list itself is presumably a supertype of List<X> (otherwise you don't need the cast to begin with).

So,

var o = <Object>[<String>[], <String>["foo"]];
var s = o.cast<List<String>>();  // Correct use of `cast`
var l1 = s[0]; // l1 has type List<String> and the read succeeds.

is a valid use of .cast<List<String>>().

The thing that makes cast methods dangerous is that they

1. can introduce runtime errors, and
2. do so much later than when you call cast.

We don't want cast to eagerly check every element of the collection. Both because it's expensive, if you're looking at every element anyway, then you could just create a new collection instead, and because it's not sufficient, because the underlying collection can be changed after the wrapper has been created. So, the call to cast succeeds, but if you ever find an element of the collection which doesn't have the cast-to type, your program crashes then. It's like an implicit cast. Sometimes it's just what you need. In very many cases it's not, and you should not be using cast then.

[drpond1]

1. My recommendation for a flagging an error is invalid. Thank you for your help.

2. Having a mechanism to "convert" a type to something like List<cast.String> might work.
   However, there is no such feature in the language,
   and using the correct type with extra casting should be
   adequate.
   There is not a sufficient need to change the language.

3. In summary

   The code in the original example should be changed.
   Use
   List<List<dynamic> meta and cast the decoded object toList<dynamic>.
   The json decoder might produce a List of Llst so the cast function might help upgrade the type. But the decoder does not produce a List<String> and the cast
   function does not change the entries of the list from dynamic to String. it just produces a different view of the type. (Hence no static typing error in declaring meta.)
   With the original code at runtime you will get an error
   when you access one of the elements and discover that the
   type is actually List<dynamic>.

4. There is insufficient need to change the language and so this issue should be closed.

[eernstg]

@drpond1, thanks for your very constructive approach to discussions like this one! I'll close this issue, we'd just create new ones as needed.

One thing to note is that there is a proposed language feature known as 'views', and that feature offers support for establishing type safe access to dynamic object structures without creating any wrapper objects (view methods are similar to extension methods in that they are resolved statically).

Again, the assumption is that the dynamic object structure satisfies some constraints, and there will be run-time errors if the given object structure actually violates those constraints. JSON decoding is a typical example where this can be a reasonable assumption: We might very well receive a List<dynamic> containing a sequence of List<dynamic>, each of which contains a sequence of String objects. We would rely on having an object structure as described because it would be obtained from a provider who has committed to deliver their data in this format. You could also say that we're relying on having this specific structure because the data follows a specific JSON schema.

Here's how we can safely navigate an object structure with that schema, using a couple of views to model the schema. I'm using 'lls' to abbreviate List of List of String in names, and 'ls' to abbreviate List of String, with the usual CamelCasing, indicating that an 'lls' is a List<dynamic> containing List<dynamic> containing String objects, etc.

// Using the proposed feature 'views'. Minimal example, just providing `[]`.

view LlsView on List<dynamic> {
  LsView operator[](int index) => this[index]!;
}

view LsView on List<dynamic> {
  String operator[](int index) => this[index]!;
}

var json = <dynamic>[<dynamic>[' Hello!']];

void main() {
  LlsView lls = json;
  LsView ls = lls[0];
  print(ls[0].trimLeft());
}

The assignment of json to lls is allowed by the compiler because we have declared the views to allow "adopting" the view implicitly (so List<dynamic> is a subtype of LlsView). The initialization of ls is allowed, because the return type of lls[_] is LsView, but lls = lls[0] would be a compile-time error. Finally ls[0] has the type String, so we can call trimLeft(). In particular, ls[0] does not have the type dynamic, and things like ls[0].isEven is a compile-time error.

So this means that we've keeping track of the underlying structure as declared in the JSON schema (and encoded in LlsView and LsView), such that the type checker will prevent us from using lls[0] as a String, or ls[0] as an int or a List<...>. In other words, we're maintaining the same discipline as we would have had with an actual List<List<String>>.

But on the other hand, the view methods are technically just like top-level functions: There is no wrapper object corresponding to the LlsView or LsView type, they are simply represented by the underlying List<dynamic> at run time. So there's no need to create and manage any wrapper objects, which means that we can adopt the LlsView treatment of a large object structure, say, millions of objects, without ever creating those millions of wrapper objects that we'd need in order to do this without views, and without creating new lists all the way down (again: perhaps millions of objects).

In this kind of situation, I would expect that we would use code generation based on an actual document specifying a JSON schema of sorts, such that we'd get all the different kinds of expectations in a given object structure correct without a lot of manual work.