Problem converting List<dynamic> to nested lists

[ChristianKleineidam]

I'm saving data objects in sembastdb and reading them again and then they come back as dynamic. Normally, Dart makes it easy to cast dynamic objects to their real type.

This doesn't seem to work with nested lists. To be able to read convert List back to List<List<List>>, currently, I seem to have to do:

    var asset = this.leaflet.assetData;
    List<List<List<String>>> newList = [];
    for (var listFirst in asset) {
      List<List<String>> firstList = [];
      for (var listSecond in listFirst) {
        List<String> secondList = [];
        for (var listThird in listSecond) {
          secondList.add(listThird);
        }
        firstList.add(secondList);
      }
      newList.add(firstList);
    }
    return newList;

And can't simply do:

return asset as List<List<List<String>>>

I can't even do:

    var list = asset
        .map((first) => first
            .map((second) =>
                second.map((third) => (third + "") as String).toList())
            .toList())
        .toList();
    return newList as List<List<List<String>>>;

Which produces the error type 'List<dynamic>' is not a subtype of type 'List<List<List<String>>>' in type cast.

I first opened an issue with SembastDb but it seems to be a more general problem with nested list and dynamic.

This seems to be all happening while at the same time in the debugger the actual types look like they should be convertible.

[lrhn]

It's dynamic all the way down, so you would have to do something like:

var newList = asset
        .map<List<List<String>>>((first) => first
            .map<List<String>>((second) =>
                second.map<String>((third) => (third + "") as String).toList())
            .toList())
        .toList();
    return newList as List<List<List<String>>>;

or

var newList = <List<List<String>>>[for (var a1 in asset) [for (var a2 in a1) [...a2]]];

[lrhn]

The cast/castFrom methods are shallow casts. They work on the individual elements of the list by casting between the original type and the cast-to type as necessary to pretend to be of the cast-to type, while really being backed by the original type.

When you do list1.cast<List<int>> you get a list that you can only get List<int> objects out of, and can only put List<int> objects into, backed by the actual List<List<Object>> that list1 refers to.
When you then do list2[0], you get the List<Object> created by [1, 2] and try to cast it to List<int>. That fails predictably.
If you has originally created list1 as:

ar list1 = <List<Object>>[
    <int>[1, 2],
    <int>[3, 4]
  ];

then list2[0] would have succeeded.

In short, using cast/castFrom only works if the collection shallowly contains only elements of the type you cast to.
If you have a List<Object> containing only String objects, then thatList.cast<String> will give you a List<String> view of the original list, with run-time checking that the type is right.

[lrhn]

The cast wrapper class has the type it promises to have. It ensures that all the value you get out of the cast list satisfies the element type too. It does that by throwing at runtime if it can't satisfy that requirement. It never gets into a situation where it evaluates to a value that doesn't have the expression's static type ... because it doesn't evaluate to a value at all in those cases.

That's exactly the same as Object() as int having static type int. It's sound, but only because it always throws. Or int x = throw "nope"; being valid because Never is a subtype of any type.
It's exactly the same as as because the cast wrapper is precisely doing as T on the return values and as S on the inputs.

A completely statically sound type system would not have downcasts at all. You'd only be able to down cast by testing and promoting, giving you an else branch for the non-matching case. An as cast doesn't have an else branch, so it must throw. That makes e as T equivalent to let x = e in (x is T ? x : throw CastError("nah-nah!")) — which you could still write in a completely statically typed language.
Dart, and most other languages, are pragmatic enough to allow throwing casts as an atomic operation, because sometimes that's just what an author needs.

[lrhn]

The biggest difference (to me) is that if (x != null) use(x) has no syntactic hint that something may fail. The code looks exactly like code that is safe (won't throw, different from "sound" which means won't invalidate the static type system).

For list.cast<...>, the call to cast is a warning that something unsafe is happening, just as if you see an x! or x as Foo, late x; ... use(x) or dynamic x; ... x.foo();.
All of these are unsafe, they may throw errors at run-time. The unsafety are syntactically visible somewhere in the code, either at the declaration, the creation or the actual cast.

I'm being inconsistent, and I know it. There is a syntactic hint that use(x) is unsafe because x is not declared as a local variable. It's just further away. The first thing you see when checking for context of use(x) is if (x != null) which looks fine. Having to check a third place to see why it isn't is a step too far. I'd prefer you to never need to look more than one context level up to see whether something can throw at run-time. Which late dynamic x; obviously doesn't satisfy either.
So, it's a difference in degree, not in kind. And I'm worried about it, not because we don't already have other things that are just as bad, but because we do. We are generally trying to move away from implicitly throwing at run-time, so adding more is a hard sell. (And, for the record, I don't think any of the other solutions to field promotion solve the problem in a more user-friendly way, it just safer ways).

I'll probably never use .cast in practice myself. It solved a problem in the Dart 1 to Dart 2 migration, and it has its situations where it's the simplest solution to going from "'Listcontaining only strings" toList, but it's often just a patch on top of an underlying type issue (why was that not a Listto begin with?). I prefer to design code to not need.cast(), late, dynamic, asor!` if possible, and only use them when I actually need the dynamic flexibility to bypass the type system.

[Levi-Lesches]

@ChristianKleineidam, would List<List<List>>.from(asset) not work? This works for me in Dartpad:

List<List<List>> parse(dynamic json) =>
  List<List<List>>.from(json);

void main() {
  List<List<List>> success = parse([[[]]]);
  List<List<List>> fail = parse([[]]);  // Error
}

Of course, if you don't actually provide a List<List<List>>, you get an error, but such is the nature of parsing.

[lrhn]

@Levi-Lesches
Your example doesn't match JSON decoded lists, which is where this problem usually occurs.

The expression [[[]]] is a List<List<List<dynamic>>> containing a List<List<dynamic>> containing a List<dynamic>.

The result of jsonDecode("[[[]]]") is a List<dynamic> containing a List<dynamic> containing a List<dynamic>. That one won't be converted to a List<List<List<dynamic>>> by your parse function.

[Levi-Lesches]

Got it, I added in the JSON and everything broke. I tried going with recursion for deep casting, but that got too messy and I don't think it will work anyway. If there was a way to use Type variables as expressions and manipulate them, then pass them through List.from, it could work, but that seems a long way off

[vishnuagbly]

is there any progress on this issue?

[lrhn]

There is currently no proposed features to make it easier to change a List<List<List<dynamic>>> to a List<List<List<SomethingElse>>> .

[Levi-Lesches]

...did I figure this out? Someone please point out the very obvious bug I must be missing:

// Imagine this is the new List<T>.from(List)
List<T> cast<T>(List list) {
  final List<T> result = [];
  for (final element in list) {
    // If T is a List, we want to cast every element of that list first
    // Let E be the type argument of T, such that T = List<E>
    // Somehow, Dart is smart enough to use `cast` here as `cast<E>(element)`
    T castedElement = T is List ? cast(element) : element;
    result.add(castedElement);
  }
  return result;
}

List<dynamic> getNestedInts() => [[1, 2, 3], [4, 5, 6]];
List<dynamic> getNestedStrings() => [[["this", "is"], ["a", "lot"]], [["of", "really"], ["long", "strings"]]];

void main() {
  List<dynamic> list1 = getNestedInts();
  List<List<int>> nestedInts = cast<List<int>>(list1);
  print(nestedInts.runtimeType);  // JSArray<List<int>>
  
  List<dynamic> list2 = getNestedStrings();
  List<List<List<String>>> nestedStrings = cast<List<List<String>>>(list2);
  print(nestedStrings.runtimeType);  // JSArray<List<List<String>>>
}

[Levi-Lesches]

Okay, I found it. You can't actually do T is List, so the condition was always silently failing. If I try to replace it with element is List, then I get an error that Dart can't figure out the E in the recursive cast call and fails because without a type argument, the cast call returns List<dynamic>, which isn't a T. So the whole thing actually is exactly equivalent to List.from.

So the new code is:

void main() {
  List<dynamic> list1 = getNestedInts();
  print(list1.runtimeType);  // JSArray<dynamic>
  List<List<int>> nestedInts = List.from(list1);
  print(nestedInts.runtimeType);  // JSArray<List<int>>
  print(nestedInts.first);  // [1, 2, 3]
  
  List<dynamic> list2 = getNestedStrings();
  print(list2.runtimeType);  // JSArray<dynamic>
  List<List<List<String>>> nestedStrings = List.from(list2);
  print(nestedStrings.runtimeType);  // JSArray<List<List<String>>>
  print(nestedStrings.first);  // [[this, is], [a, lot]]
}

While this won't, say, convert List<List<Object>> to List<List<int>> (as pointed out above), it can convert List<dynamic> to just about anything, which is still very useful.

If we really wanted to have more control, Dart would need to let us inspect Type Arguments, specifically to extract E from the example in my original comment. It's like regular recursion, when you process a variable and then pass it along, except that we can't touch type args in any meaningful way, so we can't have arbitrary-depth recursion on them.

[vishnuagbly]

@Levi-Lesches yeah but it only works if in List<dynamic> where every element is of statically the same type like in below ex it will throw error as all elements are not statically the same type :-

void main() {
    List<dynamic> list1 = [<dynamic>[2, 4], [1, 4]];
    print(list1.runtimeType); 
    List<List<int>> nestedInts = List.from(list1);  //This will throw an error
    print(nestedInts.runtimeType);
    print(nestedInts.first);
}

Also, just in case if any one find this info useful 🤷‍♂️:-
In dart typing List value = [[1, 4], [3, 1]]; is same as typing List<dynamic> value = <dynamic>[<int>[1, 3], <int>[4, 1]];,

[Levi-Lesches]

Right, thanks, I knew I was missing something obvious. So in that case, yes, I'd refer to my updated findings that we would need to have a way to inspect a type argument and extract its generics. We wouldn't have to rely on compile-time safety, since we all seem to be on the same page that dynamic runtime checks are okay.

[ChristianKleineidam]

I spent another hour with a bug that came down to the same issue. How about introducing asDeep as a new way to do the cast that actually transforms the types as desired?

Then the error message could highlight that the user might want to use asDeep.

(for context, I'm using SembastDb as my database and it returns lists that I saved in it in a way that doesn't have deep type information).

[lrhn]

An asDeep<List<List<Map<String, List<String>>>>>() on List<dynamic> would ... probably not work.
Dart doesn't provide a way to destructure types, so the asDeep function can't get hold of the List<Map<String, List<String>>> it needs to call recursively.

Alternatively, if the input is a JSON structure (so the possible types are known), it's probably possible to find the most precise types for the structure by traversing and rebuilding on the way back.

[ChristianKleineidam]

I'm a bit unsure what you mean with JSON structure here. In my use-case, I don't have access to a JSON but get an object from sembast which inturn internally has something like a JSON sturcture. Do you think that sembast (https://pub.dev/packages/sembast) is doing something wrong here and should give me other objects?