[analyzer] Surprising discrepancy when reporting a bound violation

[eernstg]

Consider the following library:

typedef F<X> = X Function(X);

extension<X> on X {
  X m<Y extends F<X>>() => this;
}

void test1() {
  g() sync* {
    yield 1;
    yield null;
  }

  g.m<F<Iterable<int?> Function()>>(); // No error, OK.
  g.m<F<Iterable<int?> Function()>>; // No error, OK.
  g.m<F<Iterable<int> Function()>>(); // Error reported, OK.
  g.m<F<Iterable<int> Function()>>; // No error, BUG.
}

void test2(Iterable<int?> h()) {  
  h.m<F<Iterable<int?> Function()>>(); // No error, OK.
  h.m<F<Iterable<int?> Function()>>; // No error, OK.
  h.m<F<Iterable<int> Function()>>(); // Error, OK.
  h.m<F<Iterable<int> Function()>>; // Error, OK.
}

The analyzer (using DartPad stable, 3.8.1, as well as main, 3.10.0-74.0.dev) reports errors as indicated in the comments. An error message would be expected for g.m<F<Iterable<int> Function()>>; (because the given actual type argument Iterable<int> Function() Function(Iterable<int> Function()) isn't a subtype of the bound Iterable<int?> Function() Function(Iterable<int?> Function()), where the bound is confirmed by the treatment of the previous two lines).

Apparently, the explicit generic function instantiation g.m<...> is treated differently than the invocation g.m<...>(), and perhaps the former isn't subjected to a bounds check at all (or the difference in nullability is ignored).

The most surprising part is that a situation that seems to be perfectly identical with respect to the types, namely h.m<F<Iterable<int> Function()>>; in test2, does incur the error.

[eernstg]

The situation also arises with a slightly simpler example:

typedef F<X> = X Function(X);

extension<X> on X {
  X m<Y extends F<X>>() => this;
}

void test1() {
  g() {
    if (2 > 1) return 1;
    return null;
  }

  g.m<F<int? Function()>>(); // No error, OK.
  g.m<F<int? Function()>>; // No error, OK.
  g.m<F<int Function()>>(); // Error reported, OK.
  g.m<F<int Function()>>; // No error, BUG.
}

void test2(int? h()) {  
  h.m<F<int? Function()>>(); // No error, OK.
  h.m<F<int? Function()>>; // No error, OK.
  h.m<F<int Function()>>(); // Error, OK.
  h.m<F<int Function()>>; // Error, OK.
}

One more reduction seems to support the assumption that there is a connection to the inference of local function return types, because there are no unexpected outcomes with this variant:

typedef F<X> = X Function(X);

extension<X> on X {
  X m<Y extends F<X>>() => this;
}

void test1(bool b) {
  final g = b ? 42 : null;

  g.m<F<int?>>(); // No error, OK.
  g.m<F<int?>>; // No error, OK.
  g.m<F<int>>(); // Error reported, OK.
  g.m<F<int>>; // Error, OK.
}

void test2(int? h) {  
  h.m<F<int?>>(); // No error, OK.
  h.m<F<int?>>; // No error, OK.
  h.m<F<int>>(); // Error, OK.
  h.m<F<int>>; // Error, OK.
}