Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some objects with transform render but do not paint #19794

Closed
VerTiGoEtrex opened this issue Jul 26, 2018 · 5 comments
Closed

Some objects with transform render but do not paint #19794

VerTiGoEtrex opened this issue Jul 26, 2018 · 5 comments

Comments

@VerTiGoEtrex
Copy link

@VerTiGoEtrex VerTiGoEtrex commented Jul 26, 2018

I built a simple 3d environment using the Flow widget along with some matrix transformations -- these transformation are all created using built-in dart functions. Nothing is created by hand.

img_3405477f3290-1

The wheel of colors should continue all the way around 360 degrees, but the cards (really containers) that are nearly perpendicular to the screen simply do not render at all. I thought this may be an issue with my matrices, but actually if use the flutter inspector to highlight the render objects, they do indeed exist and appear to be the correct.

wheel_issue

From this, I conclude that the matrices are correct, and the widgets generate valid render objects that position themselves correctly in the "render tree" and on the screen (not sure of the nomenclature to use for this), however, something prevents them from painting.

The repro attached includes a set of functions to debug the transformation of each child's coordinates as each matrix is applied to prove that the coordinates are "sane" and should be drawn on the screen.

Steps to Reproduce

I'm running on iPhone X, but I believe this should be an issue with any platform. This is a minimal example I built to demonstrate this behavior.

Please feel free to tweak the cam*,fovYRadians, and the numCards value. I noticed with greater FoV values, this issue becomes worse.

import 'dart:collection';
import 'dart:math';

import 'package:flutter/material.dart';
import 'package:vector_math/vector_math_64.dart' hide Colors;

void main() => runApp(new MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final numCards = 200;
    final debugCards = List.generate(
        numCards,
        (idx) => Container(
              color: HSVColor
                  .fromAHSV(1.0, 360 * idx / numCards, 1.0, 1.0)
                  .toColor(),
            ));

    return MaterialApp(
      home: Center(
        child: Container(
          width: 300.0,
          height: 300.0,
          child: new Flow(
            delegate: CardStackFlow(),
            children: debugCards,
          ),
        ),
      ),
    );
  }
}

class CardStackFlow extends FlowDelegate {
  double camX;
  double camY;
  double camZ;

  CardStackFlow({
    this.camX = 15.0,
    this.camY = 7.0,
    this.camZ = 10.0,
  }) : super();

  @override
  void paintChildren(FlowPaintingContext context) {
    print("paintChildren");
    print("${context.size}");
    final int numSlots = context.childCount;

    // Setup the view matrix for a camera looking at the origin of the world
    final cameraPosition = Vector3(camX, camY, camZ);
    final lookAt = Vector3.zero(); // World origin
    final upDir = Vector3(0.0, 1.0, 0.0);
    final viewMatrix = makeViewMatrix(cameraPosition, lookAt, upDir);

    // Setup the projection matrix
    final fovYRadians =
        0.39867460996464815; // Hardcoded a value that exposes the bug
    final aspectRatio = 1.0;
    final zNearPlane = cameraPosition.distanceTo(lookAt) - 0.1;
    final zFarPlane = cameraPosition.distanceTo(lookAt) * 2;
    print(
        "Cam Params: fov=$fovYRadians, aspectRatio=$aspectRatio, nearPlane=$zNearPlane, farPlane=$zFarPlane");
    final projectionMatrix =
        makePerspectiveMatrix(fovYRadians, aspectRatio, zNearPlane, zFarPlane);

    // Build matrices to get us into an OpenGL-like coordinate space
    // bottom left corner of the screen is at (-1, -1), top right is at (1, 1)
    final toOpenGlCoords = Matrix4.compose(
        Vector3(-1.0, 1.0, 0.0),
        Quaternion.axisAngle(Vector3(1.0, 0.0, 0.0), pi),
        Vector3(2 / context.size.width, 2 / context.size.height, 1.0));
    final toFlutterCoords = Matrix4.tryInvert(toOpenGlCoords);
    print("toOpenGlCoords=\n$toOpenGlCoords");
    print("toFlutterCoords=\n$toFlutterCoords");
    assert(toFlutterCoords != null);
    assert(matrixEqualEpsilon(
        toFlutterCoords * toOpenGlCoords, Matrix4.identity(), pow(0.1, 15)));

    // Paint all of the children
    print("");
    for (int i = context.childCount - 1; i >= 0; --i) {
      print("Painting child $i");

      final pivotPoint = Matrix4.compose(
          Vector3(0.0, -2.0, 0.0), Quaternion.identity(), Vector3.all(1.0));

      final pivotTransform = Matrix4.compose(
          Vector3(0.0, 2.0, 0.0),
          Quaternion.axisAngle(
              Vector3(1.0, 0.0, 0.0), pi * 2 * (i / (numSlots - 1))),
          Vector3.all(1.0));

      final modelMatrix = pivotTransform * pivotPoint;

      final mvp = projectionMatrix * viewMatrix * modelMatrix;

      final flutterCompatTransform = toFlutterCoords * mvp * toOpenGlCoords;

      // Debug points
      var transformPipeline = <TransformDebug>[
        TransformDebug("toOpenGlCoords", toOpenGlCoords),
        TransformDebug("pivotPoint", pivotPoint),
        TransformDebug("pivotTransform", pivotTransform),
        TransformDebug("viewMatrix", viewMatrix),
        TransformDebug("projectionMatrix", projectionMatrix),
        TransformDebug("toFlutterCoords", toFlutterCoords)
      ];
      if (i == 9) {
        print("sizeOfWindow = ${context.size}");
        final size = context.getChildSize(i);
        debugPoint(Vector4(0.0, 0.0, 0.0, 1.0), transformPipeline);
        debugPoint(Vector4(size.width, 0.0, 0.0, 1.0), transformPipeline);
        debugPoint(Vector4(0.0, size.height, 0.0, 1.0), transformPipeline);
        debugPoint(
            Vector4(size.width, size.height, 0.0, 1.0), transformPipeline);
      }

      // Paint the child
      context.paintChild(i, transform: flutterCompatTransform);
      print("");
    }
  }

  void debugPoint(Vector4 point, List<TransformDebug> transformations) {
    print("Debugging point $point");
    transformations.forEach((xfrmDebug) {
      point = xfrmDebug.xfrm * point;
      point.scale(1 / point.w);
      print("After applying ${xfrmDebug.name}\n$point");
    });
  }

  @override
  bool shouldRepaint(FlowDelegate oldDelegate) {
    return this == oldDelegate;
  }
}

class TransformDebug {
  final String name;
  final Matrix4 xfrm;

  TransformDebug(
    this.name,
    this.xfrm,
  );
}

bool matrixEqualEpsilon(Matrix4 left, Matrix4 right, double epsilon) {
  for (int i = 0; i < left.storage.length; ++i) {
    if ((left.storage[i] - right.storage[i]).abs() > epsilon) {
      return false;
    }
  }
  return true;
}

Logs

I suspect logs are not really valuable here, but I've attached a shortened version anyway. Happy to provide full logs if needed.

...
[ +127 ms] ------ Debug phase ------
[        ] Starting debug of <redacted (D221AP, D221AP, uknownos, unkarch) a.k.a. 'Vert iPhone' connected through USB...
[ +630 ms] [  0%] Looking up developer disk image
[  +23 ms] [ 95%] Developer disk image mounted successfully
[ +735 ms] [100%] Connecting to remote debug server
[        ] -------------------------
[  +29 ms] (lldb) command source -s 0 '/tmp/E7650D69-6F0C-4D03-851D-560268A44D11/fruitstrap-lldb-prep-cmds-<redacted>'
[        ] Executing commands in '/tmp/E7650D69-6F0C-4D03-851D-560268A44D11/fruitstrap-lldb-prep-cmds-<redacted>'.
[        ] (lldb)     platform select remote-ios --sysroot '/Users/ncrocker/Library/Developer/Xcode/iOS DeviceSupport/11.3.1 (15E302)/Symbols'
[        ]   Platform: remote-ios
[        ]  Connected: no
[        ]   SDK Path: "/Users/ncrocker/Library/Developer/Xcode/iOS DeviceSupport/11.3.1 (15E302)/Symbols"
[        ] (lldb)     target create "/Users/ncrocker/dev/personal/broken_flow_render/build/ios/iphoneos/Runner.app"
[+3143 ms] Current executable set to '/Users/ncrocker/dev/personal/broken_flow_render/build/ios/iphoneos/Runner.app' (arm64).
[        ] (lldb)     script fruitstrap_device_app="/private/var/containers/Bundle/Application/5084B3EA-88B9-4733-A6A2-7F8FCDD14CC8/Runner.app"
[        ] (lldb)     script fruitstrap_connect_url="connect://127.0.0.1:50610"
[        ] (lldb)     target modules search-paths add /usr "/Users/ncrocker/Library/Developer/Xcode/iOS DeviceSupport/11.3.1 (15E302)/Symbols/usr" /System "/Users/ncrocker/Library/Developer/Xcode/iOS DeviceSupport/11.3.1 (15E302)/Symbols/System" "/private/var/containers/Bundle/Application/5084B3EA-88B9-4733-A6A2-7F8FCDD14CC8" "/Users/ncrocker/dev/personal/broken_flow_render/build/ios/iphoneos" "/var/containers/Bundle/Application/5084B3EA-88B9-4733-A6A2-7F8FCDD14CC8" "/Users/ncrocker/dev/personal/broken_flow_render/build/ios/iphoneos" /Developer "/Users/ncrocker/Library/Developer/Xcode/iOS DeviceSupport/11.3.1 (15E302)/Symbols/Developer"
[  +53 ms] (lldb)     command script import "/tmp/E7650D69-6F0C-4D03-851D-560268A44D11/fruitstrap_<redacted>.py"
[   +2 ms] (lldb)     command script add -f fruitstrap_<redacted>.connect_command connect
[        ] (lldb)     command script add -s asynchronous -f fruitstrap_<redacted>.run_command run
[        ] (lldb)     command script add -s asynchronous -f fruitstrap_<redacted>.autoexit_command autoexit
[        ] (lldb)     command script add -s asynchronous -f fruitstrap_<redacted>.safequit_command safequit
[        ] (lldb)     connect
[  +28 ms] (lldb)     run
[ +382 ms] success
[        ] (lldb)     safequit
[ +114 ms] Process 888 detached
[  +27 ms] Application launched on the device. Waiting for observatory port.
[+1700 ms] Observatory URL on device: http://127.0.0.1:63676/
[   +6 ms] /usr/local/bin/iproxy 8102 63676 <redacted>
[   +4 ms] Forwarded port ForwardedPort HOST:8102 to DEVICE:63676
[        ] Forwarded host port 8102 to device port 63676 for Observatory
[   +6 ms] Connecting to service protocol: http://127.0.0.1:8102/
[ +136 ms] Successfully connected to service protocol: http://127.0.0.1:8102/
[   +3 ms] getVM: {}
[  +11 ms] getIsolate: {isolateId: isolates/353767474}
[   +2 ms] _flutter.listViews: {isolateId: isolates/353767474}
[  +58 ms] DevFS: Creating new filesystem on the device (null)
[        ] _createDevFS: {fsName: broken_flow_render}
[  +19 ms] DevFS: Created new filesystem on the device (file:///private/var/mobile/Containers/Data/Application/675A98AE-3304-4A70-8CA2-BB112FEC84B3/tmp/broken_flow_renderPDtHh7/broken_flow_render/)
[   +1 ms] Updating assets
[ +209 ms] flutter: paintChildren
[   +3 ms] flutter: Size(300.0, 300.0)
[        ] flutter: Cam Params: fov=0.39867460996464815, aspectRatio=1.0, nearPlane=19.239079605813714, farPlane=38.67815921162743
[   +8 ms] flutter: toOpenGlCoords=
[        ] [0] 0.006666666666666667,0.0,0.0,-1.0
[        ] [1] 0.0,-0.006666666666666667,-1.2246467991473532e-16,1.0
[        ] [2] 0.0,8.164311994315689e-19,-1.0,0.0
[        ] [3] 0.0,0.0,0.0,1.0
[        ] flutter: toFlutterCoords=
[        ] [0] 149.99999999999997,0.0,0.0,149.99999999999997
[        ] [1] 0.0,-149.99999999999997,1.8369701987210297e-14,149.99999999999997
[        ] [2] 0.0,-1.2246467991473532e-16,-1.0,1.2246467991473532e-16
[        ] [3] 0.0,0.0,0.0,1.0
[        ] flutter:
[        ] flutter: Painting child 199
[   +6 ms] flutter:
[        ] flutter: Painting child 198
[        ] flutter:
[        ] flutter: Painting child 197
[        ] flutter:
[        ] flutter: Painting child 196
[        ] flutter:
[   +1 ms] flutter: Painting child 195
...
Analyzing broken_flow_render...

   info • Unused import: 'dart:collection' • lib/main.dart:1:8

1 issue found. (ran in 3.0s)
[✓] Flutter (Channel beta, v0.5.1, on Mac OS X 10.13.4 17E199, locale en-US)
    • Flutter version 0.5.1 at /Users/ncrocker/dev/flutter
    • Framework revision c7ea3ca377 (8 weeks ago), 2018-05-29 21:07:33 +0200
    • Engine revision 1ed25ca7b7
    • Dart version 2.0.0-dev.58.0.flutter-f981f09760

[!] Android toolchain - develop for Android devices (Android SDK 27.0.3)
    • Android SDK at /Users/ncrocker/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-27, build-tools 27.0.3
    • Java binary at: /Library/Java/JavaVirtualMachines/jdk-10.0.1.jdk/Contents/Home/bin/java
    • Java version Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
    ✗ Android license status unknown.

[✓] iOS toolchain - develop for iOS devices (Xcode 9.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 9.4.1, Build version 9F2000
    • ios-deploy 1.9.2
    • CocoaPods version 1.5.3

[✗] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[✓] IntelliJ IDEA Community Edition (version 2018.1.4)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 26.0.2
    • Dart plugin version 181.4892.1

[!] VS Code (version 1.25.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected devices (1 available)
    • Vert iPhone • <REDACTED> • ios • iOS 11.3.1

! Doctor found issues in 3 categories.
@jonahwilliams

This comment has been minimized.

Copy link
Contributor

@jonahwilliams jonahwilliams commented Jul 26, 2018

@VerTiGoEtrex on master branch I get a slightly different result - is this correct? Might have been fixed by #19062, where we sometimes incorrectly culled layers with perspective transforms applied to them.

@VerTiGoEtrex

This comment has been minimized.

Copy link
Author

@VerTiGoEtrex VerTiGoEtrex commented Jul 26, 2018

@jonahwilliams, you're right, the master channel doesn't have this bug.

I briefly read over the description of the issue you linked. For my own benefit, can you confirm my understanding? Basically, if a transform matrix transforms a child bounding box to completely outside of its parent bounds, that rect is "culled" and therefore the view is not painted (or participates in hit testing? Not sure of the extent of "culling" in flutter engine).

Feel free to close the issue, and thanks!

@jonahwilliams

This comment has been minimized.

Copy link
Contributor

@jonahwilliams jonahwilliams commented Jul 26, 2018

The culling on the engine side is quite tame and mostly meant to save some work on layers that "obviously" won't get painted. For example, if you have a clipping layer and then some child of that is painting a picture entirely outside of the clip, you can just drop it.

Now if there is a transform between the clip and the child, you need to invert it to understand where the child will get painted. But there was a bug where certain perspective transforms make it appear that children are being painted far offscreen. I still haven't tracked down the root cause, whether we have a logical issue or there is a numerical bug in the Skia api we are using. Since perspective transforms are fairly uncommon, just assuming that the child won't be clipped for these cases doesn't really cost us anything.

Hit testing on the other hand is entirely on the framework side. Any part of a widget that paints outside it's bounds won't participate in hit testing. The bounds define the hit testable area, not the paint. There is some weirdness also with hit testing on transformed objects - see #18408.

@VerTiGoEtrex

This comment has been minimized.

Copy link
Author

@VerTiGoEtrex VerTiGoEtrex commented Jul 26, 2018

@jonahwilliams Thanks. Is the hit testing by design? It seems hit testing should be determined by paint. I've run into this exact behavior you describe and was surprised by it. I think the Transform and Flow widgets but have this "issue" since they both repaint rather than relayout.

@VerTiGoEtrex

This comment has been minimized.

Copy link
Author

@VerTiGoEtrex VerTiGoEtrex commented Jul 26, 2018

If you have any type of view interaction, Flow seems generally useless. Even with a regular Transform widget, it seems any transform other than translation do not transform hit tests. Is this by design as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.