[wayland] streambuilder (or rather flutter pipeline) leaks memory

[masteranza]

We've been trying to use streambuilder to frequently update our ui according to updates incoming through the network, but found that the app memory usage (e.g. linux RSS) grows continuously without bound only when built using embedded-linux.
Since then we've been running various test and found out that even the simplest streambuilder of the kind below leaks:

final stream = Stream.periodic(const Duration(milliseconds: 20), (n) { return Random(n).nextDouble();});
@override
Widget build(BuildContext context) {
    return Container(child: Center(child: SizedBox(width: 300, height: 100, 
          child: StreamBuilder<double>(stream: stream, initialData: 0.0,
          builder: (BuildContext context, AsyncSnapshot<double> snapshot) 
          {
            return Container(key: UniqueKey(), width: 300.0*snapshot.data!, height: 100, color: Colors.red);
          })
      ))
    );
}

the UniqueKey() was added with the hope to help the GC to collect the container, but the leaks occurs even without it.
We've adjusted Duration in Stream.periodic and verified that the amount of data leaked stays roughly constant, independent of the period roughly 0.335kb with each stream event (tested with: 20ms, 200ms, 2000ms).

This comes as a surprise, because according to the flutter docs, StreamBuilder should trigger rebuild according to "flutter pipeline", i.e. very fast stream events are expected to be skipped implicitly by the engine (pipeline), quote:

The builder is called at the discretion of the Flutter pipeline, and will thus receive a timing-dependent sub-sequence of the snapshots that represent the interaction with the stream.

Is this issue known? Can it be fixed? We've seen a similar issue regarding leaks during gif animations, but we're not sure whether those two are related.

More info:
The build is done in docker using flutter-elinux build elinux --release and run with params: ./app -n -b . -f on NXP SOC iMX8M mini (quad core ARM Cortex-A53+ single Cortex-M4.

[HidenoriMatsubayashi]

Is it possible to share a test program (flutter project) to reproduce in my environment? Also, Could you please confirm whether the memory leak happen when you run your app on Flutter desktop for Linux/Windows/Mac? I would like to break down the issue into parts (Flutter engine, Flutter framework, this embedder, etc).

[masteranza]

The full project (not the simplified-example presented above) runs without a leak on linux and macos but leaks when built with elinux and run on linux.

The below is basically a standalone that you could run to observe the leak:

import 'dart:math';
import 'dart:ui' as ui;
import 'package:flutter/material.dart';
class Home extends StatelessWidget {
  final stream = Stream.periodic(const Duration(milliseconds: 10), (n) 
  {
    return Random(n).nextDouble();
  });
  
  Home({Key? key}) : super(key: key);
  @override
  Widget build(BuildContext context) {
    return  MediaQuery(
        data: MediaQueryData.fromWindow(ui.window),
        child: Directionality(
            textDirection: TextDirection.rtl,
            child: Container(
      child: 
      Center(
        child: SizedBox(width: 300, height: 100, 
          child: StreamBuilder<double>(stream: stream, initialData: 0.0,
          builder: (BuildContext context, AsyncSnapshot<double> snapshot) 
          {
            return Container(key: UniqueKey(), 
              width: 300.0*snapshot.data!, height: 100, color: Colors.red);
          })
      )))
    ));
  }
}

we've used MediaQuery and Directionality to eliminate the possibility that the leak is due Material Scaffold.

[HidenoriMatsubayashi]

Thanks. What kind of wayland compositors do you use?

[HidenoriMatsubayashi]

Sorry, I haven't been able to find clear evidence of a memory leak. It looks like the memory is freed normally.

[masteranza]

@HidenoriMatsubayashi we use weston compositor. Interesting that you're not seeing it, we're seeing it clearly at around 10m run on 20ms version. Below RSS/time taken from influx from various test containers (colors)

[HidenoriMatsubayashi]

I checked it on Weston and GNOME (Wayland), but the issue seems to happen on Weston only. I'll investigate more.

[masteranza]

@HidenoriMatsubayashi thank you, please let us know what you'll find out. We've been googling weston issues a little bit in meantime and there seem to be a few issues related to memory leaks:
https://community.nxp.com/t5/i-MX-Processors/Weston-dma-buffer-leaks-on-imx8mq-imx8mm/m-p/947357
https://community.nxp.com/t5/i-MX-Processors/i-MX8M-GPU-seems-still-OOM-after-apply-patch/td-p/1059364

[masteranza]

@HidenoriMatsubayashi any perspectives for solving this issue?

[HidenoriMatsubayashi]

Sorry, I still don't have enough time to investigate it, but we can debug with memory leak detector tools such as Valgrind.

[HidenoriMatsubayashi]

I tested it, but I could not find any memory leaks on flutter-embedded-linux side. Also, I tested the app on Sway, but there was any memory leaks. I guess this memory leak happens on Weston only, and the root cause is in Weston.

[masteranza]

Thank you @HidenoriMatsubayashi, this is sad news. May I ask which versions of weston did you test?
Our weston is installed outside the container, while the flutter app in the container leaks. So I suspect, that perhaps this might be a lack of some kind of clearance from weston for flutter to dispose something... What do you suspect might be going on?

[HidenoriMatsubayashi]

Hmm, I'm not sure. Can you try address sanitizer in your env? I used weston version 9.0.0 on Ubuntu 22.

[masteranza]

Never used that before, but I can try.
Did it give any worthwhile info when applied to the example I've provided?

[masteranza]

Would it be possible for you @HidenoriMatsubayashi to test newer version of weston? I know I'm asking for a considerable favor, but this is a rather large time investment on my side and I thought that perhaps it could be simple for you.

Also here's my weston log - perhaps it helps:

Date: 2022-08-06 UTC
[12:50:21.842] weston 9.0.0
               https://wayland.freedesktop.org
               Bug reports to: https://gitlab.freedesktop.org/wayland/weston/issues/
               Build: 9.0.0
[12:50:21.842] Command line: /usr/bin/weston --log=/run/user/1001/weston.log --modules=systemd-notify.so
[12:50:21.842] OS: Linux, 5.10.72-lts-5.10.y+gd7915b943352, #1 SMP PREEMPT Fri Apr 8 09:13:52 UTC 2022, aarch64
[12:50:21.844] Using config file '/etc/xdg/weston/weston.ini'
[12:50:21.844] Output repaint window is 100 ms maximum.
[12:50:21.845] Loading module '/usr/lib/libweston-9/drm-backend.so'
[12:50:21.857] initializing drm backend
[12:50:21.862] logind: session control granted
[12:50:21.872] using /dev/dri/card1
[12:50:21.872] DRM: supports atomic modesetting
[12:50:21.872] DRM: does not support GBM modifiers
[12:50:21.872] DRM: supports picture aspect ratio
[12:50:21.873] Loading module '/usr/lib/libweston-9/gl-renderer.so'
[12:50:21.891] EGL client extensions: EGL_EXT_client_extensions
               EGL_EXT_platform_base EGL_KHR_platform_wayland
               EGL_EXT_platform_wayland EGL_KHR_platform_gbm
[12:50:21.896] EGL version: 1.5
[12:50:21.896] EGL vendor: Vivante Corporation
[12:50:21.896] EGL client APIs: OpenGL_ES
[12:50:21.896] EGL extensions: EGL_KHR_fence_sync EGL_KHR_reusable_sync
               EGL_KHR_wait_sync EGL_KHR_image EGL_KHR_image_base
               EGL_KHR_image_pixmap EGL_KHR_gl_texture_2D_image
               EGL_KHR_gl_texture_cubemap_image EGL_KHR_gl_renderbuffer_image
               EGL_EXT_image_dma_buf_import
               EGL_EXT_image_dma_buf_import_modifiers EGL_KHR_lock_surface
               EGL_KHR_create_context EGL_KHR_no_config_context
               EGL_KHR_surfaceless_context EGL_KHR_get_all_proc_addresses
               EGL_EXT_buffer_age EGL_ANDROID_native_fence_sync
               EGL_WL_bind_wayland_display
               EGL_WL_create_wayland_buffer_from_image EGL_KHR_partial_update
               EGL_EXT_swap_buffers_with_damage
               EGL_KHR_swap_buffers_with_damage EGL_EXT_pixel_format_float
[12:50:21.896] EGL_KHR_surfaceless_context available
[12:50:21.913] GL version: OpenGL ES 2.0 V6.4.3.p2.336687
[12:50:21.913] GLSL version: OpenGL ES GLSL ES 1.0.0
[12:50:21.913] GL vendor: Vivante Corporation
[12:50:21.913] GL renderer: Vivante GC7000NanoUltra
[12:50:21.913] GL extensions: GL_OES_vertex_half_float
               GL_OES_element_index_uint GL_OES_mapbuffer
               GL_OES_vertex_array_object GL_OES_compressed_ETC1_RGB8_texture
               GL_OES_compressed_paletted_texture GL_OES_texture_npot
               GL_OES_rgb8_rgba8 GL_OES_depth_texture
               GL_OES_depth_texture_cube_map GL_OES_depth24 GL_OES_depth32
               GL_OES_packed_depth_stencil GL_OES_fbo_render_mipmap
               GL_OES_get_program_binary GL_OES_fragment_precision_high
               GL_OES_standard_derivatives GL_OES_EGL_image
               GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3
               GL_OES_EGL_sync GL_OES_required_internalformat
               GL_OES_surfaceless_context GL_OES_texture_border_clamp
               GL_EXT_texture_type_2_10_10_10_REV
               GL_EXT_texture_compression_dxt1 GL_EXT_texture_format_BGRA8888
               GL_EXT_texture_compression_s3tc GL_EXT_read_format_bgra
               GL_EXT_multi_draw_arrays GL_EXT_frag_depth
               GL_EXT_discard_framebuffer GL_EXT_blend_minmax
               GL_EXT_multisampled_render_to_texture GL_EXT_robustness
               GL_EXT_texture_sRGB_decode GL_EXT_texture_border_clamp
               GL_EXT_texture_rg GL_EXT_unpack_subimage GL_VIV_direct_texture
[12:50:21.913] GL ES 2 renderer features:
               read-back format: BGRA
               wl_shm sub-image to texture: yes
               EGL Wayland extension: yes
[12:50:21.931] event0  - gpio_buttons0: not tagged as supported input device
[12:50:21.931] event0  - not using input device '/dev/input/event0'
[12:50:22.045] event1  - Zytronic Displays Limited Zytronic Touchscreen Controller Touchscreen: is tagged by udev as: Tablet
[12:50:22.046] event1  - Zytronic Displays Limited Zytronic Touchscreen Controller Touchscreen: libinput bug: missing tablet capabilities: btn-stylus resolution. Ignoring this device.
[12:50:22.046] event1  - Zytronic Displays Limited Zytronic Touchscreen Controller Touchscreen: device is a tablet
[12:50:22.047] event1  - failed to create input device '/dev/input/event1'
[12:50:22.055] event2  - Zytronic Displays Limited Zytronic Touchscreen Controller Mouse: is tagged by udev as: Mouse
[12:50:22.055] event2  - Zytronic Displays Limited Zytronic Touchscreen Controller Mouse: device is a pointer
[12:50:22.056] libinput: configuring device "Zytronic Displays Limited Zytronic Touchscreen Controller Mouse".
[12:50:22.056] input device event2 has no enabled output associated (none named), skipping calibration for now.
[12:50:22.127] DRM: head 'HDMI-A-1' found, connector 35 is connected, EDID make 'TVL', model 'MONITOR_HDMI', serial 'MONITOR_HDMI'
[12:50:22.127] Registered plugin API 'weston_drm_output_api_v1' of size 24
[12:50:22.130] Chosen EGL config details: id:  21 rgba: 8 8 8 0 buf: 24 dep:  0 stcl: 0 int: 1-60 type: win|pix|pbf|swap_preserved vis_id: XRGB8888 (0x34325258)
[12:50:22.130] Output HDMI-A-1 (crtc 33) video modes:
               1024x768@60.0, preferred, current, 65.0 MHz
               1920x1080@60.0, 148.5 MHz
               1920x1080@60.0 16:9, 148.5 MHz
               1920x1080@50.0 16:9, 148.5 MHz
               1280x720@60.0, 74.2 MHz
               1280x720@60.0 16:9, 74.2 MHz
               1280x720@50.0 16:9, 74.2 MHz
               832x624@74.6, 57.3 MHz
               800x600@75.0, 49.5 MHz
               720x576@50.0 16:9, 27.0 MHz
               720x480@59.9 16:9, 27.0 MHz
               720x480@51.6, 27.0 MHz
               640x480@66.7, 30.2 MHz
               720x400@70.1, 28.3 MHz
[12:50:22.131] associating input device event2 with output HDMI-A-1 (none by udev)
[12:50:22.132] Output 'HDMI-A-1' enabled with head(s) HDMI-A-1
[12:50:22.132] Compositor capabilities:
               arbitrary surface rotation: yes
               screen capture uses y-flip: yes
               presentation clock: CLOCK_MONOTONIC, id 1
               presentation clock resolution: 0.000000001 s
[12:50:22.133] Loading module '/usr/lib/weston/desktop-shell.so'
[12:50:22.136] launching '/usr/libexec/weston-keyboard'
[12:50:22.141] Loading module '/usr/lib/weston/systemd-notify.so'
[12:50:22.142] info: add 1 socket(s) provided by systemd
[12:50:22.143] Loading module '/usr/lib/libweston-9/xwayland.so'
[12:50:22.193] Registered plugin API 'weston_xwayland_v1' of size 32
[12:50:22.193] Registered plugin API 'weston_xwayland_surface_v1' of size 16
[12:50:22.194] xserver listening on display :0
[12:50:22.194] launching '/usr/libexec/weston-desktop-shell'

[HidenoriMatsubayashi]

Would it be possible for you @HidenoriMatsubayashi to test newer version of weston?

Yes, I can. But What does it mean? I just only asked you to try sanitizer in your environment.

[masteranza]

I'm working on it. Isn't it clear however that it's on weston side rather than flutter?

[HidenoriMatsubayashi]

I still don't think this is a problem on the embedder side. It doesn't depend on display backends. I think it is on the dart side (perhaps user program), but I am not familiar with the use of streambuilder.

[masteranza]

Well, in meantime we've been testing some backup options and found out that setState(...) also leaks (I'll ask a college to provide you with an example)

[OneAndZero24]

@HidenoriMatsubayashi Here is an example without use of any Streams. It also leads to memory leak. You can adjust the timing to achieve different rates of leakage. Hope this will help with finding the problem.

import 'package:flutter/material.dart';
import 'dart:async';

class MyWidget extends StatefulWidget {
  const MyWidget({Key? key}) : super(key: key);

  @override
  State<MyWidget> createState() => _MyWidgetState();
}

class _MyWidgetState extends State<MyWidget> {
  int counter = 0;

  void callback(int new_counter)
  {
    setState(() {
      counter = new_counter;
    });
  }

  @override
  void initState() {
    Timer.periodic(const Duration(seconds: 1), (timer){callback(timer.tick);});
    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    return Text(counter.toString());
  }
}

[HidenoriMatsubayashi]

Please see https://stackoverflow.com/questions/66669029/calling-setstate-inside-the-timer-will-cause-memory-leak-in-flutter. I think your code is wrong. You should add cancel method.

[OneAndZero24]

Sorry, I forgot to add it to the snippet. I've tested again with timer cancel in dispose, but it still leaks. As I've thought because this widget is always on the screen so dispose of it's state will never be called. It behaves the same way with or without this cancel.

[masteranza]

@HidenoriMatsubayashi just as @OneAndZero24 said. Did you have a chance to test newer weston/wayland?
This information would be extremely helpful for us, as upgrade of our yocto to newer weston/wayland is time consuming, but knowing that it works would make it worthwhile. Any chance you could find time for doing this for us?

[HidenoriMatsubayashi]

Did you have a chance to test newer weston/wayland?

Sorry, I'm not sure why I should try it. I don't think it's a problem on the embedder side. If it really leaks memory, it's a problem on dart (flutter/flutter) side.

[masteranza]

Sorry, if it wasn’t clear, but I’m just asking whether you could, not that you should. You’ve confirmed previosly that the issue occurs on weston + elinux, but not other display protocols (and we have verified that it doesn’t occur also without elinux) which makes it unlukily to be a bug in flutter itself.

…

-- Best regards, Michal Mandrysz

[HidenoriMatsubayashi]

You’ve confirmed previosly that the issue occurs on weston + elinux

Sorry, it was my misunderstanding. As long as I use your sample code, it seems to happen in all flutter (linux, elinux, windows, etc). And this is not a memory leak, it just looks like it's just using memory all the time. First, please check if the same problem occurs on linux, windows, android instead of elinux. If there is a problem, it is a problem on the dart side.

[masteranza]

@HidenoriMatsubayashi so this was a mistake?

Sorry, I haven't been able to find clear evidence of a memory leak. It looks like the memory is freed normally.

We've been testing all morning and we don't see a leak on both MacOS and Linux when running pure flutter (without eLinux).

[HidenoriMatsubayashi]

Thanks for the tests. But I have no ideas to solve the problem because I cannot find bugs in the embedder side.

[masteranza]

Alright I understand, is there perhaps a way to tune GC to a more aggressive mode?

[HidenoriMatsubayashi]

is there perhaps a way to tune GC to a more aggressive mode?

GC in Dart? Sorry, I don't know.

[masteranza]

I've searched a little bit and it looks https://dart-review.googlesource.com/c/sdk/+/150503 like this is what is missing to allow StreamBuilder to work effectively.

[vm, api] Add Dart_HintFreed to the embedding API.
Allows an embedder (or native extension) to inform the VM it suspects memory has become unreachable.

[masteranza]

While we have been able to make a regular flutter app to "leak" or putting it in other words:
give the GC little time to breathe through constant UI updates, this issue is far less critical as the GC eventually stabilizes.
This is not the case in elinux build, with even the simplest examples using any ui update method, we've tested:

• ValueListenableBuilder
• StreamBuilder
• SetState

In all cases the GC is not able to clean, even after the whole widget together with the stream updating it has been pop'ed of the screen.