Request for Access to Plugin Code in Unit Tests for Better Native Code Testing

[stephane-archer]

Description:

Currently, unit tests in Flutter do not provide a straightforward way to access and test native code used within plugins. This limitation forces developers to rely on either:

1. Mocking the native code, which doesn't allow for comprehensive and accurate testing of how the app interacts with actual native functionality.
2. Integration tests, which are often too slow and focused on testing the app and UI, rather than specific, isolated functions that call native code.

As a result, there is a gap in testing where developers cannot properly validate the integration and behavior of native code in isolation, slowing down development and making it harder to catch issues early in the development cycle.

Problem:

• Mocking native code: While mocking works in some cases, it doesn't properly simulate the native code’s actual behavior, leading to false positives/negatives in tests.
• Slow integration tests: Integration tests, while useful, are generally slow, and focus more on end-to-end behavior (app + UI) rather than unit-level functionality. This makes testing a single function that relies on native code cumbersome and inefficient.

Proposed Solution:

I would like to request a feature where Flutter allows access to plugin code in unit tests, so we can test the native functionality directly in a more isolated and efficient manner.

• This would allow us to write unit tests that can call native functions directly without resorting to mocking.
• It would speed up the testing process by ensuring that we can test native interactions in a controlled environment, without having to rely on slow integration tests that test multiple layers of the application.

Benefits:

• Faster unit tests: Unit tests will run faster since they won't require full integration setups or UI interaction.
• More accurate testing: Testing native code directly will allow us to ensure that our app's native functionality behaves as expected, without the abstraction or inaccuracies introduced by mocking.
• Improved developer experience: The ability to write more fine-grained tests that focus on specific functionality rather than having to test everything end-to-end.

I believe this feature would greatly improve the testing experience in Flutter and enable more reliable testing practices for developers working with native plugins.

[lrhn]

I'm not sure what Dart feature would help with this (or what a Flutter plugin is at all).
Maybe this would be better if requested in the Flutter repo, possibly in https://github.com/flutter/flutter/issues.

[stephane-archer]

In my experience, @lrhn requests on Flutter's repo are useless. It's just a constant fight not to get the issue set as a duplicate on some vaguely related issues.

Flutter plugin might be the wrong name. it's just a package that calls native code using usually "Platform Channels".
are "Platform Channels" not related to the Dart SDK?

[lrhn]

I think Platform Channels is a Flutter thing too. The Dart approach to calling native code is dart:ffi.

[blaugold]

In a sense, the solution for this issue is what is being worked on in dart-lang/native. IIUC the end goal is to have good enough support for autogenerated bindings for native APIs (which directly call those APIs through dart:ffi), so that the Flutter specific way to interact with native APIs through platform channels becomes obsolete.

[stephane-archer]

Sorry for my limited understanding of the subject. In my mind dart::ffi is a way to call C functions while platform Chanel is way to call swift code on iOS (and probably Kotlin on android)

I'm surprise this binding mechanism is not provided by the languages/compiler and is flutter specific.

I'm also surprised that the flutter project seems to have it's own stuff regarding binding and testing.

The main issue here is "flutter test" doesn't call native code and when you start mocking the native code that do the work you are actually not testing anything

But I guess "dart test" would not have the same limitations?

[blaugold]

The packages in dart-lang/native can generate Dart bindings for C, Swift, Objectiv-C and Java that use dart:ffi under the hood. So the mechanism that Flutter currently provides to call native/platform APIs will become obsolete since there will be a language level mechanism to do this.

If you use these bindings you will always directly call the native APIs regardless of whether running with dart run, dart test, flutter run or flutter test.

But note that these tools are not stable yet, and it will probably take a bit more time until they are.

[stuartmorgan-g]

I'm not sure what Dart feature would help with this (or what a Flutter plugin is at all).

Plugin packages are a Flutter-specific extension to Dart packages that (among other things) compiles native code in the package into the resulting Flutter app, by grafting it into the Flutter build system. There is no Dart feature that would allow accessing the native code in a plugin because without flutter the native code won't even be built.

The in-progress native assets feature would provide an alternative whose output dart is aware of, but any code based on Flutter's plugin package system rather than native assets is by definition Flutter-specific.

I think Platform Channels is a Flutter thing too.

Correct; platform channels are defined and implemented entirely at the Flutter level.

[stuartmorgan-g]

If you use these bindings you will always directly call the native APIs regardless of whether running with dart run, dart test, flutter run or flutter test.

This is only sort of true. If you are using jnigen, you would need to have a JVM loaded into the dart test process to call Java APIs. If you are calling Android or iOS SDK methods (common in Flutter plugin use cases), they won't exist in the context in which dart test runs.

[stephane-archer]

@stuartmorgan would you be able to explain why "flutter test" doesn't give you access to platform channels?

Does this mecanism use dart::ffi under the hood?

———-

I also would like to apologize for a conversation we had on the Flutter repo, As described above, I have been frustrated with the Flutter repo issue system and I ended up sharing this frustration with contributors, I was wrong doing so

[lrhn]

This is not the best repo for getting information about why flutter test does what it does.

The request, as written, is not a Dart issue.
There is nothing directly actionable in the Dart SDK here, unless it intersects with what is being done as part of native interop anyway. If there are specific requests to Dart native interop that isn't covered already, I'd open an issue specifically for that.