[vm/ffi] clarify DynamicLibrary executable and process

[dcharkes]

We have two factory constructors in DynamicLibrary, process and executable, which currently have identical behavior. They can lookup symbols in the executable and statically linked libraries, but not dynamically linked libraries.(x)

They would behave differently if DynamicLibrart.open would load a library with global visibility instead of local visibility. In that case DynamicLibrart.process would be able to see the symbols, but DynamicLibrart.executable would not.

We could do one of the following:

1. Deprecate DynamicLibrary.process and always use DynamicLibrary.executable. (DynamicLibrary.process is not available on Windows, DynamicLibrary.executable is.)
2. Change DynamicLibrary.open to enable loading a library with global visibility by adding a named argument.

class DynamicLibrary {
  external static DynamicLibrary open(String path, {bool globalVisibility = false});
}

(x) One can do a lookup of dlopen, and then dlopen with global visibility. That would make the symbol visible in DynamicLibrart.process.

Some experiments: https://dart-review.googlesource.com/c/sdk/+/182629. On the various OSes we have different behavior.

[virtualzeta]

dcharkes
They would behave differently if DynamicLibrart.open would load a library with global visibility instead of local visibility. In that case DynamicLibrart.process would be able to see the symbols, but DynamicLibrart.executable would not.

Hi dcharkes, I've been studying external library management lately and I've read a lot of your interesting posts and I congratulate you on your work.

I'm sticking to this post, hoping not to go OT and get some answers.

In my SDK (shown below) I don't yet have an argument like globalVisibility in the DynamicLibrary.open constructor but I describe what I tested.

Part A: Dart

After calling the native library with:

var library = DynamicLibrary.open('<path_library>/lib<name_library>.so');

I see with DynamicLibrary.process().providesSymbol('<name_symbol>') that the symbol is present while with DynamicLibrary.executable().providesSymbol('<name_symbol>') I get false.

So if I can see in process the symbols loaded with open they should already be of global type in this version.

I can get response from the native function using the lookup method of the library variable (both as an instance obtained with open and process, after loading it) and I can also use it having marked it with @FfiNative in this way.

For example, if the function is add I declare (as a method of a class or not):

    @FfiNative<Int32 Function(Int32, Int32)>('add')
    external int add(
      int a,
      int b,
    );

No problem both ways.

Part B: Flutter

When I'm in Flutter workspace, using the Android platform in debug mode, things change and after loading the native library with:

var library = DynamicLibrary.open('<library_name>.so');

see the following differences:

1. the process and executable constructors do not see the symbol and only the instance obtained with open sees it
2. I can use the native function only using the lookup method of the DynamicLibrary instance obtained with open
3. using @FfiNative I don't get the symbol match and I get the error "Invalid Arguments: Unable to resolve native function 'add' in 'package:...' : undefined symbol: add."

My goal (not final but partial) is to understand how to use @FfiNative also in Flutter with native libraries.

I think in this case the symbols are no longer global.
Although I haven't tried it, it seems that with iOS you get a mapping of symbols globally when loading the app, but in any case I don't know if this would work.

I know that FfiNative is deprecated in favor of Native but this last class I can't use/call it (because system reserved, I think) and anyway I don't know if it would make a difference.

Is there a way to be able to do this by acting on the Dart code, native code, CMakeLists,txt or elsewhere to correctly compile the external functions and/or have the symbols globally?

Thank you.

Flutter 3.7.3 • stable channel • Dart 2.19.2

[dcharkes]

#50105 (comment)

MacOS and Linux open with GLOBAL, while Android does not. That explains the difference between the Flutter and Dart experience. Flutter Desktop should also work with GLOBAL.

The workaround is to call dlopen yourself with the right arguments

/// On Linux and Android.
const RTLD_LAZY = 0x00001;

/// On Android Arm.
const RTLD_GLOBAL_android_arm32 = 0x00002;

/// On Linux and Android Arm64.
const RTLD_GLOBAL_rest = 0x00100;

final RTLD_GLOBAL = Abi.current() == Abi.androidArm
    ? RTLD_GLOBAL_android_arm32
    : RTLD_GLOBAL_rest;

@Native<Pointer<Void> Function(Pointer<Char>, Int)>()
external Pointer<Void> dlopen(Pointer<Char> file, int mode);

/// Returns dylib
Object dlopenGlobalPlatformSpecific(String name, {String? path}) {
  if (Platform.isLinux || Platform.isAndroid || Platform.isFuchsia) {
    // TODO(https://dartbug.com/50105): enable dlopen global via package:ffi.
    return using((arena) {
      final dylibHandle = dlopen(
          platformPath(name).toNativeUtf8(allocator: arena).cast(),
          RTLD_LAZY | RTLD_GLOBAL);
      return dylibHandle;
    });
  } else {
    // The default behavior on these platforms is RLTD_GLOBAL already.
    return dlopenPlatformSpecific(name, path: path);
  }
}

In the future, you will be able to use @Native without relying on opening things global. #50565 The native assets feature is available behind an experimental flag in Dart. It is not yet available in Flutter. flutter/flutter#129757

[virtualzeta]

#50105 (comment)

MacOS and Linux open with GLOBAL, while Android does not. That explains the difference between the Flutter and Dart experience. Flutter Desktop should also work with GLOBAL.

The workaround is to call dlopen yourself with the right arguments

/// On Linux and Android.
const RTLD_LAZY = 0x00001;

/// On Android Arm.
const RTLD_GLOBAL_android_arm32 = 0x00002;

/// On Linux and Android Arm64.
const RTLD_GLOBAL_rest = 0x00100;

final RTLD_GLOBAL = Abi.current() == Abi.androidArm
    ? RTLD_GLOBAL_android_arm32
    : RTLD_GLOBAL_rest;

@Native<Pointer<Void> Function(Pointer<Char>, Int)>()
external Pointer<Void> dlopen(Pointer<Char> file, int mode);

/// Returns dylib
Object dlopenGlobalPlatformSpecific(String name, {String? path}) {
  if (Platform.isLinux || Platform.isAndroid || Platform.isFuchsia) {
    // TODO(https://dartbug.com/50105): enable dlopen global via package:ffi.
    return using((arena) {
      final dylibHandle = dlopen(
          platformPath(name).toNativeUtf8(allocator: arena).cast(),
          RTLD_LAZY | RTLD_GLOBAL);
      return dylibHandle;
    });
  } else {
    // The default behavior on these platforms is RLTD_GLOBAL already.
    return dlopenPlatformSpecific(name, path: path);
  }
}

In the future, you will be able to use @Native without relying on opening things global. #50565 The native assets feature is available behind an experimental flag in Dart. It is not yet available in Flutter. flutter/flutter#129757

Thanks for your fast reply.

As often happens, an answer brings other doubts with it and, after having tried to resolve them on my own, I think it is appropriate that I ask you some questions (even basic ones) to dispel them.
I also hope it will help others like me who still have a lot to learn from Dart, Flutter and also from C.

I state that I am about to release an app and I would like to avoid changing SDK versions at this moment or prepare for the use of SDK manager, leaving the thing to other moments in which I can test the work environment more serenely .

I remember that at the moment I'm using this configuration: Flutter 3.7.3 • stable channel • Dart 2.19.2.

Here are some statements and questions for you to check.

1. What you said and linked to me made me understand that the difference in behavior is not related to the SDK but to the platform and, in this case, the question is really related to whether the symbols are local or global.
   I presume that using Dart standalone the reference platform is in reference to the operating system used: I am using Windows and therefore have a the type of the RLTD is global. I also tried Flutter with Windows target platform and in fact I can use the example function because the symbols are global.

2. Waiting to have that eventual globalVisibility argument in the DynamicLibrary.open constructor in some release, I can now directly access the dlopen native function passing library name and Runtime Linker/Loader parameter depending on the platform.

3. The dlopen is a basic C function present in the dlfcn.h library and if in Dart I want to call it in the same way in the declaration below I omit the name as per the specifications of the Native class bearing in mind that in C it is declared as void *dlopen(const char *filename, int flags) (or I'd let ffigen do it if I'm starting from custom native code).

@Native<Pointer<Void> Function(Pointer<Char>, Int)>()
external Pointer<Void> dlopen(Pointer<Char> file, int mode);

4. My IDE (VS Code) doesn't give me any results for the using, platformPath and dlopenPlatformSpecific.
   After unsuccessfully searching for them in my SDK and in the latest downloaded version for checking (unsuccessfully) I finally found using by importing package:ffi/ffi.dart (unlike other used classes which stay in dart:ffi) and understood that the others were just personal functions to handle library name + path and native library fetched normally because the referenced platform already provides global symbols.
   I used these, for test:

DynamicLibrary dlopenPlatformSpecific(String name, {String? path}) {
  DynamicLibrary nativeLib =
      DynamicLibrary.open(platformPath(name, path: path));
  return nativeLib;
}

String platformPath(String name, {String? path}) {
  path ??= "";
  return "$path$name";
}

5. The use of the --enable-experiment=native-assets flag of Dart is to be used only when launching/compiling/testing the Dart project concerned without it modifying the SDK in any way or remaining active subsequently for other projects?
   I tried first if the commands --disable-experiment=native-assets and --disable-experiment responded (as for --disable-analytics) but they are not recognized.

6. Is the Native class, which as I said I don't have available, usable/recognized via the Dart flag?

7. As an alternative to having the original Native class available, I tried to declare one in the same file (or externally) as per Dart specifications like below, but obviously it is a class to implement and I would like to know where this is done for Native and FfiNative to know the code used.

class Native<T> {
   finalString? symbol;
   finalString? assets;
   final bool isLeaf;

   const Native({
     this.asset,
     this.isLeaf: false,
     this.symbol,
   });
}

The following error is returned:

I/flutter ( 6393): NoSuchMethodError: No top-level method 'dlopen' declared.
I/flutter ( 6393): Receiver: top-level
I/flutter ( 6393): Tried calling: dlopen()

8. Considering that the use made with Native here does not use other parameters not present in FfiNative, I tried to use that one using the non-omissable nativeName 'dlopen', taking into consideration that since loading Flutter that DynamicLibrary.process().providesSymbol("dlopen") and DynamicLibrary.executable().providesSymbol("dlopen") are true and therefore the symbol is certainly present.

@FfiNative<Pointer<Void> Function(Pointer<Char>, Int)>("dlopen")
external Pointer<Void> dlopen(Pointer<Char> file, int mode);

But although in Android this does not throw an error with the dlopen it returns an handle with address=0x0 and I certainly can't fetch symbols.

But with the Windows platform (which shouldn't be affected given the premises but I used it as a test) the behavior is different and directly returns an error.

flutter: Invalid argument(s): Couldn't resolve native function 'dlopen' in 'package:...' : None of the loaded modules contained the requested symbol 'dlopen'.

9. I was thinking that the return of the handle with zeroed address can somehow also be connected to an incorrect path. But since I'm sure the path to the library is correct for Android (I just put no path but just the standard name of the library lib<library_name>.so) and I can read the function with DynamicLibrary.open constructor and lookup, is it possible that using it adds a part of the path that you need to add manually with dlopen instead?
   The error with Windows leaves me a bit perplexed but I'm inclined towards a path error.

I tried to add via path_provider package (with trailing slash):

• /data/user/0/com.example.<app_name>/app_flutter
• /data/user/0/com.example.<app_name>/cache
• /data/user/0/com.example.<app_name>/files

Although Directory(path).existsSync() is true, File(platformPath(name, path: path)).existsSync() is always false.
And finally I found the correct path is /data/user/0/com.example.<app_name>/lib/lib<library_name>.so, manually adding "/lib" because getLibraryDirectory() is not compatible with Android.

Too bad that even with the correct path to pass to dlopen native function, the handler returns with zeroed address. >:-|

10. In the future, after the experimentation, will it be enough to use @Native with an external function declaration without worrying about whether the symbol is global or not because all the symbols of the loaded libraries will be present in the assets, without the need for them to be global or local?

I look forward to your instructions in order to successfully start the script.

[virtualzeta]

I was thinking that the return of the handle with zeroed address can somehow also be connected to an incorrect path. But since I'm sure the path to the library is correct for Android (I just put no path but just the standard name of the library lib<library_name>.so) and I can read the function with DynamicLibrary.open constructor and lookup, is it possible that using it adds a part of the path that you need to add manually with dlopen instead?
The error with Windows leaves me a bit perplexed but I'm inclined towards a path error.
I tried to add via path_provider package (with trailing slash):

/data/user/0/com.example.<app_name>/app_flutter
/data/user/0/com.example.<app_name>/cache
/data/user/0/com.example.<app_name>/files
Although Directory(path).existsSync() is true, File(platformPath(name, path: path)).existsSync() is always false.
And finally I found the correct path is /data/user/0/com.example.<app_name>/lib/lib<library_name>.so, manually adding "/lib" because getLibraryDirectory() is not compatible with Android.

Too bad that even with the correct path to pass to dlopen native function, the handler returns with zeroed address. >:-|

Regarding points 8-9 I opened a new problem to analyze it individually and even if I understood the main problem I also have to understand how to solve it.

dart-lang/native#923

[dcharkes]

4. My IDE (VS Code) doesn't give me any results for the using, platformPath and dlopenPlatformSpecific.

using is in package:ffi.

The other two you'll have to define yourself, for your convenience:

String platformPath(String name, {String path = ""}) {
  if (Platform.isLinux || Platform.isAndroid || Platform.isFuchsia)
    return path + "lib" + name + ".so";
  if (Platform.isMacOS) return path + "lib" + name + ".dylib";
  if (Platform.isWindows) return path + name + ".dll";
  throw Exception("Platform not implemented");
}

5. The use of the --enable-experiment=native-assets flag of Dart is to be used only when launching/compiling/testing the Dart project concerned without it modifying the SDK in any way or remaining active subsequently for other projects?

Only in the one invocation. (Dart experiments are per invocation. Flutter experiments are global and persisted. So once native assets are available there it will be a persisted setting.)

6. Is the Native class, which as I said I don't have available, usable/recognized via the Dart flag?

It's available @Since('2.19'). Are you using an older Dart?

But although in Android this does not throw an error with the dlopen it returns an handle with address=0x0 and I certainly can't fetch symbols.

If you can call it, you should be able to use it. Maybe you're passing the wrong path to the dynamic library or the wrong flags.

But with the Windows platform (which shouldn't be affected given the premises but I used it as a test) the behavior is different and directly returns an error.

Windows has a different API and does not use dlopen at all. When you use DynamicLibrary.open in Dart on a Windows machine it calls the corresponding Windows API instead. You should not have to do this on Windows, because by default everything is opened globally.

[virtualzeta]

using is in package:ffi.

Yes, as written previously I used import 'package:ffi/ffi.dart'. Alone package:ffi does not find the path.

It's available @SInCE('2.19'). Are you using an older Dart?

Yes, as stated and indicated in my comments, for a correct analysis, I use Dart 2.19.2.

In my version @FfiNative is already indicated as deprecated BUT @Native is not available as I have seen in the latest version (and certainly in other previous ones). I suspect there was some botch in the release.

Anyway this in my case shouldn't make any difference because the native function is also found with @FfiNative (as well as using lookup) and the problem is that since they are not the global symbols and so are unknown. This as a synthesis of what is indicated more extensively in the previous comments and in the problem opened separately following numerous tests.

[dcharkes]

Correct, @FfiNative and @Native should be identical in functionality. It's just a different syntax.