[Flutter] openStore() is not working across multiple FlutterEngines

[navaronbracke]

I have a use case where I have a Flutter app that does two things.

• a void main(){} entrypoint that runs a regular Flutter app (i.e. runApp(MaterialApp()))
  This app uses the database normally. It also schedules tasks using the workmanager plugin
• the workmanager plugin that executes the tasks that the app schedules

The problem

The void main(){} entrypoint is executed from the default FlutterEngine (created by the Activity/AppDelegate)
The workmanager plugin creates a new FlutterEngine for each task it needs to run. This is because the DartExecutor from the original FlutterEngine is executing the void main(){} entrypoint. A DartExecutor can only run one entrypoint at a time.

Because there are two FlutterEngines (each with their own Isolate pool and such), the Dart code is not in sync between the Engines. That is, one engine might have a Store _myStore that is not null, but the other engine still has one that is null (because it does not have the same memory pool allocated).

This results in the following code failing on the FlutterEngine that didn't open the store:

class MyDbService {
  static Store _myStore;

  Future<void> init() async {
   // The store is null in the other entrypoint, since that is an entire new block of memory, owned by another engine.
   // This results in the error
   // Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path 
   _store ??= await openStore();
  }
}

Describe the solution you'd like

I'd like to be able to use a Store across different FlutterEngines.
If openStore() would return a new connection, regardless of the FlutterEngine, that would be sufficient I think. (I.e. using a connection pool)
I'd expect the ObjectBox API to work as-is through this connection. I.e. CRUD / observable queries should work on this new connection.

Then I could do something like this to fix my problem:

main.dart

void main() async {
   await DbService().initialize(); // Open the database service in the first FlutterEngine

  runApp(MyApp()); // Does call to `WorkManager.registerTask()` which invokes the task runner function
}

my_app.dart

class MyApp extends StatefulWidget {
 // ...
}

class _MyAppState extends State<MyApp> {
 @override
 Widget build(BuildContext context){
    return Scaffold(
      appBar: AppBar(title: Text('My app')),
      body: Center(
        child: ElevatedButton(
           child: Text('Schedule task'),
           onPressed: (){
             WorkManager().registerTask('MyTask', {'input': 'foo'});
           }
        ),  
      ),
    );
  }


  @override
  void dispose() async {
   await DbService().close(); //Close the database service in the first FlutterEngine
   super.dispose();
  }
}

task_runner.dart

@pragma("vm:entry-point")
void _taskRunner(){
   WidgetsFlutterBinding.ensureInitialized();

    Workmanager().executeTask((taskName, inputData) async {
      await DbService().initialize(); // open a new connection in this FlutterEngine

      switch(taskName){
        case 'MyTask':
          // do work
          break;
      }

      await DbService().close(); // open the connection that was closed in this FlutterEngine
    }
}

Additional note

This could also benefit the add-to-app use case where people use a FlutterEngine per view they embed into an existing native app.

[greenrobot-team]

Thanks for reporting!

First of all, using multiple Flutter Engines seems to be a valid use case. The API is advertised to be used to have one or more Flutter screens/views in an otherwise non-Flutter app as the note above says.

I'm not familiar with how this API works, so the following might not be right: the ObjectBox store is created via FFI and refered to using a native pointer. So it should technically be accessible from anything that runs in the same Dart VM. So to access an open store attach to it using Store.attach?

[navaronbracke]

@greenrobot-team I could try to use Store.attach() in the handler that is run on the second FlutterEngine. However, since the second FlutterEngine is created by a background thread (started natively by WorkManager), I think I'll end up with a second Dart VM, which won't see the initialized store from the first one (or the other way around). I'll give it a try and let you know.

[navaronbracke]

@greenrobot-team I had another shot at it and Store.attach() did not work. I used the following code:

  Future<String> _getDatabaseDirectoryPath() async {
    final dir = await getApplicationDocumentsDirectory();

    // The default object box dir is inside the application documents dir,
    // under the `/objectbox` folder.
    return '${dir.path}${Platform.pathSeparator}objectbox';
  }

  Future<void> initialize() async {
    final dbPath = await _getDatabaseDirectoryPath();

    try {
      // Try to open the store normally.
      _store ??= await openStore(directory: dbPath);
    } catch (error) {
      // If the store cannot be opened, it might already be open.
      // Try to attach to it instead.
      _store = Store.attach(getObjectBoxModel(), dbPath);
    }
  }

In the background thread I never end up in the Store.attach() phase since the Store is null in that Isolate (because it runs on a different FlutterEngine and thus does not share memory with the first FlutterEngine).
Only in the first Isolate the Store is not null.
This results in the openStore() function throwing

Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path

I think I need a way to check if the native store is open (internally that should check the FFI pointer) and then I could use Store.attach()? It should be possible since the native store throws that state error?

I'll be happy to provide a minimal reproducible sample app to pinpoint the problem.

[navaronbracke]

@greenrobot-team In relation to the other issue I had, I'll try to check if the store is open with that static method. Maybe that fixes this issue?

[navaronbracke]

@greenrobot-team I got it working using Store.isOpen() and using attach if its already open. I have one more question though: Does the Store emit database updates to each connection?
I.e. if I make changes in connection 2, will connection 1 be able to see them?
My use case is that the background worker modifies the database and the app observes those changes through its own connection.

[greenrobot-team]

I read through the code example and docs from Flutter: there should only exist a single Dart VM for all FlutterEngines. And yes, they do not share state.

So _store ??= doesn't work. Using Store.isOpen(path) and then calling attach instead of open as you mentioned is the the way to go then.

Change notifications should happen on any isolate/engine as the native side is handling notifications. E.g. it should be possible to put an object in a background worker which is observed by a watched query in the UI.

Edit: let me know if this resolves your original request (and this can be closed).

[navaronbracke]

@greenrobot-team This does indeed resolve my problem, thank you. And yes I managed to use Store.isOpen() to fix the connection issue.

Closing as working as intended.

[madrojudi]

Hello!
@navaronbracke and @animedev1, please, can you tell me how you did it?

This is my code to init my store

if(Store.isOpen(null)) {
        print("first attach");
        _store = Store.attach(getObjectBoxModel(), null);
        print("first attach done");
      } else {
        try {
          print("try open");
          _store = await openStore();
          print("try open done");
        }catch (ex) {
          print("catch to open $ex");
        }
      }

When my application is not closed, but it is not in the foreground, it generate this error, using Firebase Cloud Messaging on background :
Bad state: failed to create store: 10001 Cannot open store: another store is still open using the same path: "/data/user/999/io.artcreativity.app/app_flutter/objectbox"

When app is closed, it work well.

You can learn more about my issue here #451

Thank you!

[navaronbracke]

@madrojudi I did it like this:

    if (Store.isOpen(dbPath)) {
      _store = Store.attach(getObjectBoxModel(), dbPath);

      return;
    }

    _store = await openStore(directory: dbPath);

_store is a variable of type Store? which I use to store the opened store. dbPath is the path to the database as specified by attach & openStore, but you probably have that already.

[madrojudi]

Thank you @navaronbracke
Unfortunately it doesn't always work for me.
But I found an alternative that is currently working.

1. I check if DB is open with Store.isOpen(path). If it is true, I use Store.attach
2. If Store.isOpen is false, I try to open new Store by openStore. When it open, I save the reference into a file
3. If Store.isOpen throw error, I check to read the reference which I save in 2. and I use Store.fromReference to open Store

Currently, It is working. I will continue testing to see if it is stable.

[meomap]

Thank you madrojudi

I tried you way and it works sometimes. Other times, Saving reference then reading back will throw error: [ERROR:flutter/lib/ui/ui_dart_state.cc(198)] Unhandled Exception: Invalid argument(s): Reference.processId 692 doesn't match the current process PID 2046

As in our case, we fire background process when users interact with Widget remote views so the current workaround will eventually lead to race condition

[greenrobot-team]

As an alternative to an open check and then doing either open or attach, see #442 (comment) for a workaround on Android.

According to this comment having multiple engines is a more common occurrence than thought (e.g. when deep-linking or opening from a notification), so maybe we should update the docs and maybe even offer API for this.

[techouse]

@greenrobot-team please also mention in the docs how to properly get the default path of the ObjectBox store in Flutter, i.e.

import 'package:path/path.dart' as path;
import 'package:path_provider/path_provider.dart';

Future<void> main() async {
  final String storeDirectoryPath = path.join(
   (await getApplicationDocumentsDirectory()).path,
   Store.defaultDirectoryPath,
  );
}

[greenrobot-team]

@techouse Thanks, as mentioned above already have done this. It just wasn't released, yet.

objectbox-dart/objectbox/lib/src/native/store.dart

Lines 446 to 454 in fe91dbf

	/// Returns if an open store (i.e. opened before and not yet closed) was found
	/// for the given [directoryPath].
	///
	/// For Flutter apps, the default [directoryPath] can be obtained with
	/// `(await defaultStoreDirectory()).path` from `objectbox_flutter_libs`
	/// (or `objectbox_sync_flutter_libs`).
	///
	/// For Dart Native apps, pass null to use the [defaultDirectoryPath].
	static bool isOpen(String? directoryPath) {

Edit: this was included with release 2.0.0.