Database.copy throwing an error

[PritamSangani]

Describe the bug

I am trying to copy a pre-built database (zipped up) using the Database.copy method using code from the bottom of this page as a guide https://cbl-dart.dev/prebuilt-database/#creating-pre-built-database.

When unzipped the prebuilt database folder structure is like:

dbname-pre-built.cblite2
    > db.sqlite3
    > Attachments

with the arrows representing the hierachy of the folder.

The database gets loaded into the app directory succesfully and I have verified this by using the adb file explorer. However, when the code gets to the Database.copy step an unhandled exception is thrown. The exception looks like this:

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: DatabaseException(errno 21: Couldn't copy file from /data/user/0/[package name]/app_flutter/databases/[db name]-prebuilt.cblite2/db.sqlite3 to /data/user/0/[package name]/app_flutter/databases/.cblite/7GZwRI/: Is a directory, code: unexpectedError)

To Reproduce

The code I am using is very similar to what is on the page I have linked above, but I'll post the relevant code here anyway (leaving out the part of checking if db exists)

    final databasesDirectory = '${(await getApplicationDocumentsDirectory()).path}/databases';
    const prebuiltDatabaseAssetFileName = 'assets/database/prebuiltDatabase.zip';
    final prebuiltDatabaseZip = await rootBundle.load(prebuiltDatabaseAssetFileName);
    if (prebuiltDatabaseZip != null && prebuiltDatabaseZip.lengthInBytes > 0) {
      final prebuiltDatabaseFileName = '$databasesDirectory/example-prebuilt.cblite2';
      // Decompress the zip file into bytes and then convert them into a
      // Uint8List, which is required by the Archive framework.
      final archive = ZipDecoder().decodeBytes(prebuiltDatabaseZip.buffer.asUint8List());
      for (final file in archive) {
        final fileName = '$prebuiltDatabaseFileName/${file.name}';
        if (file.isFile) {
          final fileData = file.content as List<int>;
          File(fileName)
            ..createSync(recursive: true)
            ..writeAsBytesSync(fileData);
        } else {
          Directory(fileName).createSync(recursive: true);
        }
      }

      await Database.copy(
        from: prebuiltDatabaseFileName,
        name: databaseName,
        config: config,
      );

    }

Expected behavior

The database should get copied over successfully

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Android
• cbl: ^2.1.2
• cbl_flutter: ^2.0.2
• cbl_flutter_ce: ^2.1.2

Additional context
I have verified that the database got built successfully by running cblite ls on the .cblite2 file and all the document ids were logged.

[blaugold]

Thanks for filing this issue!

There were a few issues with the example code, and I have rewritten it:

import 'dart:io';

import 'package:archive/archive_io.dart';
import 'package:cbl/cbl.dart';
import 'package:flutter/services.dart';
import 'package:path/path.dart' as path;
import 'package:path_provider/path_provider.dart';

Future<void> installMyPrebuiltDatabase() async {
  final zipArchive =
      await rootBundle.load('asset/database/my-database-prebuilt.zip');
  final documentsDirectory = await getApplicationDocumentsDirectory();
  final databasesDirectory = path.join(documentsDirectory.path, 'databases');

  await installPrebuiltDatabase(
    zipArchive: Uint8List.sublistView(zipArchive),
    nameInArchive: 'my-database',
    name: 'my-database',
    config: DatabaseConfiguration(directory: databasesDirectory),
  );
}

/// Installs a prebuilt database from a [zipArchive].
///
/// The prebuilt database must have the name specified by [nameInArchive] and be
/// located at the root of the archive.
///
/// To create the archive, use the `zip` command line tool:
///
/// ```sh
/// zip -r my-database-prebuilt.zip my-database.cblite2
/// ```
///
/// [name] and [config] are used to install the database.
Future<void> installPrebuiltDatabase({
  required Uint8List zipArchive,
  required String nameInArchive,
  required DatabaseConfiguration config,
  required String name,
}) async {
  // Dont't overwrite an existing database.
  if (await Database.exists(name, directory: config.directory)) {
    return;
  }

  // Use a temporary directory for unpacking the prebuilt database to avoid
  // naming conflicts.
  final tempDirectory =
      await Directory(config.directory).createTemp('prebuilt-');
  final prebuiltDatabaseFileName =
      path.join(tempDirectory.path, '$nameInArchive.cblite2');

  try {
    // Unpack the prebuilt database.
    final archive = ZipDecoder().decodeBytes(zipArchive);
    await extractArchiveToDiskAsync(archive, tempDirectory.path);

    // Make a copy of the prebuilt database.
    await Database.copy(
      from: prebuiltDatabaseFileName,
      name: name,
      config: config,
    );
  } finally {
    // Clean up the temporary directory.
    await tempDirectory.delete(recursive: true);
  }
}

Future<void> extractArchiveToDiskAsync(
  Archive archive,
  String outDirectory,
) async {
  for (final file in archive.files) {
    final fileName = path.join(outDirectory, file.name);
    if (file.isFile) {
      final ioFile = File(fileName);
      await ioFile.create(recursive: true);
      await ioFile.writeAsBytes(file.content as List<int>);
    } else {
      await Directory(fileName).create(recursive: true);
    }
  }
}

Could you take a look at it and try it out?

[PritamSangani]

Hi @blaugold - thanks for taking a look into this. I tried using the code that you provided. I made one small change to create the databases directory first before creating the temp directory. The zip file gets extracted to the temp directory successfully. However, the same exception is still being raised during the Database.copy call.

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: DatabaseException(errno 21: Couldn't copy file from /data/user/0/[package name]/app_flutter/databases/prebuilt-HEBYXR/[nameInArchive].cblite2 to /data/user/0/[package name]/app_flutter/databases/.cblite/l4hBOW/: Is a directory, code: unexpectedError)

[blaugold]

Thanks for trying this.

I understand the error to mean that we are trying to copy a file (.../prebuilt-HEBYXR/[nameInArchive].cblite2) but this fails because it is actually a directory. In this error message, this makes sense since a .cblite2 database is just a directory, but I don't understand why we would be trying to copy it as a file.

In the first error message [db name]-prebuilt.cblite2/db.sqlite3 is referenced which should be a file, so there should be no issue copying it as one. My first guess was that there is a problem with the archive. Can you unzip it and verify that the database is still intact by using the cblite tool on it?

I haven't tried to replicate the issue on Android yet, so I'll try that too. It might be specific to that OS.

[PritamSangani]

I removed the line to delete the temp directory and then using adb saved the cblite2 file that was extracted. I can confirm that the file is still intact and returns all the document ids when running cblite ls on it. I also just tried it on ios using the same code and zip file and it works so it does seem like it is an issue specific to android.

[biozal]

I wrote using a prebuild database for our learning path, and when I tested it on both iOS and Android, it worked fine (of course, this was a few weeks ago). My code is here:

https://github.com/couchbase-examples/flutter_cbl_learning_path/blob/main/src/lib/features/database/database_provider.dart#L76

I wrote how to do this as part of our learning path for Flutter using CBL-Dart, which is a work in progress but was tested about a month ago.
https://developer.couchbase.com/dart-flutter-prebuilt-database?learningPath=learn/flutter-dart-sdk-app-services

Not sure if this helps or not.

[blaugold]

@PritamSangani Thanks for letting me know.

@biozal Definitely good to know that you got it working! I hadn't seen the learning path on the developer portal yet. Very cool!

[biozal]

@blaugold - it's still a work in progress I have 2 things to finish up on it, but some App Services stuff has been chewing my free time. My plan is to finish it up with 3 more sections in the next few weeks.

[PritamSangani]

@biozal - Thanks for sharing. The learning path is really interesting and I like that it is a complex app, which really showcases what can be done with couchbase!

The example repo you posted seems to be working for ios but it doesn't work for android. For android, I am getting the same error at the Database.copy call as I have mentioned above.

@blaugold

I double checked the log output for android and ios and I noticed that the directory that the database is being copied to is different for android and ios.

Using the example from the link that @biozal posted, startingWarehouses.cblite2 is the name of the pre-built database and warehouse is the name of the database. I have truncated the full paths for brevity.

For ios the copy was happening from .../databases/startingWarehouses.cblite2 to .../databases/warehouse.cblite2.

However, for android the copy is happening from .../databases/startingWarehouses.cblite2 to .../databases/.cblite/[id]/

I think the database should be copied to [databaseName].cblite2 not a .cblite/[id] folder (this is probably what the error, "Is a directory", is referring to)

I believe there is a bug in how the to path is being derived for android platform. Is the to path being derived within this library or is that done within the couchbase-lite-c library?

[PritamSangani]

@blaugold - have you had a chance to take a look at whether this is an issue in this library or the native cblite c library?

[blaugold]

Sorry for not responding earlier. I found the problem but forgot to update the issue.

The Couchbase Lite C SDK expects the path to the source database to end with a path separator, e.g. /. This is also what Database.path returns. On Apple platforms, the code that copies files can handle it if the trailing path separator is missing, but on all platforms it can't.

I would classify that as a bug, since the expected format is not documented and is not super common. I think the Couchbase Lite C SDK should take paths with and without trailing slash, but I can implement a fix in CBL Dart meanwhile.

[PritamSangani]

This worked! Thanks for your help and quick responses @blaugold!