Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add utility to collect headers from google/mediapipe #10

Merged
merged 9 commits into from
Nov 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .flutter-mediapipe-root
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
// Used to normalize the paths of commands.
// The contents of this file do not matter.
Binary file not shown.
3 changes: 3 additions & 0 deletions tool/builder/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# https://dart.dev/guides/libraries/private-files
# Created by `dart pub`
.dart_tool/
3 changes: 3 additions & 0 deletions tool/builder/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## 1.0.0

- Initial version with headers command to sync C headers from `google/mediapipe`.
20 changes: 20 additions & 0 deletions tool/builder/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Flutter MediaPipe builder

Helper utility which performs build or CI/CD step operations necessary to develop, release, and use `flutter-mediapipe`.

### Usage:

Usage depends on which task you need to accomplish. All supported workflows are described below.

#### Header aggregation

Header files across all tasks in `google/flutter-mediapipe` have to stay in sync with their origin, `google/mediapipe`. To
resync these files, check out both repositories on the same machine (ideally next to each other on the file system) and run:

```sh
$ dart tool/builder/bin/main.dart headers
```

--

Check back in the future for any additional development workflows this command may support.
Piinks marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions tool/builder/analysis_options.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
include: package:lints/recommended.yaml

25 changes: 25 additions & 0 deletions tool/builder/bin/main.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is a new file the year should be 2023. Unless it's copied from somewhere?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the standard Flutter authors license.

// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:io' as io;
import 'package:args/command_runner.dart';
import 'package:builder/download_model.dart';
import 'package:builder/sync_headers.dart';
import 'package:logging/logging.dart';

final runner = CommandRunner(
'build',
'Performs build operations for google/flutter-mediapipe that '
'depend on contents in this repository',
)
..addCommand(SyncHeadersCommand())
..addCommand(DownloadModelCommand());

void main(List<String> arguments) {
Logger.root.onRecord.listen((LogRecord record) {
io.stdout
.writeln('${record.level.name}: ${record.time}: ${record.message}');
});
runner.run(arguments);
}
144 changes: 144 additions & 0 deletions tool/builder/lib/download_model.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:io' as io;
import 'package:args/command_runner.dart';
import 'package:builder/repo_finder.dart';
import 'package:http/http.dart' as http;
import 'package:logging/logging.dart';
import 'package:path/path.dart' as path;

final _log = Logger('DownloadModelCommand');

enum Models {
textclassification,
languagedetection,
}

class DownloadModelCommand extends Command with RepoFinderMixin {
@override
String description = 'Downloads a given MediaPipe model and places it in '
'the designated location.';
@override
String name = 'model';

DownloadModelCommand() {
argParser
..addOption(
'model',
abbr: 'm',
allowed: [
// Values will be added to this as the repository gets more
// integration tests that require new models.
Models.textclassification.name,
Models.languagedetection.name,
],
help: 'The desired model to download. Use this option if you want the '
'standard model for a given task. Using this option also removes any '
'need to use the `destination` option, as a value here implies a '
'destination. However, you still can specify a destination to '
'override the default location where the model is placed.\n'
'\n'
'Note: Either this or `custommodel` must be used. If both are '
'supplied, `model` is used.',
)
..addOption(
'custommodel',
abbr: 'c',
help: 'The desired model to download. Use this option if you want to '
'specify a specific and nonstandard model. Using this option means '
'you *must* use the `destination` option.\n'
'\n'
'Note: Either this or `model` must be used. If both are supplied, '
'`model` is used.',
)
..addOption(
'destination',
abbr: 'd',
help:
'The location to place the downloaded model. This value is required '
'if you use the `custommodel` option, but optional if you use the '
'`model` option.',
);
}

static final Map<String, String> _standardModelSources = {
Models.textclassification.name:
'https://storage.googleapis.com/mediapipe-models/text_classifier/bert_classifier/float32/1/bert_classifier.tflite',
Models.languagedetection.name:
'https://storage.googleapis.com/mediapipe-models/language_detector/language_detector/float32/1/language_detector.tflite',
};

static final Map<String, String> _standardModelDestinations = {
Models.textclassification.name:
'packages/mediapipe-task-text/example/assets/',
Models.languagedetection.name:
'packages/mediapipe-task-text/example/assets/',
};

@override
Future<void> run() async {
final io.Directory flutterMediaPipeDirectory = findFlutterMediaPipeRoot();

late final String modelSource;
late final String modelDestination;

if (argResults!['model'] != null) {
modelSource = _standardModelSources[argResults!['model']]!;
modelDestination = (_isArgProvided(argResults!['destination']))
? argResults!['destination']
: _standardModelDestinations[argResults!['model']]!;
} else {
if (argResults!['custommodel'] == null) {
throw Exception(
'You must use either the `model` or `custommodel` option.',
);
}
if (argResults!['destination'] == null) {
throw Exception(
'If you do not use the `model` option, then you must supply a '
'`destination`, as a "standard" destination cannot be used.',
);
}
modelSource = argResults!['custommodel'];
modelDestination = argResults!['destination'];
}

io.File destinationFile = io.File(
path.joinAll([
flutterMediaPipeDirectory.absolute.path,
modelDestination,
modelSource.split('/').last,
]),
);
ensureFolders(destinationFile);
await downloadModel(modelSource, destinationFile);
}

Future<void> downloadModel(
String modelSource,
io.File destinationFile,
) async {
_log.info('Downloading $modelSource');

// TODO(craiglabenz): Convert to StreamedResponse
final response = await http.get(Uri.parse(modelSource));

if (response.statusCode != 200) {
throw Exception('${response.statusCode} ${response.reasonPhrase} :: '
'$modelSource');
}

if (!(await destinationFile.exists())) {
_log.fine('Creating file at ${destinationFile.absolute.path}');
await destinationFile.create();
}

_log.fine('Downloaded ${response.contentLength} bytes');
_log.info('Saving to ${destinationFile.absolute.path}');
await destinationFile.writeAsBytes(response.bodyBytes);
}
}

bool _isArgProvided(String? val) => val != null && val != '';
120 changes: 120 additions & 0 deletions tool/builder/lib/repo_finder.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:io' as io;
import 'package:args/args.dart';
import 'package:args/command_runner.dart';
import 'package:path/path.dart' as path;
import 'package:io/ansi.dart';

/// Mixin to help [Command] subclasses locate both `google/mediapipe` and
/// the root of `google/flutter-mediapipe` (this repository).
///
/// The primary methods are [findFlutterMediaPipeRoot] and [findMediaPipeRoot].
///
/// By default, the root for `google/flutter-mediapipe` is determined by the
/// firest ancestor directory which contains a `.flutter-mediapipe-root` file
/// (whose contents are irrelevant), and the root of `google/mediapipe` is
/// expected to be a sibling of that. However, the `--source` flag can overwrite
/// this expectation and specify an absolute path where to find `google/mediapipe`.
///
/// Note that it is not possible to override the method of locating the root of
/// `google/flutter-mediapipe`.
mixin RepoFinderMixin on Command {
/// Name of the file which, when found, indicates the root of this repository.
static String sentinelFileName = '.flutter-mediapipe-root';

void addSourceOption(ArgParser argParser) {
argParser.addOption(
'source',
abbr: 's',
help: 'The location of google/mediapipe. Defaults to being '
'adjacent to google/flutter-mediapipe.',
);
}

/// Looks upward for the root of the `google/mediapipe` repository. This assumes
/// the `dart build` command is executed from within said repository. If it is
/// not executed from within, then this searching algorithm will reach the root
/// of the file system, log the error, and exit.
io.Directory findFlutterMediaPipeRoot() {
final placesChecked = <io.Directory>[];
io.Directory dir = io.Directory(path.current);
while (true) {
if (_isFlutterMediaPipeRoot(dir)) {
return dir;
}
placesChecked.add(dir);
dir = dir.parent;
if (dir.parent.path == dir.path) {
io.stderr.writeln(
wrapWith(
'Failed to find google/flutter-mediapipe root directory. '
'Did you execute this command from within the repository?\n'
'Looked in:',
[red],
),
);
io.stderr.writeln(
wrapWith(
placesChecked
.map<String>((dir) => ' - ${dir.absolute.path}')
.toList()
.join('\n'),
[red],
),
);
io.exit(1);
}
}
}

/// Finds the `google/mediapipe` checkout where artifacts built in this
/// repository should be sourced. By default, this command assumes the two
/// repositories are siblings on the file system, but the `--source` flag
/// allows for this assumption to be overridden.
io.Directory findMediaPipeRoot(
io.Directory flutterMediaPipeDir,
String? source,
) {
final mediaPipeDirectory = io.Directory(
source ??
path.joinAll([flutterMediaPipeDir.parent.absolute.path, 'mediapipe']),
);

if (!mediaPipeDirectory.existsSync()) {
io.stderr.writeln(
'Could not find ${mediaPipeDirectory.absolute.path}. '
'Folder does not exist.',
);
io.exit(1);
}
return mediaPipeDirectory;
}

/// Looks for the sentinel file of this repository's root directory. This allows
/// the `dart build` command to be run from various locations within the
/// `google/mediapipe` repository and still correctly set paths for all of its
/// operations.
bool _isFlutterMediaPipeRoot(io.Directory dir) {
return io.File(
path.joinAll(
[dir.absolute.path, sentinelFileName],
),
).existsSync();
}

/// Builds any missing folders between the file and the root of the repository
void ensureFolders(io.File file) {
io.Directory parent = file.parent;
List<io.Directory> dirsToCreate = [];
while (!parent.existsSync()) {
dirsToCreate.add(parent);
parent = parent.parent;
}
for (io.Directory dir in dirsToCreate.reversed) {
dir.createSync();
}
}
}
Loading