merge the ffi-wrapper branch into main (#23)

* Adds mediapipe_core package (#11) * adds mediapipe_core package * adds makefile for all packages * fixes typo in Makefile * Apply suggestions from code review * Update Makefile * Apply suggestions from code review * updated generated code's location and license (for 3P status) * code review responses including: * comments * licensing * resolving testing nits * updated README * setup sharing of base analysis_options * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * add free methods to core structs * code review updates to options * adds publish blocker * Add GitHub actions for CI/CD (#14) * adds CI/CD for mediapipe_core * add newlines * moves file into workflows dir * uncomment flutter doctor * testing PR config to run this now * added master and beta CI scripts * add executable permissions to CI scripts * adds ffiwrapper to ci/cd --------- Co-authored-by: Kate Lovett <katelovett@google.com> * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <katelovett@google.com> * [FFI] MediaPipe SDKs finder automation (#16) * adds sdks_finder command to builder utility * propagates changes to existing commands * adds manifest files generated by new sdks_finder command * updates in response to code review * Adds mediapipe_text package (#12) * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * Apply suggestions from code review * Adds example to `mediapipe-task-text` (#15) * initial commit of example * build file changes from `flutter pub get` * update main.dart * removes commented code * updates example for isolates design * Use `native-assets` to vendor MediaPipe SDK (#9) * adding bare structure for native assets * add MVP / first draft of build.dart * build.dart updates * update build.dart TODO: stream file * model memory troubleshooting * vendoring script tweak * remove development logging * removes pointless build method * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <katelovett@google.com> * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * native assets troubleshooting this commit is broken * Removes redundant count field * update build.dart for correct bindings path * download text classification model for CI * better memory freeing in executor * added SafeArea to example * added CI to PRs into text package * ci tooling change * remove accidentally commited model * more CI shenanigans * lowers minimum Dart version for builder * added smoke test for text example * Added CI/CD for examples * More CI tweaks * entering "please work" territory * d'oh * trying more random stuff * one more time * it'd be funny if this helped * more print statements * enable reaching new print statements * more logging * see what's in build dir * another test * adding flutter config list * turn off fail-fast for beta and master * moar logs * way moar prints * moare things * commit rest of rename * moar whatevers * adds manifest files generated by new sdks_finder command * adds sdks_finder command to builder utility * propagates changes to existing commands * updates in response to code review * updates to build.dart and tests * add Android runtime * sdks_finder logging improvement for when build folders change names * refreshed symbols from google/mediapipe * cleanup * loosens closeness thresholds in integration tests * separate build commands for macos architectures * restores fail-fast setting to CI * removed stale logging statements from CI * removes accidentally committed lines * add formatting of sdk_downloads.dart for CI * fixes broken example test * code touch ups from @Piinks code review --------- Co-authored-by: Kate Lovett <katelovett@google.com> * added base Dart class ClassificationResult to consolidate results logic in `mediapipe_core` in doing so, removed meaningless TextClassifierResult.timestamp field * Update Makefile * Update Makefile * Update packages/mediapipe-task-text/build.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/lib/src/tasks/text_classification/text_classification_executor.dart Co-authored-by: Kate Lovett <katelovett@google.com> * code review responses * formatting * removes stale comment * adds Dart to Native converters, with tests * changes from code review * updates mediapipe-core to prepare for IO/web split * Improves memory management in core tests * updated ffigen / bindings * refactors text package for better memory management and eventual web/io split * removed stale test * cleanup on aisle COMMENTS * Comments and documentation improvements * Moved log statement * Removed native memory management helpers in favor of `free` extensions on pointers * Renamed abstract classes to have Base prefix * sorted out class constructors * moved `fake` constructor to default unnamed constructor * leaning on the fact that the native constructors will be hidden by conditional exports, reducing confusion * Improved docstrings explaining memory ownership * Convert lists to lazy iterable / generators * added missing licenses * Update packages/mediapipe-task-text/example/test/widgets_test.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/lib/main.dart Co-authored-by: Kate Lovett <katelovett@google.com> * completed return style change * CI troubleshooting * moved around debugging code * logging tweak * cat native-assets.yaml * moar logs * removed bad echo * fixed native-assets typo should be underscore! * Removes CI debugging statements --------- Co-authored-by: Kate Lovett <katelovett@google.com> * Native Assets CI fix (#20) * adds native assets debugging statements * Try only downloading target arch * Revert "adds native assets debugging statements" This reverts commit b2bc215. --------- Co-authored-by: Daco Harkes <dacoharkes@google.com> * Text Embedding task (#21) * updated and re-ran generators * added embedding concepts to mediapipe-core * fixed embedding header file and bindings * adds text embedding classes to text pkg * updates example with text embedding * removed dead file * added more embedding tests * added embedding model download to CI script * touch ups * Update packages/mediapipe-core/lib/src/io/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/.gitignore Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/lib/text_embedding_demo.dart Co-authored-by: Kate Lovett <katelovett@google.com> * moved worker dispose method to base class * docstring & comment improvements * throw exceptions in impossible code paths instead of returning null * class hierarchy improvements * fixed outdates tests * cleaned up dispose methods * various tidying * fixed deprecation warning * moves repeated widgets into helper method --------- Co-authored-by: Kate Lovett <katelovett@google.com> --------- Co-authored-by: Craig Labenz <craig.labenz@gmail.com> Co-authored-by: Kate Lovett <katelovett@google.com> Co-authored-by: Daco Harkes <dacoharkes@google.com>
google · Apr 16, 2024 · 64c65dc · 64c65dc
1 parent 3046f3e
commit 64c65dc
Show file tree

Hide file tree

Showing 198 changed files with 10,127 additions and 0 deletions.
diff --git a/.flutter-mediapipe-root b/.flutter-mediapipe-root
@@ -0,0 +1,2 @@
+// Used to normalize the paths of commands.
+// The contents of this file do not matter.
diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
@@ -0,0 +1,38 @@
+name: Main Branch CI
+
+# Declare default permissions as read only.
+permissions: read-all
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main, ffi-wrapper, ffi-wrapper-text-pkg]
+  workflow_dispatch:
+  schedule:
+    - cron: "0 0 * * *" # Every day at midnight
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  flutter-tests:
+    name: Test mediapipe_core against ${{ matrix.flutter_version }}
+    runs-on: ${{ matrix.os }}
+    # Skip running job on forks
+    if: github.repository == 'google/flutter-mediapipe'
+    strategy:
+      fail-fast: false
+      matrix:
+        flutter_version: [master]
+        # TODO(craiglabenz): reverse once CI works
+        # flutter_version: [stable, beta, master]
+        # TODO(craiglabenz): Add `ubuntu-latest` and `windows-latest` when those artifacts exist
+        os: [macos-latest]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: subosito/flutter-action@v2
+        with:
+          channel: ${{ matrix.flutter_version }}
+      - run: ./tool/mediapipe_ci_script_${{ matrix.flutter_version }}.sh
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,8 @@
+{
+    "files.associations": {
+        "base_options.h": "c",
+        "__config": "c",
+        "text_classifier.h": "c",
+        "category.h": "c"
+    }
+}
diff --git a/Makefile b/Makefile
@@ -0,0 +1,60 @@
+# Runs the utility to pull in all header files from `google/mediapipe`
+headers:
+	cd tool/builder && dart bin/main.dart headers
+
+# Downloads all necessary task models
+models:
+	cd tool/builder && dart bin/main.dart model -m textclassification
+	cd tool/builder && dart bin/main.dart model -m textembedding
+
+
+# Runs `ffigen` for all packages
+generate: generate_core generate_text
+
+# Runs `ffigen` for all packages and runs all tests
+test: generate_core test_core generate_text test_text
+
+# Runs all tests for all packages
+test_only: test_core test_text
+
+# Rebuilds the MediaPipe task for macOS
+# Assumes google/mediapipe and google/flutter-mediapipe are siblings on the file system
+compile_text_classifier_macos_arm:
+	cd ../mediapipe && bazel build --linkopt -s --config darwin_arm64 --strip always --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/tasks/c/text/text_classifier:libtext_classifier.dylib
+	cd ../mediapipe && sudo cp bazel-bin/mediapipe/tasks/c/text/text_classifier/libtext_classifier.dylib ../flutter-mediapipe/packages/mediapipe-task-text/example/assets/libtext_classifier_arm64.dylib
+
+compile_text_classifier_macos_x86:
+	cd ../mediapipe && bazel build --linkopt -s --config darwin_x86_64 --strip always --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/tasks/c/text/text_classifier:libtext_classifier.dylib
+	cd ../mediapipe && sudo cp bazel-bin/mediapipe/tasks/c/text/text_classifier/libtext_classifier.dylib ../flutter-mediapipe/packages/mediapipe-task-text/example/assets/libtext_classifier_x64.dylib
+
+# Runs `sdks_finder` to update manifest files
+sdks:
+	dart tool/builder/bin/main.dart sdks
+
+analyze:
+	cd packages/mediapipe-core && dart format -o write .
+	cd packages/mediapipe-task-text && dart format -o write .
+
+# Core ---
+
+# Runs `ffigen` for `mediapipe_core`
+generate_core:
+	cd packages/mediapipe-core && dart run ffigen --config=ffigen.yaml
+
+# Runs unit tests for `mediapipe_core`
+test_core:
+	cd packages/mediapipe-core && dart test
+
+# Text ---
+
+# Runs `ffigen` for `mediapipe_text`
+generate_text:
+	cd packages/mediapipe-task-text && dart --enable-experiment=native-assets run ffigen --config=ffigen.yaml
+
+# Runs all text tests
+test_text:
+	cd packages/mediapipe-task-text && flutter test
+	cd packages/mediapipe-task-text/example && flutter test
+
+example_text:
+	cd packages/mediapipe-task-text/example && flutter run -d macos
diff --git a/README.md b/README.md
@@ -1,3 +1,21 @@
 # Flutter-MediaPipe
 
 This repository will be home to the source code for the mediapipe_task_vision, mediapipe_task_audio, and mediapipe_task_text plugins for Flutter.
+
+## Releasing
+
+### Updating MediaPipe SDKs
+
+Anytime MediaPipe releases new versions of their SDKs, this package will need to be updated to incorporate those latest builds. SDK versions are pinned in the `sdk_downloads.json` files in each package, which are updated by running the following command from the root of the repository:
+
+```
+$ make sdks
+```
+
+The Google Cloud Storage bucket in question only gives read-list access to a specific list of Googlers' accounts, so this command must be run from such a Googler's corp machines.
+
+After this, create and merge a PR with the changes and then proceed to `Releasing to pub.dev`.
+
+### Releasing to pub.dev
+
+TODO
diff --git a/packages/analysis_options.yaml b/packages/analysis_options.yaml
@@ -0,0 +1,6 @@
+include: package:lints/recommended.yaml
+
+linter:
+  rules:
+    - public_member_api_docs # see https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#documentation-dartdocs-javadocs-etc
+
diff --git a/packages/mediapipe-core/.gitignore b/packages/mediapipe-core/.gitignore
@@ -0,0 +1,7 @@
+# https://dart.dev/guides/libraries/private-files
+# Created by `dart pub`
+.dart_tool/
+
+# Avoid committing pubspec.lock for library packages; see
+# https://dart.dev/guides/libraries/private-files#pubspeclock.
+pubspec.lock
diff --git a/packages/mediapipe-core/CHANGELOG.md b/packages/mediapipe-core/CHANGELOG.md
@@ -0,0 +1,3 @@
+## 0.0.1
+
+- Initial version.
diff --git a/packages/mediapipe-core/README.md b/packages/mediapipe-core/README.md
@@ -0,0 +1,27 @@
+# MediaPipe Core for Flutter
+
+![pub package](https://img.shields.io/pub/v/mediapipe_core)
+
+A Flutter plugin to use the MediaPipe Core API, which enables multiple Mediapipe tasks.
+
+To learn more about MediaPipe, please visit the [MediaPipe website](https://developers.google.com/mediapipe)
+
+## Getting Started
+
+To get started with MediaPipe, please [see the documentation](https://developers.google.com/mediapipe/solutions/guide).
+
+<!-- ASPIRATIONAL
+## Usage
+
+To use this plugin, please visit the [Core Usage documentation](https://github.com/google/flutter-mediapipe#Usage)
+-->
+
+## Issues and feedback
+
+Please file Flutter-MediaPipe specific issues, bugs, or feature requests in our [issue tracker](https://github.com/google/flutter-mediapipe/issues/new).
+
+Issues that are specific to Flutter can be filed in the [Flutter issue tracker](https://github.com/flutter/flutter/issues/new).
+
+To contribute a change to this plugin,
+please review our [contribution guide](https://github.com/google/flutter-mediapipe/blob/master/CONTRIBUTING.md)
+and open a [pull request](https://github.com/google/flutter-mediapipe/pulls).
diff --git a/packages/mediapipe-core/analysis_options.yaml b/packages/mediapipe-core/analysis_options.yaml
@@ -0,0 +1,9 @@
+include: ../analysis_options.yaml
+
+linter:
+  rules:
+    - public_member_api_docs # see https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#documentation-dartdocs-javadocs-etc
+
+analyzer:
+  exclude:
+    - "**/mediapipe_common_bindings.dart"
diff --git a/packages/mediapipe-core/ffigen.yaml b/packages/mediapipe-core/ffigen.yaml
@@ -0,0 +1,26 @@
+name: "MediaPipeCommonBindings"
+description: "Bindings for shared MediaPipe structs common across many tasks"
+output:
+  bindings: "lib/src/io/third_party/mediapipe/generated/mediapipe_common_bindings.dart"
+  symbol-file:
+    output: "package:mediapipe_core/generated/core_symbols.yaml"
+    import-path: "package:mediapipe_core/src/io/third_party/mediapipe/generated/mediapipe_common_bindings.dart"
+headers:
+  entry-points:
+    - "third_party/mediapipe/tasks/c/**.h"
+preamble: |
+  /* Copyright 2023 The MediaPipe Authors.
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+  ==============================================================================*/
+ffi-native:
diff --git a/packages/mediapipe-core/lib/generated/core_symbols.yaml b/packages/mediapipe-core/lib/generated/core_symbols.yaml
@@ -0,0 +1,46 @@
+format_version: 1.0.0
+files:
+  package:mediapipe_core/src/io/third_party/mediapipe/generated/mediapipe_common_bindings.dart:
+    used-config:
+      ffi-native: false
+    symbols:
+      c:@S@BaseOptions:
+        name: BaseOptions
+      c:@S@Categories:
+        name: Categories
+      c:@S@Category:
+        name: Category
+      c:@S@ClassificationResult:
+        name: ClassificationResult
+      c:@S@Classifications:
+        name: Classifications
+      c:@S@ClassifierOptions:
+        name: ClassifierOptions
+      c:@S@EmbedderOptions:
+        name: EmbedderOptions
+      c:@S@Embedding:
+        name: Embedding
+      c:@S@EmbeddingResult:
+        name: EmbeddingResult
+      c:@S@__darwin_pthread_handler_rec:
+        name: __darwin_pthread_handler_rec
+      c:@S@_opaque_pthread_attr_t:
+        name: _opaque_pthread_attr_t
+      c:@S@_opaque_pthread_cond_t:
+        name: _opaque_pthread_cond_t
+      c:@S@_opaque_pthread_condattr_t:
+        name: _opaque_pthread_condattr_t
+      c:@S@_opaque_pthread_mutex_t:
+        name: _opaque_pthread_mutex_t
+      c:@S@_opaque_pthread_mutexattr_t:
+        name: _opaque_pthread_mutexattr_t
+      c:@S@_opaque_pthread_once_t:
+        name: _opaque_pthread_once_t
+      c:@S@_opaque_pthread_rwlock_t:
+        name: _opaque_pthread_rwlock_t
+      c:@S@_opaque_pthread_rwlockattr_t:
+        name: _opaque_pthread_rwlockattr_t
+      c:@S@_opaque_pthread_t:
+        name: _opaque_pthread_t
+      c:@UA@__mbstate_t:
+        name: __mbstate_t
diff --git a/packages/mediapipe-core/lib/interface.dart b/packages/mediapipe-core/lib/interface.dart
@@ -0,0 +1,5 @@
+// Copyright 2014 The Flutter Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+export 'src/interface/interface.dart';
diff --git a/packages/mediapipe-core/lib/io.dart b/packages/mediapipe-core/lib/io.dart
@@ -0,0 +1,6 @@
+// Copyright 2014 The Flutter Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+export 'src/interface/containers.dart' show EmbeddingType;
+export 'src/io/mediapipe_core.dart';
diff --git a/packages/mediapipe-core/lib/mediapipe_core.dart b/packages/mediapipe-core/lib/mediapipe_core.dart
@@ -0,0 +1,13 @@
+// Copyright 2014 The Flutter Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+/// Package containing core dependencies for MediaPipe's text, vision, and
+/// audio-based tasks.
+library mediapipe_core;
+
+export 'src/extensions.dart';
+export 'src/interface/containers.dart' show EmbeddingType;
+export 'universal_mediapipe_core.dart'
+    if (dart.library.html) 'src/web/mediapipe_core.dart'
+    if (dart.library.io) 'src/io/mediapipe_core.dart';
diff --git a/packages/mediapipe-core/lib/src/extensions.dart b/packages/mediapipe-core/lib/src/extensions.dart
@@ -0,0 +1,14 @@
+// Copyright 2014 The Flutter Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+/// Shortens long strings for logging, printing, etc.
+extension DebuggableString on String {
+  /// Shortens the string to its first X characters, replacing the rest with an
+  /// ellipses and the total length. If the string is equal to or shorter than
+  /// the given length, then [shorten] is a no-op.
+  String shorten([int x = 10]) {
+    if (length <= x) return this;
+    return '${substring(0, x)}...[$length]';
+  }
+}