Language Detector task (#22)

* Adds mediapipe_core package (#11) * adds mediapipe_core package * adds makefile for all packages * fixes typo in Makefile * Apply suggestions from code review * Update Makefile * Apply suggestions from code review * updated generated code's location and license (for 3P status) * code review responses including: * comments * licensing * resolving testing nits * updated README * setup sharing of base analysis_options * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-core/lib/src/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * add free methods to core structs * code review updates to options * adds publish blocker * Add GitHub actions for CI/CD (#14) * adds CI/CD for mediapipe_core * add newlines * moves file into workflows dir * uncomment flutter doctor * testing PR config to run this now * added master and beta CI scripts * add executable permissions to CI scripts * adds ffiwrapper to ci/cd --------- Co-authored-by: Kate Lovett <katelovett@google.com> * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <katelovett@google.com> * [FFI] MediaPipe SDKs finder automation (#16) * adds sdks_finder command to builder utility * propagates changes to existing commands * adds manifest files generated by new sdks_finder command * updates in response to code review * Adds mediapipe_text package (#12) * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * Apply suggestions from code review * Adds example to `mediapipe-task-text` (#15) * initial commit of example * build file changes from `flutter pub get` * update main.dart * removes commented code * updates example for isolates design * Use `native-assets` to vendor MediaPipe SDK (#9) * adding bare structure for native assets * add MVP / first draft of build.dart * build.dart updates * update build.dart TODO: stream file * model memory troubleshooting * vendoring script tweak * remove development logging * removes pointless build method * Add utility to collect headers from google/mediapipe (#10) * adds cmd to pull header files from google/mediapipe * polish and missing parts from git surgery * More comments and touch ups * Apply suggestions from code review * moves build command into `tool/` directory and renames folder `build_cmd` -> `builder` * complete build_cmd -> builder rename * Update readme * Added licenses * Adds DownloadModelCommand --------- Co-authored-by: Kate Lovett <katelovett@google.com> * adds mediapipe_text package * Update .vscode/settings.json added newline * resync headers * regenerated core bindings * native assets troubleshooting this commit is broken * Removes redundant count field * update build.dart for correct bindings path * download text classification model for CI * better memory freeing in executor * added SafeArea to example * added CI to PRs into text package * ci tooling change * remove accidentally commited model * more CI shenanigans * lowers minimum Dart version for builder * added smoke test for text example * Added CI/CD for examples * More CI tweaks * entering "please work" territory * d'oh * trying more random stuff * one more time * it'd be funny if this helped * more print statements * enable reaching new print statements * more logging * see what's in build dir * another test * adding flutter config list * turn off fail-fast for beta and master * moar logs * way moar prints * moare things * commit rest of rename * moar whatevers * adds manifest files generated by new sdks_finder command * adds sdks_finder command to builder utility * propagates changes to existing commands * updates in response to code review * updates to build.dart and tests * add Android runtime * sdks_finder logging improvement for when build folders change names * refreshed symbols from google/mediapipe * cleanup * loosens closeness thresholds in integration tests * separate build commands for macos architectures * restores fail-fast setting to CI * removed stale logging statements from CI * removes accidentally committed lines * add formatting of sdk_downloads.dart for CI * fixes broken example test * code touch ups from @Piinks code review --------- Co-authored-by: Kate Lovett <katelovett@google.com> * added base Dart class ClassificationResult to consolidate results logic in `mediapipe_core` in doing so, removed meaningless TextClassifierResult.timestamp field * Update Makefile * Update Makefile * Update packages/mediapipe-task-text/build.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/lib/src/tasks/text_classification/text_classification_executor.dart Co-authored-by: Kate Lovett <katelovett@google.com> * code review responses * formatting * removes stale comment * adds Dart to Native converters, with tests * changes from code review * updates mediapipe-core to prepare for IO/web split * Improves memory management in core tests * updated ffigen / bindings * refactors text package for better memory management and eventual web/io split * removed stale test * cleanup on aisle COMMENTS * Comments and documentation improvements * Moved log statement * Removed native memory management helpers in favor of `free` extensions on pointers * Renamed abstract classes to have Base prefix * sorted out class constructors * moved `fake` constructor to default unnamed constructor * leaning on the fact that the native constructors will be hidden by conditional exports, reducing confusion * Improved docstrings explaining memory ownership * Convert lists to lazy iterable / generators * added missing licenses * Update packages/mediapipe-task-text/example/test/widgets_test.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/lib/main.dart Co-authored-by: Kate Lovett <katelovett@google.com> * completed return style change * CI troubleshooting * moved around debugging code * logging tweak * cat native-assets.yaml * moar logs * removed bad echo * fixed native-assets typo should be underscore! * Removes CI debugging statements --------- Co-authored-by: Kate Lovett <katelovett@google.com> * Native Assets CI fix (#20) * adds native assets debugging statements * Try only downloading target arch * Revert "adds native assets debugging statements" This reverts commit b2bc215. --------- Co-authored-by: Daco Harkes <dacoharkes@google.com> * Text Embedding task (#21) * updated and re-ran generators * added embedding concepts to mediapipe-core * fixed embedding header file and bindings * adds text embedding classes to text pkg * updates example with text embedding * removed dead file * added more embedding tests * added embedding model download to CI script * touch ups * Update packages/mediapipe-core/lib/src/io/containers.dart Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/.gitignore Co-authored-by: Kate Lovett <katelovett@google.com> * Update packages/mediapipe-task-text/example/lib/text_embedding_demo.dart Co-authored-by: Kate Lovett <katelovett@google.com> * moved worker dispose method to base class * docstring & comment improvements * throw exceptions in impossible code paths instead of returning null * class hierarchy improvements * fixed outdates tests * cleaned up dispose methods * various tidying * fixed deprecation warning * moves repeated widgets into helper method --------- Co-authored-by: Kate Lovett <katelovett@google.com> * updated and re-ran generators * fixed embedding header file and bindings * adds text embedding classes to text pkg * moved worker dispose method to base class * class hierarchy improvements * cleaned up dispose methods * initial commit of language detection task * finishes language detection impl * adds language detection demo * backfilling improvements to classification and embedding * adds language detection tests * add new model download to CI script * fixes stale classification widget test, adds language detection widget test * copied latest headers from mediapipe * Update packages/mediapipe-task-text/lib/src/io/tasks/language_detection/language_detector_executor.dart Co-authored-by: Kate Lovett <katelovett@google.com> * comments / documentation improvements --------- Co-authored-by: Kate Lovett <katelovett@google.com> Co-authored-by: Daco Harkes <dacoharkes@google.com>
google · May 12, 2024 · 4809ba1 · 4809ba1
1 parent 6d9c37a
commit 4809ba1
Show file tree

Hide file tree

Showing 31 changed files with 1,324 additions and 190 deletions.
diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@ headers:
 models:
 	cd tool/builder && dart bin/main.dart model -m textclassification
 	cd tool/builder && dart bin/main.dart model -m textembedding
-
+	cd tool/builder && dart bin/main.dart model -m languagedetection
 
 # Runs `ffigen` for all packages
 generate: generate_core generate_text

diff --git a/packages/mediapipe-core/lib/universal_mediapipe_core.dart b/packages/mediapipe-core/lib/universal_mediapipe_core.dart
@@ -62,7 +62,7 @@ class Classifications extends BaseClassifications {
   });
 
   @override
-  Iterable<BaseCategory> get categories => throw UnimplementedError();
+  Iterable<Category> get categories => throw UnimplementedError();
 
   @override
   int get headIndex => throw UnimplementedError();

diff --git a/...ediapipe-core/third_party/mediapipe/tasks/c/components/containers/classification_result.h b/...ediapipe-core/third_party/mediapipe/tasks/c/components/containers/classification_result.h
@@ -28,7 +28,6 @@ struct Classifications {
   // The array of predicted categories, usually sorted by descending scores,
   // e.g. from high to low probability.
   struct Category* categories;
-
   // The number of elements in the categories array.
   uint32_t categories_count;
 
@@ -58,7 +57,6 @@ struct ClassificationResult {
   // exceed the maximum size that the model can process: to solve this, the
   // input data is split into multiple chunks starting at different timestamps.
   int64_t timestamp_ms;
-
   // Specifies whether the timestamp contains a valid value.
   bool has_timestamp_ms;
 };

diff --git a/...s/mediapipe-core/third_party/mediapipe/tasks/c/components/processors/classifier_options.h b/...s/mediapipe-core/third_party/mediapipe/tasks/c/components/processors/classifier_options.h
@@ -41,15 +41,13 @@ struct ClassifierOptions {
   // category name is not in this set will be filtered out. Duplicate or unknown
   // category names are ignored. Mutually exclusive with category_denylist.
   const char** category_allowlist;
-
   // The number of elements in the category allowlist.
   uint32_t category_allowlist_count;
 
   // The denylist of category names. If non-empty, detection results whose
   // category name is in this set will be filtered out. Duplicate or unknown
   // category names are ignored. Mutually exclusive with category_allowlist.
   const char** category_denylist;
-
   // The number of elements in the category denylist.
   uint32_t category_denylist_count;
 };

diff --git a/packages/mediapipe-task-text/example/lib/enumerate.dart b/packages/mediapipe-task-text/example/lib/enumerate.dart
@@ -2,12 +2,34 @@
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
-extension Enumeratable<T> on List<T> {
-  Iterable<S> enumerate<S>(S Function(T, int) fn) sync* {
+extension EnumeratableList<T> on List<T> {
+  /// Invokes the callback on each element of the list, optionally stopping
+  /// after [max] (inclusive) invocations.
+  Iterable<S> enumerate<S>(S Function(T, int) fn, {int? max}) sync* {
     int count = 0;
     while (count < length) {
       yield fn(this[count], count);
       count++;
+
+      if (max != null && count >= max) {
+        return;
+      }
+    }
+  }
+}
+
+extension EnumeratableIterable<T> on Iterable<T> {
+  /// Invokes the callback on each element of the iterable, optionally stopping
+  /// after [max] (inclusive) invocations.
+  Iterable<S> enumerate<S>(S Function(T, int) fn, {int? max}) sync* {
+    int count = 0;
+    for (final T obj in this) {
+      yield fn(obj, count);
+      count++;
+
+      if (max != null && count >= max) {
+        return;
+      }
     }
   }
 }
diff --git a/packages/mediapipe-task-text/example/lib/language_detection_demo.dart b/packages/mediapipe-task-text/example/lib/language_detection_demo.dart
@@ -0,0 +1,158 @@
+// Copyright 2014 The Flutter Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+import 'dart:async';
+import 'dart:typed_data';
+import 'package:flutter/material.dart';
+import 'package:getwidget/getwidget.dart';
+import 'package:mediapipe_text/mediapipe_text.dart';
+import 'enumerate.dart';
+
+class LanguageDetectionDemo extends StatefulWidget {
+  const LanguageDetectionDemo({super.key, this.detector});
+
+  final LanguageDetector? detector;
+
+  @override
+  State<LanguageDetectionDemo> createState() => _LanguageDetectionDemoState();
+}
+
+class _LanguageDetectionDemoState extends State<LanguageDetectionDemo>
+    with AutomaticKeepAliveClientMixin<LanguageDetectionDemo> {
+  final TextEditingController _controller = TextEditingController();
+  final Completer<LanguageDetector> _completer = Completer<LanguageDetector>();
+  final results = <Widget>[];
+  String? _isProcessing;
+
+  @override
+  void initState() {
+    super.initState();
+    _controller.text = 'Quiero agua, por favor';
+    _initDetector();
+  }
+
+  Future<void> _initDetector() async {
+    if (widget.detector != null) {
+      return _completer.complete(widget.detector!);
+    }
+
+    ByteData? bytes = await DefaultAssetBundle.of(context)
+        .load('assets/language_detector.tflite');
+
+    final detector = LanguageDetector(
+      LanguageDetectorOptions.fromAssetBuffer(
+        bytes.buffer.asUint8List(),
+      ),
+    );
+    _completer.complete(detector);
+    bytes = null;
+  }
+
+  void _prepareForDetection() {
+    setState(() {
+      _isProcessing = _controller.text;
+      results.add(const CircularProgressIndicator.adaptive());
+    });
+  }
+
+  Future<void> _detect() async {
+    _prepareForDetection();
+    _completer.future.then((detector) async {
+      final result = await detector.detect(_controller.text);
+      _showDetectionResults(result);
+      result.dispose();
+    });
+  }
+
+  void _showDetectionResults(LanguageDetectorResult result) {
+    setState(
+      () {
+        results.last = Card(
+          key: Key('prediction-"$_isProcessing" ${results.length}'),
+          margin: const EdgeInsets.all(10),
+          child: Column(
+            children: [
+              Padding(
+                padding: const EdgeInsets.all(10),
+                child: Text(_isProcessing!),
+              ),
+              Padding(
+                padding: const EdgeInsets.all(10.0),
+                child: Wrap(
+                  children: <Widget>[
+                    ...result.predictions
+                        .enumerate<Widget>(
+                          (prediction, index) => _languagePrediction(
+                            prediction,
+                            predictionColors[index],
+                          ),
+                          // Take first 4 because the model spits out dozens of
+                          // astronomically low probability language predictions
+                          max: predictionColors.length,
+                        )
+                        .toList(),
+                  ],
+                ),
+              ),
+            ],
+          ),
+        );
+        _isProcessing = null;
+      },
+    );
+  }
+
+  static final predictionColors = <Color>[
+    Colors.blue[300]!,
+    Colors.orange[300]!,
+    Colors.green[300]!,
+    Colors.red[300]!,
+  ];
+
+  Widget _languagePrediction(LanguagePrediction prediction, Color color) {
+    return Padding(
+      padding: const EdgeInsets.only(right: 8),
+      child: GFButton(
+        onPressed: null,
+        text: '${prediction.languageCode} :: '
+            '${prediction.probability.roundTo(8)}',
+        shape: GFButtonShape.pills,
+        color: color,
+      ),
+    );
+  }
+
+  @override
+  Widget build(BuildContext context) {
+    super.build(context);
+    return Scaffold(
+      body: SafeArea(
+        child: Padding(
+          padding: const EdgeInsets.all(16.0),
+          child: SingleChildScrollView(
+            child: Column(
+              children: <Widget>[
+                TextField(controller: _controller),
+                ...results.reversed,
+              ],
+            ),
+          ),
+        ),
+      ),
+      floatingActionButton: FloatingActionButton(
+        onPressed:
+            _isProcessing != null && _controller.text != '' ? null : _detect,
+        child: const Icon(Icons.search),
+      ),
+    );
+  }
+
+  @override
+  bool get wantKeepAlive => true;
+}
+
+extension on double {
+  double roundTo(int decimalPlaces) =>
+      double.parse(toStringAsFixed(decimalPlaces));
+}
diff --git a/packages/mediapipe-task-text/example/lib/main.dart b/packages/mediapipe-task-text/example/lib/main.dart
@@ -1,4 +1,5 @@
 import 'package:flutter/material.dart';
+import 'language_detection_demo.dart';
 import 'logging.dart';
 import 'text_classification_demo.dart';
 import 'text_embedding_demo.dart';
@@ -31,7 +32,7 @@ class TextTaskPages extends StatefulWidget {
 class TextTaskPagesState extends State<TextTaskPages> {
   final PageController controller = PageController();
 
-  final titles = <String>['Classify', 'Embed'];
+  final titles = <String>['Classify', 'Embed', 'Detect Languages'];
   int titleIndex = 0;
 
   void switchToPage(int index) {
@@ -61,28 +62,39 @@ class TextTaskPagesState extends State<TextTaskPages> {
         children: const <Widget>[
           TextClassificationDemo(),
           TextEmbeddingDemo(),
+          LanguageDetectionDemo(),
         ],
       ),
-      bottomNavigationBar: ColoredBox(
-        color: Colors.blueGrey,
-        child: Row(
-          mainAxisAlignment: MainAxisAlignment.spaceEvenly,
-          children: <Widget>[
-            TextButton(
-              onPressed: () => switchToPage(0),
-              child: Text(
-                'Classify',
-                style: titleIndex == 0 ? activeTextStyle : inactiveTextStyle,
+      bottomNavigationBar: SizedBox(
+        height: 50,
+        child: ColoredBox(
+          color: Colors.blueGrey,
+          child: Row(
+            mainAxisAlignment: MainAxisAlignment.spaceEvenly,
+            children: <Widget>[
+              TextButton(
+                onPressed: () => switchToPage(0),
+                child: Text(
+                  'Classify',
+                  style: titleIndex == 0 ? activeTextStyle : inactiveTextStyle,
+                ),
               ),
-            ),
-            TextButton(
-              onPressed: () => switchToPage(1),
-              child: Text(
-                'Embed',
-                style: titleIndex == 1 ? activeTextStyle : inactiveTextStyle,
+              TextButton(
+                onPressed: () => switchToPage(1),
+                child: Text(
+                  'Embed',
+                  style: titleIndex == 1 ? activeTextStyle : inactiveTextStyle,
+                ),
               ),
-            ),
-          ],
+              TextButton(
+                onPressed: () => switchToPage(2),
+                child: Text(
+                  'Detect Languages',
+                  style: titleIndex == 2 ? activeTextStyle : inactiveTextStyle,
+                ),
+              ),
+            ],
+          ),
         ),
       ),
     );