feat: text embeddings #163

jakmro · 2025-03-27T17:21:15Z

Description

Add text embeddings

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update (improves or adds clarity to existing documentation)

Tested on

iOS
Android

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

chmjkb · 2025-03-31T13:29:56Z

android/src/main/java/com/swmansion/rnexecutorch/models/TextEmbeddings/TextEmbeddingsModel.kt

+  fun preprocess(input: String): Array<LongArray> {
+    val inputIds = tokenizer.encode(input).map { it.toLong() }.toLongArray()
+    val attentionMask = inputIds.map { if (it != 0L) 1L else 0L }.toLongArray()
+    return arrayOf(inputIds, attentionMask) // Shape: [2, max_length]


Where is max_length specified? I think mentioning it here would be nice

max_length is specified inside tokenizer.json

chmjkb · 2025-03-31T13:39:33Z

src/hooks/natural_language_processing/useTextEmbeddings.ts

+  modelSource: string | number;
+  tokenizerSource: string | number;


use ResourceSource instead of string | number

src/hooks/natural_language_processing/useTextEmbeddings.ts

src/modules/natural_language_processing/TextEmbeddingsModule.ts

mkopcins · 2025-04-01T11:50:31Z

ios/RnExecutorch/models/BaseModel.mm

+- (NSArray *)forwardMultiple:(NSArray *)inputs {
+  NSMutableArray *shapes = [NSMutableArray new];
+  NSMutableArray *inputTypes = [NSMutableArray new];
+  NSNumber *numberOfInputs = [module getNumberOfInputs];
+
+  for (NSUInteger i = 0; i < [numberOfInputs intValue]; i++) {
+    [shapes addObject:[module getInputShape:[NSNumber numberWithInt:i]]];
+    [inputTypes addObject:[module getInputType:[NSNumber numberWithInt:i]]];
+  }
+
+  NSArray *result = [module forward:inputs shapes:shapes inputTypes:inputTypes];
+


This does not look correct, wouldn't it be better to actually check if length on inputs is the same as numberOfInputs and if not abort? Then we don't need to create another method just to copy 99% of the code. You can either add if sattement here checking if input is array of arrays or change it to only work with array of inputs

and then change the name

This will be changed in another PR as mentioned in BaseModel.h. This change requires changes in multiple files and I decided to not include them here.

ios/RnExecutorch/models/text_embeddings/TextEmbeddingsModel.mm

examples/text-embeddings/App.tsx

src/hooks/natural_language_processing/useTextEmbeddings.ts

mkopcins · 2025-04-01T12:30:19Z

src/native/RnExecutorchModules.ts

+const TextEmbeddingsSpec = require('./NativeTextEmbeddings').default;
+const TextEmbeddings = TextEmbeddingsSpec
+  ? TextEmbeddingsSpec
+  : new Proxy(
+      {},
+      {
+        get() {
+          throw new Error(LINKING_ERROR);
+        },
+      }
+    );
+


add class for this like for other modules, e.g.

class _ClassificationModule { async forward(input: string): ReturnType<ClassificationInterface['forward']> { return await Classification.forward(input); } async loadModule( modelSource: string | number ): ReturnType<ClassificationInterface['loadModule']> { return await Classification.loadModule(modelSource); } }

src/modules/natural_language_processing/TextEmbeddingsModule.ts

mkopcins · 2025-04-01T13:12:32Z

examples/text-embeddings/package.json

+  "devDependencies": {
+    "@babel/core": "^7.25.2",
+    "@types/react": "~18.3.12",
+    "react-native-executorch": "file:../../react-native-executorch-20250401133031.tgz",


This will be cleaned up once v0.4.0 will be released on npm

mkopcins · 2025-04-10T13:37:20Z

docs/docs/hookless-api/TextEmbeddingsModule.md

+title: TextEmbeddingsModule
+sidebar_position: 8
+---
+


Also add the description and the caution banner like in useTextEmbeddings page

Co-authored-by: Mateusz Kopcinski <120639731+mkopcins@users.noreply.github.com>

Remove Llama Export, refactor docs - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings --------- Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>

Modified forward function inside BaseModel to accept multiple inputs. Removed unnecessary includes/imports. Made the model loading synchronous. - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) - [x] iOS - [ ] Android - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings

Code refactor (example) - [x] iOS - [x] Android - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings --------- Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com>

Add tokenizer documentation - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings --------- Co-authored-by: kopcion <mati3111@gmail.com>

## Description Add text embeddings ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [x] iOS - [x] Android ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings --------- Co-authored-by: Mateusz Kopcinski <120639731+mkopcins@users.noreply.github.com> Co-authored-by: Jakub Chmura <92989966+chmjkb@users.noreply.github.com> Co-authored-by: kopcion <mati3111@gmail.com>

jakmro force-pushed the @jakmro/text-embeddings-ios branch 3 times, most recently from 4cf7634 to 11bff0f Compare March 28, 2025 11:50

jakmro changed the title ~~feat: text embeddings ios~~ feat: text embeddings Mar 28, 2025

jakmro force-pushed the @jakmro/text-embeddings-ios branch from 11bff0f to d34a6bd Compare March 28, 2025 11:51

jakmro requested review from chmjkb and mkopcins March 31, 2025 10:39

jakmro marked this pull request as ready for review March 31, 2025 10:39

chmjkb reviewed Mar 31, 2025

View reviewed changes

mkopcins requested changes Apr 1, 2025

View reviewed changes

mkopcins reviewed Apr 1, 2025

View reviewed changes

mkopcins requested changes Apr 10, 2025

View reviewed changes

mkopcins approved these changes Apr 16, 2025

View reviewed changes

jakmro force-pushed the @jakmro/text-embeddings-ios branch 2 times, most recently from 06125c3 to a43a3ba Compare April 17, 2025 12:40

pweglik mentioned this pull request Apr 22, 2025

feat: support tokenizer config #181

Merged

10 tasks

jakmro self-assigned this Apr 23, 2025

jakmro and others added 13 commits April 23, 2025 12:40

Add text embedding model (iOS)

cc85723

Add text embedding model (Android)

c60bf18

Add useTextEmbeddings hook

5dac0bb

Code improvements

bfbba7c

Fix download progress calculation

28f414b

Format code

014bbd7

Add text embeddings URLs and change types

83fd07c

Add text embeddings demo app

f2ad795

Update ios/RnExecutorch/models/text_embeddings/TextEmbeddingsModel.mm

1af1680

Co-authored-by: Mateusz Kopcinski <120639731+mkopcins@users.noreply.github.com>

Add text embeddings docs and benchmarks

eb92c35

Add model status information and fix styling

83f1a1a

jakmro and others added 3 commits April 23, 2025 12:45

Repair S2T

568007b

jakmro force-pushed the @jakmro/text-embeddings-ios branch from f0e49d6 to 568007b Compare April 23, 2025 11:04

jakmro added 3 commits April 23, 2025 16:11

Repair S2T v2

8c3340f

Repair S2T v3

4018133

Fix android LLM imports

297e839

jakmro merged commit deb768a into v0.4.0-rc1 Apr 23, 2025
1 check passed

jakmro deleted the @jakmro/text-embeddings-ios branch April 23, 2025 15:36

		modelSource: string \| number;
		tokenizerSource: string \| number;

feat: text embeddings #163

feat: text embeddings #163

Uh oh!

Conversation

jakmro commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tested on

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakmro commented Mar 27, 2025 •

edited

Loading