branch-4.1: [Feature](multimodal_embed) Support multimodal(file) embed - #66461
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
3 tasks
) Support image/video/audio embedding from json. Supported multimodal_embed AI provider : Gemini(image, video, audio), Qwen(image, video), Voyage(image, video), Jina(image) Supported authentication methods include IAM + EXTERNAL_ID / AK + SK ```text -- ima role, video mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/video/45944deac7d96c872f559f2ef94ea0a9.mp4", '> "content_type": "video/mp4", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS video_embed_size; +------------------+ | video_embed_size | +------------------+ | 2560 | +------------------+ -- ima role, video mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png", '> "content_type": "image/png", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS img_embed_size; +----------------+ | img_embed_size | +----------------+ | 2560 | +----------------+ -- gemini multimodal embed mysql> SELECT ARRAY_SIZE( -> EMBED( -> 'gemini_mul_embed', -> CAST('{ '> "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png", '> "content_type": "image/png", '> "provider": "AWS", '> "endpoint": "s3.us-east-1.amazonaws.com", '> "region": "us-east-1", '> "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test", '> "external_id": "1001" '> }' AS JSON) -> ) -> ) AS img_embed_size; +----------------+ | img_embed_size | +----------------+ | 3072 | +----------------+ -- ak, sk Doris> SELECT ARRAY_SIZE( -> EMBED( -> 'qwen_mul_embed', -> CAST('{ '> "uri": "s3://<bucket>/<key>/test_img.png", '> "content_type": "image/png", '> "endpoint": "cos.ap-hongkong.myqcloud.com", '> "region": "ap-hongkong", '> "ak": "<ak>", '> "sk": "<sk>" '> }' AS JSON) -> ) -> ) AS img_embed_size_ak_sk; +----------------------+ | img_embed_size_ak_sk | +----------------------+ | 2560 | +----------------------+ ```
linrrzqqq
force-pushed
the
pick-62147-branch-4.1
branch
from
August 5, 2026 12:23
fc4f189 to
85ee3c6
Compare
Collaborator
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Problem Summary: The framework's default NULL implementation unwraps Nullable arguments and executes AI functions for every input row. For partially NULL inputs, the nested placeholder values of NULL rows are still included in prompts and sent to external AI providers. This causes unnecessary remote requests and token consumption. It also requires special handling for embedding results to preserve the original row order without duplicating large embedding vectors. This PR: - Disables the framework's default NULL implementation for all `AIFunction` subclasses. - Determines the Nullable return type in the `AIFunction` base class. - Extracts nested prompt columns and merges argument null maps in the common AI execution path. - Skips NULL rows before building prompts or sending requests. - Restores NULL rows in the final result while preserving the original row order. - Handles both text and multimodal Nullable inputs for `EMBED`. - Expands embedding array offsets in place, avoiding a copy of the nested Float32 embedding data. - Returns a constant NULL column when all input rows are NULL. ### Release note Fix AI scalar functions to skip NULL input rows instead of sending their placeholder values to external AI providers.
linrrzqqq
force-pushed
the
pick-62147-branch-4.1
branch
from
August 6, 2026 02:09
85ee3c6 to
66f1783
Compare
Collaborator
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick: #62147 and #66242