Skip to content

feat: Replace ML.GENERATE_TEXT with AI.GENERATE for audio transcription #2151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 9, 2025
Merged

feat: Replace ML.GENERATE_TEXT with AI.GENERATE for audio transcription #2151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1316272
change to ai.generate
shuoweil Oct 7, 2025
ebea037
convert the input data type
shuoweil Oct 8, 2025
9179103
remove default value setting
shuoweil Oct 9, 2025
084cfe9
add audio_transcribe testcase to notebook
shuoweil Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 11 additions & 17 deletions bigframes/operations/blob.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -804,35 +804,29 @@ def audio_transcribe(
raise ValueError("Must specify the engine, supported value is 'bigquery'.")

import bigframes.bigquery as bbq
import bigframes.ml.llm as llm
import bigframes.pandas as bpd

# col name doesn't matter here. Rename to avoid column name conflicts
audio_series = bigframes.series.Series(self._block)

prompt_text = "**Task:** Transcribe the provided audio. **Instructions:** - Your response must contain only the verbatim transcription of the audio. - Do not include any introductory text, summaries, or conversational filler in your response. The output should begin directly with the first word of the audio."

llm_model = llm.GeminiTextGenerator(
model_name=model_name,
session=self._block.session,
connection_name=connection,
)
# Convert the audio series to the runtime representation required by the model.
audio_runtime = audio_series.blob._get_runtime("R", with_metadata=True)

# transcribe audio using ML.GENERATE_TEXT
transcribed_results = llm_model.predict(
X=audio_series,
prompt=[prompt_text, audio_series],
temperature=0.0,
transcribed_results = bbq.ai.generate(
prompt=(prompt_text, audio_runtime),
connection_id=connection,
endpoint=model_name,
model_params={"generationConfig": {"temperature": 0.0}},
)

transcribed_content_series = cast(
bpd.Series, transcribed_results["ml_generate_text_llm_result"]
).rename("transcribed_content")
transcribed_content_series = transcribed_results.struct.field("result").rename(
"transcribed_content"
)

if verbose:
transcribed_status_series = cast(
bpd.Series, transcribed_results["ml_generate_text_status"]
)
transcribed_status_series = transcribed_results.struct.field("status")
results_df = bpd.DataFrame(
{
"status": transcribed_status_series,
Expand Down
Loading