Feature/whisper

[mcrensh]

Issues:

• Create a Whisper component for speech-to-text and and translation openmpf#1598

---

This change is 

[hhuangMITRE]

Reviewed 11 of 12 files at r1, all commit messages.
Reviewable status: 11 of 12 files reviewed, 11 unresolved discussions (waiting on @mcrensh)

---

python/WhisperDetection/README.md line 11 at r1 (raw file):

# Input Properties
- `WHISPER_MODEL_SIZE`: Size of the Whisper model. Whisper has tiny, base, small, medium, and large models available.

Include a quick note about model limits for English-only models.
Also include info if English-only models are not meant for translation or language-ID tasks.

Lastly, include some info about the odd properties of Whisper handling Multi-Lingual situations:

I think for simplicity here, we can include a copy of your investigation and link it to the README, we can check with Jeff later if that works. There are useful details from the investigation about this component and it would be beneficial for any users as well.

PLEASE NOTE: I have already created a Markdown converted version of your investigation and linked it in our shared repo: it's right next to your original Whisper investigation doc, except with a markdown .md extension.

Please feel free to add a copy of that and link to README. If you do use it, 'also include the source/citation for the text inputs`.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 24 at r1 (raw file):

        {
          "name": "WHISPER_MODEL_LANG",
          "description": "Whisper has English-only models and multilingual models. Set to en for English-only models and multi for multilingual models.",

Add single quotation marks to highlight user-input options:

"Set to 'en' for English-only models and 'multi' for multilingual models."

I will also need to update my own Tika hotfix after this: I used backticks but during this review, I noticed that other descriptor files apply single quotes for these cases. These things generally are more noticeable from the reviewer side.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 30 at r1 (raw file):

        {
          "name": "WHISPER_MODEL_SIZE",
          "description": "Whisper models come in multiple sizes; tiny, base, small, medium, and large. Multilingual models are available in all 5 sizes, English-only is not available in large.",

Update description to:

Whisper models come in five sizes: 'tiny', 'base', 'small', 'medium', and 'large'. Multilingual models are available in all five sizes. English-only transcription models are not available in 'large' size, but are available in the other four sizes .

Feel free to adjust as needed if I misquoted something. I also assume English-only models are only good for transcription. The notes you provided on Whisper Multi-Lingual Audio mentioned English multimodal models, so I assume there is a difference here.

Please also mention the model size limits and limits of English-only models in the README.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 36 at r1 (raw file):

        {
          "name": "WHISPER_MODE",
          "description": "Set to 0 for spoken language detection, 1 for speech-to-text transcription, and 2 for speech translation.",

Mark '0', '1', and '2'.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 42 at r1 (raw file):

        {
          "name": "AUDIO_LANGUAGE",
          "description": "Language audio to transcribe or translate is in. If none is provided then Whisper will automatically detect language from the first 30 seconds of audio.",

Consider rewording to:

"Optional property that indicates the language to use for audio translation or transcription. If left as an empty string, Whisper will automatically detect a single language from the first 30 seconds of audio."

Please update the revised comment ("will automatically detect a single language") if Whisper can actually detect multiple languages.

I recalled it cannot based on our previous discussion and your notes on Whisper for multi-lingual audio showed that it auto translates/transcribes for a single language.

---

python/WhisperDetection/tests/test_whisper_detection.py line 33 at r1 (raw file):

import warnings
from whisper_detection_component import WhisperDetectionComponent

Leaving a note to myself to test this out later.

---

python/WhisperDetection/tests/test_whisper_detection.py line 107 at r1 (raw file):

    def test_transcribe_given_language(self):
        expected_text = (

I really like this text sample, could you add a quick site if it came from a source? Thanks!

---

python/WhisperDetection/tests/test_whisper_detection.py line 142 at r1 (raw file):

    def test_translation(self):
        expected_text = (
            'I communicate with different people in English and Spanish or mixing both languages. '

Could you add a quick site if it came from a source? Thanks!

---

python/WhisperDetection/tests/test_whisper_detection.py line 163 at r1 (raw file):

        job_props = dict(WHISPER_MODE=2, AUDIO_LANGUAGE='es')
        job = mpf.AudioJob('test', self._get_test_file('bilingual.mp3'), 0, -1, job_props, {})

Add a quick citation on the audio samples in NOTICE, thanks!

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 40 at r1 (raw file):

class WhisperDetectionComponent:
    detection_type = 'SPEECH'

As Jeff noted, it's too bad we can't adjust this property for different cases, as Whisper also does translation. Just leaving an observation if anything changes in the future.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 88 at r1 (raw file):

                # Convert timestamps back to frames

                # track_start_frame = int(floor(fpms * track.start_time))

Are these commented lines still useful for debugging? We can keep it if that's the case. Otherwise, they would be removed.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 156 at r1 (raw file):

        if model_lang == "en" and model_size == "large":
            raise mpf.DetectionError.INVALID_PROPERTY.exception("Whisper does not have a large English model available.")

Please include a note of this in the README regarding Whisper English model size limits.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 169 at r1 (raw file):

        if mode == 0:
            if model_lang != "multi":
                raise mpf.DetectionError.INVALID_PROPERTY.exception("Whisper does not support language detection "

Also, add a note in the README explaining that English-only models cannot perform language detection.

I've created a markdown copy of your audio investigation, we should include that and link to README so users are aware of the multilingual quirks as well.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 187 at r1 (raw file):

            )

            track = mpf.AudioTrack(

Modes 0-2 seem to have similar code so I'm wondering if we can shorten it down. However this is a nitpick and we still need to test out Whisper further before the merge, so this is an optional change.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 251 at r1 (raw file):

        return properties

I will review this file again after further testing and discussion.
Right now Argos Translate is quite promising (compared to Azure) so I placed my attention there to merge in as a priority.

It's still good to have this component and the efforts to investigate it documented down in the meantime, so I added some notes of what we can include in the README.

[mcrensh]

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 24 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Add single quotation marks to highlight user-input options:

"Set to 'en' for English-only models and 'multi' for multilingual models."

I will also need to update my own Tika hotfix after this: I used backticks but during this review, I noticed that other descriptor files apply single quotes for these cases. These things generally are more noticeable from the reviewer side.

Added single quotes

[mcrensh]

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 30 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Update description to:

Whisper models come in five sizes: 'tiny', 'base', 'small', 'medium', and 'large'. Multilingual models are available in all five sizes. English-only transcription models are not available in 'large' size, but are available in the other four sizes .

Feel free to adjust as needed if I misquoted something. I also assume English-only models are only good for transcription. The notes you provided on Whisper Multi-Lingual Audio mentioned English multimodal models, so I assume there is a difference here.

Please also mention the model size limits and limits of English-only models in the README.

Added single quotes and edited README. There's only English and multilingual models, the expected behavior for the English-only model is it can only transcribe English audio, but my notes found it can also translate. However, I don't think that is something it should be able to do that, English-only models can't do language detection and transcribing with English-only models automatically sets the language to English. I looked through the GitHub and paper again real quick and they don't define what they mean by English-only, but I would assume it only accepts English audio and therefore translating to English would be a moot point.

[mcrensh]

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 36 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Mark '0', '1', and '2'.

Added single quotes

[mcrensh]

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 42 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Consider rewording to:

"Optional property that indicates the language to use for audio translation or transcription. If left as an empty string, Whisper will automatically detect a single language from the first 30 seconds of audio."

Please update the revised comment ("will automatically detect a single language") if Whisper can actually detect multiple languages.

I recalled it cannot based on our previous discussion and your notes on Whisper for multi-lingual audio showed that it auto translates/transcribes for a single language.

Yep, it will only transcribe/translate for a single language, and detects language from the first 30 seconds of audio.

[mcrensh]

python/WhisperDetection/tests/test_whisper_detection.py line 107 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

I really like this text sample, could you add a quick site if it came from a source? Thanks!

It's a transcription of bilingual.mp3, I've added a NOTICE to the test data folder.

[mcrensh]

python/WhisperDetection/tests/test_whisper_detection.py line 142 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Could you add a quick site if it came from a source? Thanks!

Added NOTICE

[mcrensh]

python/WhisperDetection/tests/test_whisper_detection.py line 163 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Add a quick citation on the audio samples in NOTICE, thanks!

Added NOTICE

[mcrensh]

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 88 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Are these commented lines still useful for debugging? We can keep it if that's the case. Otherwise, they would be removed.

Whisper doesn't break audio up into utterances so these lines were unnecessary, I've removed them.

[mcrensh]

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 156 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Please include a note of this in the README regarding Whisper English model size limits.

Added to README

[mcrensh]

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 169 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Also, add a note in the README explaining that English-only models cannot perform language detection.

I've created a markdown copy of your audio investigation, we should include that and link to README so users are aware of the multilingual quirks as well.

Added the markdown and linked it in the README.

[mcrensh]

python/WhisperDetection/README.md line 11 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Include a quick note about model limits for English-only models.
Also include info if English-only models are not meant for translation or language-ID tasks.

Lastly, include some info about the odd properties of Whisper handling Multi-Lingual situations:

I think for simplicity here, we can include a copy of your investigation and link it to the README, we can check with Jeff later if that works. There are useful details from the investigation about this component and it would be beneficial for any users as well.

PLEASE NOTE: I have already created a Markdown converted version of your investigation and linked it in our shared repo: it's right next to your original Whisper investigation doc, except with a markdown .md extension.

Please feel free to add a copy of that and link to README. If you do use it, 'also include the source/citation for the text inputs`.

Clarified English-only can only transcribe English audio, added your markdown and linked it in the README.

[hhuangMITRE]

Overall, looks good! I added some requested tooltip updates and an request for adding ISO-639-2 formatted language detection outputs (optional) + update to sample code. Aside from that, let me know moving forward if it's alright for me to make some quick edits as well to the component.

Reviewed 5 of 5 files at r2, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @mcrensh)

---

python/WhisperDetection/README.md line 11 at r1 (raw file):

Previously, mcrensh wrote…

Clarified English-only can only transcribe English audio, added your markdown and linked it in the README.

Thanks!

---

python/WhisperDetection/README.md line 12 at r2 (raw file):

# Input Properties
- `WHISPER_MODEL_SIZE`: Size of the Whisper model. Whisper has tiny, base, small, medium, and large models available for multilingual models. English-only models are available in tiny, base, small, and medium. 
- `WHISPER_MODE`: Determines whether Whisper will perform language detection, speech-to-text transcription, or speech translation. English-only models can only transcribe English audio. 

Please also copy over the description of 0,1, and 2 options for WHISPER_MODE.

---

python/WhisperDetection/README.md line 16 at r2 (raw file):

# Output Properties
- `DETECTED_LANGUAGE`: Language with the highest confidence value.

Please add a short list of language outputs provided by Whisper, in the same format returned by Whisper if possible.

See discussion below on perhaps adding an ISO_LANGUAGE property, that request is optional as this component already handles translation.

---

python/WhisperDetection/README.md line 17 at r2 (raw file):

# Output Properties
- `DETECTED_LANGUAGE`: Language with the highest confidence value.
- `DETECTED_LANGUAGE_CONFIDENCE`: The confidence value of the language with the highest confidence.

Can we also get descriptions of the following outputs?

TRANSCRIPT - for transcription runs.
TRANSLATED_AUDIO - for translated audio runs.

Please also mention that the output property changes if the user sets WHISPER_MODE to translate or transcribe, so that there is no confusion about what the component returns under different modes.

---

python/WhisperDetection/sample_whisper_detector.py line 10 at r2 (raw file):

# 52.227-14, Alt. IV (DEC 2007).                                            #
#                                                                           #
# Copyright 2022 The MITRE Corporation. All Rights Reserved.                #

Copyright is now 2023.

If it's alright, next time I can make this change myself and push in the change (with a note of what changed). Let me know if that works.

---

python/WhisperDetection/sample_whisper_detector.py line 36 at r2 (raw file):

def main():
    if len(sys.argv) != 2:
        sys.exit(f'Usage {sys.argv[0]} <audio_file>')

Can we also include an optional third parameter to set WHISPER_MODE for Transcription/Auto-Translate to English, and view those results too?

Thanks!

---

python/WhisperDetection/setup.cfg line 34 at r2 (raw file):

packages = whisper_detection_component
install_requires =
    mpf_component_api>=7.0

I believe our component API number has been updated, please review the latest python component for master/develop and update, thanks.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 24 at r1 (raw file):

Previously, mcrensh wrote…

Added single quotes

Thanks!

Moving forward, let me know if it's ok for me to make the edit and attach a note after to avoid feeling nitpicky. I don't want to cause extra hassle and it's no trouble on my end to make the updates, if that works.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 30 at r1 (raw file):

Previously, mcrensh wrote…

Added single quotes and edited README. There's only English and multilingual models, the expected behavior for the English-only model is it can only transcribe English audio, but my notes found it can also translate. However, I don't think that is something it should be able to do that, English-only models can't do language detection and transcribing with English-only models automatically sets the language to English. I looked through the GitHub and paper again real quick and they don't define what they mean by English-only, but I would assume it only accepts English audio and therefore translating to English would be a moot point.

Interesting, I agree that the additional unexpected feature is a bit odd, considering that they have specific translation models designed for this task. Perhaps it's something inherent to their models?

If their notes specify that these are English-only models, I agree with your assessment that it's a moot point to have translation available too. Again, seems odd...

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 42 at r1 (raw file):

Previously, mcrensh wrote…

Yep, it will only transcribe/translate for a single language, and detects language from the first 30 seconds of audio.

Got it, thanks!

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 3 at r2 (raw file):

{
  "componentName": "WhisperDetection",
  "componentVersion": "7.0",

This should be updated. Moving forward, if it's ok, I can update this as well to save trouble.

---

python/WhisperDetection/tests/test_whisper_detection.py line 107 at r1 (raw file):

Previously, mcrensh wrote…

It's a transcription of bilingual.mp3, I've added a NOTICE to the test data folder.

Got it, and I see that you've attributed the source from CC 3.0 site.

---

python/WhisperDetection/tests/test_whisper_detection.py line 142 at r1 (raw file):

Previously, mcrensh wrote…

Added NOTICE

Thank you!

---

python/WhisperDetection/tests/data/NOTICE line 10 at r2 (raw file):

Copyright 1999-2015 Carnegie Mellon University. All Rights Reserved. http://cmusphinx.sourceforge.net/
Source code: https://github.com/cmusphinx/sphinx4

Is the copyright below just for your test media or also for the Whisper component library?

If we are planning to land this as a non-proprietary component. Please do the following:

• Include a LICENSE file with the standard OpenMPF license.

• Attach this license below (if it applies to any code, otherwise, let's check with Jeff) and specify the general files that are associated with the license.

Feel free to use this format for additional licenses: https://github.com/openmpf/openmpf/blob/hotfix/tika-lang-feed-forward/LICENSE

I assume that Argos and Whisper both have their own LICENSE files. Please ensure that they are also included in the LICENSE info. If they use the same one as OpenMPF, we can just mention that instead.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 88 at r1 (raw file):

Previously, mcrensh wrote…

Whisper doesn't break audio up into utterances so these lines were unnecessary, I've removed them.

Got it, thank you.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 39 at r2 (raw file):

class WhisperDetectionComponent:
    detection_type = 'SPEECH'

As mentioned before, wish there was a way to adjust detection type as Whisper does both STT and translation. Just leaving a note here.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 168 at r2 (raw file):

                                                                    "models.")

            audio = whisper.load_audio(target_file)

From Brian's component:

        # Certain components that process videos and don't do tracking will create tracks with a
        # single detection. The track and detection will have the same set of properties, so
        # it is relatively likely we will run in to duplicate text.

He then uses a dictionary to hold past TEXT results and reuse them if the same tracks are encountered again during a job. Upon examining this code, I don't think we'll encounter the same issue for this component. The audio is processed per media submitted instead.

However, we can likely optimize our Argos translation service, as that component may run into multiple duplicate TEXT properties during submission. I'm leaving a note here just for awareness as I'm moving to reviewing that component next.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 179 at r2 (raw file):

            properties = dict(
                DETECTED_LANGUAGE=detected_language

To standardize our outputs, please consider adding an ISO_LANGUAGE property for any text results so that further processing can be applied, in ISO-639-2 format, just like the Argos component. This is an optional request.

Regardless, please mention in the README, which languages are supported/output by Whisper's detection feature as well as the format of the detected language (i.e. "English" vs "eng" vs "en").

I've found a list of supported languages here, but please double-check: https://github.com/openai/whisper

[hhuangMITRE]

Update: from discussion with Jeff.

1. Can we get language detected results from whisper modes 1 and 2?
2. Please support the ISO_LANGUAGE output property as well.

Reviewable status: all files reviewed, 10 unresolved discussions (waiting on @mcrensh)

---

python/WhisperDetection/README.md line 16 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Please add a short list of language outputs provided by Whisper, in the same format returned by Whisper if possible.

See discussion below on perhaps adding an ISO_LANGUAGE property, that request is optional as this component already handles translation.

As a follow-up, Jeff said it would be good if we can include this property. Please add it as a to-do task for this PR, thanks.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 202 at r2 (raw file):

            audio_tracks.append(track)
        elif mode == 2:

From discussion with Jeff: For transcription and translation, can we also retrieve the reported detected language? End users would like to know what the original language would be.

Jeff asked if this would be easy or difficult to retrieve from the transcription or translation modes.

If this info is not accessible from translation/transcription, please let us know. I assume the next best option might be to auto-run detect language first.

[hhuangMITRE]

Additional tasks:

Questions for the Whisper team:

• Can we get timestamps for the transcription?
• Can we run translation independently on raw text?
• Can we get the transcription output when we do STT + translation? (Currently, we need to run the STT part twice).

For the language property output by the component, we would like:

1. For Language Detection: Keep the DETECTED_LANGUAGE property as is.

2. For Transcription: Return a DECODED_LANGUAGE property for the STT step. The user may provide it, or it is found by the component.

3. Return an ISO_LANGUAGE property converting DETECTED_LANGUAGE and DECODED_LANGUAGE into standardized ISO-639-2 format.

Reviewable status: all files reviewed, 11 unresolved discussions (waiting on @mcrensh)

---

python/WhisperDetection/README.md line 16 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

As a follow-up, Jeff said it would be good if we can include this property. Please add it as a to-do task for this PR, thanks.

For clarification: "DETECTED_LANGUAGE" is kept for audio language ID

In transcription and translation: we will call this "DECODED_LANGUAGE" instead.
In both cases, we'll also convert to ISO_LANGUAGE for standardization of language outputs.

---

python/WhisperDetection/README.md line 17 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Can we also get descriptions of the following outputs?

TRANSCRIPT - for transcription runs.
TRANSLATED_AUDIO - for translated audio runs.

Please also mention that the output property changes if the user sets WHISPER_MODE to translate or transcribe, so that there is no confusion about what the component returns under different modes.

Correction: Mention that Translation returns both properties but Transcription only returns TRANSCRIPT.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 202 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

From discussion with Jeff: For transcription and translation, can we also retrieve the reported detected language? End users would like to know what the original language would be.

Jeff asked if this would be easy or difficult to retrieve from the transcription or translation modes.

If this info is not accessible from translation/transcription, please let us know. I assume the next best option might be to auto-run detect language first.

Update: retracted as translation/transcription should return a "DECODED_LANGUAGE" for their steps.

We'll reach out to the Whisper Team to see if this processing step could be optimized. Right now, we run transcription twice in order to get the native transcript + translated text for translation (mode=2) runs.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 236 at r2 (raw file):

            properties = dict(
                DETECTED_LANGUAGE=result['language'],
                TRANSCRIPT=result['text'].strip()

Preference would be to add an "ISO_LANGUAGE" and convert "DETECTED_LANGUAGE" -> "DECODED LANGUAGE" for transcription tasks.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 242 at r2 (raw file):

            result = self.model.transcribe(target_file, language=language)
            properties = dict(
                TRANSCRIPT=result['text'].strip()

Preference would be to add an "ISO_LANGUAGE" and convert "DETECTED_LANGUAGE" -> "DECODED LANGUAGE" for transcription tasks. In this case, the user provides a transcript language model to use, so we'll return that as the DECODED_LANGUAGE property.

[mcrensh]

Reviewable status: 5 of 15 files reviewed, 11 unresolved discussions (waiting on @hhuangMITRE and @mcrensh)

---

python/WhisperDetection/README.md line 12 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Please also copy over the description of 0,1, and 2 options for WHISPER_MODE.

Done.

---

python/WhisperDetection/README.md line 16 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

For clarification: "DETECTED_LANGUAGE" is kept for audio language ID

In transcription and translation: we will call this "DECODED_LANGUAGE" instead.
In both cases, we'll also convert to ISO_LANGUAGE for standardization of language outputs.

Switched transcription and translation to use decoded language, added iso language for both decoded and detected language.

---

python/WhisperDetection/README.md line 17 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Correction: Mention that Translation returns both properties but Transcription only returns TRANSCRIPT.

Done.

---

python/WhisperDetection/sample_whisper_detector.py line 10 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Copyright is now 2023.

If it's alright, next time I can make this change myself and push in the change (with a note of what changed). Let me know if that works.

Updated the copyright. Totally fine with you updating it too next time!

---

python/WhisperDetection/sample_whisper_detector.py line 36 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Can we also include an optional third parameter to set WHISPER_MODE for Transcription/Auto-Translate to English, and view those results too?

Thanks!

Added

---

python/WhisperDetection/setup.cfg line 34 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

I believe our component API number has been updated, please review the latest python component for master/develop and update, thanks.

updated API version

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 24 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Thanks!

Moving forward, let me know if it's ok for me to make the edit and attach a note after to avoid feeling nitpicky. I don't want to cause extra hassle and it's no trouble on my end to make the updates, if that works.

Sure! That works with me

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 30 at r1 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Interesting, I agree that the additional unexpected feature is a bit odd, considering that they have specific translation models designed for this task. Perhaps it's something inherent to their models?

If their notes specify that these are English-only models, I agree with your assessment that it's a moot point to have translation available too. Again, seems odd...

Yeah I agree it is odd. They do call them English-only models so the behavior doesn't make any sense to me

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 3 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

This should be updated. Moving forward, if it's ok, I can update this as well to save trouble.

Updated, I'm fine with you updating this moving forward

---

python/WhisperDetection/tests/data/NOTICE line 10 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Is the copyright below just for your test media or also for the Whisper component library?

If we are planning to land this as a non-proprietary component. Please do the following:

• Include a LICENSE file with the standard OpenMPF license.

• Attach this license below (if it applies to any code, otherwise, let's check with Jeff) and specify the general files that are associated with the license.

Feel free to use this format for additional licenses: https://github.com/openmpf/openmpf/blob/hotfix/tika-lang-feed-forward/LICENSE

I assume that Argos and Whisper both have their own LICENSE files. Please ensure that they are also included in the LICENSE info. If they use the same one as OpenMPF, we can just mention that instead.

Added LICENSE

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 179 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

To standardize our outputs, please consider adding an ISO_LANGUAGE property for any text results so that further processing can be applied, in ISO-639-2 format, just like the Argos component. This is an optional request.

Regardless, please mention in the README, which languages are supported/output by Whisper's detection feature as well as the format of the detected language (i.e. "English" vs "eng" vs "en").

I've found a list of supported languages here, but please double-check: https://github.com/openai/whisper

I couldn't find a definitive list of supported languages so I used the list you linked, added to the README

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 236 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Preference would be to add an "ISO_LANGUAGE" and convert "DETECTED_LANGUAGE" -> "DECODED LANGUAGE" for transcription tasks.

Added ISO_LANGAUGAE, switched to DECODED_LANGUAGE for transcription and translation tasks

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 242 at r2 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

Preference would be to add an "ISO_LANGUAGE" and convert "DETECTED_LANGUAGE" -> "DECODED LANGUAGE" for transcription tasks. In this case, the user provides a transcript language model to use, so we'll return that as the DECODED_LANGUAGE property.

Done

[hhuangMITRE]

I've completed my review of the component. Updated the license file to use the 2023 copyright. Updated the test script to check for ISO_LANGUAGE and updated API version for the setup.cfg file to 7.1

Reviewed 10 of 10 files at r3.
Reviewable status: 12 of 15 files reviewed, 1 unresolved discussion (waiting on @mcrensh)

---

python/WhisperDetection/Dockerfile line 33 at r3 (raw file):

FROM ${BUILD_REGISTRY}openmpf_python_executor_ssb:${BUILD_TAG}

RUN apt install git-all

I'm not sure why but this line is failing for my Docker builds. I will debug over the week and see if it's an issue on my end.

@mcrensh please double check that the docker build and unit test is working. I'll submit a build on Reviewable as well.

I've updated the unit tests to check for the ISO_LANGUAGE property. Please ensure that this test passes with the change or update the check as needed.

---

python/WhisperDetection/LICENSE line 2 at r3 (raw file):

/******************************************************************************
 * Copyright 2022 The MITRE Corporation                                       *

Fixing the copyright in an update to this PR.

---

python/WhisperDetection/README.md line 16 at r2 (raw file):

Previously, mcrensh wrote…

Switched transcription and translation to use decoded language, added iso language for both decoded and detected language.

Thanks, I've added a check to the test script for ISO_LANGUAGE

---

python/WhisperDetection/sample_whisper_detector.py line 10 at r2 (raw file):

Previously, mcrensh wrote…

Updated the copyright. Totally fine with you updating it too next time!

Ok great, I will update the copyright for the LICENSE file then.

---

python/WhisperDetection/setup.cfg line 34 at r2 (raw file):

Previously, mcrensh wrote…

updated API version

I didn't see the API version updated so I made the update.

Referencing other python .cfg files, the mpf_component_api is now 7.1

[hhuangMITRE]

Reviewed 2 of 3 files at r4.
Reviewable status: 14 of 15 files reviewed, 1 unresolved discussion (waiting on @mcrensh)

---

python/WhisperDetection/tests/test_whisper_detection.py line 55 at r5 (raw file):

        self.assertEqual(1, len(result))
        self.assertEqual('en', result[0].detection_properties['DETECTED_LANGUAGE'])
        self.assertEqual('eng', result[0].detection_properties['ISO_LANGUAGE'])

Running a new Jenkins build to check that tests pass and images are built properly.

[hhuangMITRE]

Resolved Docker build issues for Argos and Whisper during tests. @mcrensh please review and accept all changes.

Reviewable status: 11 of 15 files reviewed, 1 unresolved discussion (waiting on @brosenberg42 and @mcrensh)

---

python/WhisperDetection/Dockerfile line 36 at r3 (raw file):

RUN pip install --no-cache-dir git+https://github.com/openai/whisper.git

Your python setup file includes the necessary Whisper dependency so I've cleaned up this line.
Brian has noted that we also may want to download some of the Whisper models here as well. Leaving this open for Brian's review.

---

python/WhisperDetection/setup.cfg line 36 at r7 (raw file):

    mpf_component_api>=7.1
    mpf_component_util>=7.1
    openai-whisper==20230314

Swapped to python installation, with a specific tag.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 3 at r7 (raw file):

{
  "componentName": "WhisperDetection",
  "componentVersion": "7.1",

Fixed typo in descriptor.

---

python/WhisperDetection/tests/test_whisper_detection.py line 107 at r7 (raw file):

        self.assertEqual(1, len(result))
        self.assertEqual('en', result[0].detection_properties['DECODED_LANGUAGE'])
        self.assertEqual('eng', result[0].detection_properties['ISO_LANGUAGE'])

Added a quick check for ISO_LANGUAGE

---

python/WhisperDetection/tests/test_whisper_detection.py line 157 at r7 (raw file):

            'try to be, to not be different than I consider myself to be. As part of that community. '
            'I will mix, if I am working with people that are friends of mine, that are bilingual. '
            'Say the word in the language that comes easiest.'

Updated test as the Whisper team has apparently improved their models.

[brosenberg42]

Please create a PR to add this to https://github.com/openmpf/openmpf-docker/blob/master/docker-compose.components.yml

Reviewable status: 11 of 15 files reviewed, 16 unresolved discussions (waiting on @hhuangMITRE and @mcrensh)

---

python/WhisperDetection/Dockerfile line 32 at r6 (raw file):

ARG BUILD_TAG=latest
FROM ${BUILD_REGISTRY}openmpf_python_executor_ssb:${BUILD_TAG}

Add RUN pip install --no-cache-dir openai-whisper==20230314 'numpy<1.24,>=1.18' to prevent the dependencies from being downloaded every time the component code changes.

---

python/WhisperDetection/Dockerfile line 32 at r6 (raw file):

ARG BUILD_TAG=latest
FROM ${BUILD_REGISTRY}openmpf_python_executor_ssb:${BUILD_TAG}

After adding the pip install command I mentioned add lines like:

RUN python -c 'import whisper; whisper.load_model("base")'

RUN python -c 'import whisper; whisper.load_model("tiny")'

So the necessary models are downloaded at build time.

---

python/WhisperDetection/README.md line 23 at r7 (raw file):

# Behavior
Some quirks in Whisper's behavior when transcribing or translating audio with multiple languages has been observed. See [whisper_behavior_notes.md](whisper_behavior_notes.md) for more details.

The important information in whisper_behavior_notes.md is too spread out. We need a more concise and easily understandable description of the behavior. I think we should include the tables below. We should explain that the tables contain the results of running an audio file where the beginning was in Spanish and the end was in English. We should also say "See whisper_behavior_notes.md for more details and examples."

### Transcribe ###

Size  | Provided Language | Result for Spanish Part | Result for English Part
------|-------------------|-------------------------|-------------------------
base  | Auto-detected     | Correctly transcribed   | Gibberish
large | Auto-detected     | Correctly transcribed   | Translated to Spanish
base  | English           | Translated to English   | Correctly transcribed 
large | English           | Translated to English   | Correctly transcribed 




### Translate ###

Size  | Provided Language | Result for Spanish Part | Result for English Part
------|-------------------|-------------------------|------------------------
base  | Auto-detected     | Correctly translated    | Not included in output
base  | English           | Correctly translated    | Transcribed
large | Auto-detected     | Correctly translated    | Mostly skipped
large | English           | Correctly translated    | Mostly skipped

---

python/WhisperDetection/README.md line 30 at r7 (raw file):

All translations are to English.

| ISO-639-1 | ISO-639-2 | Language         |

Change the references to "ISO-639-2" to "ISO 639-3". In the other components, ISO_LANGUAGE is the ISO 639-3 language code. The language codes you have listed are the same in ISO 639-2 and ISO 639-3, so don't need to change any of the language codes you listed.

---

python/WhisperDetection/setup.cfg line 36 at r6 (raw file):

    mpf_component_api>=7.0
    mpf_component_util>=7.0
    openai-whisper==20230314

I needed to add numpy<1.24,>=1.18, otherwise numba will fail to initialize.

---

python/WhisperDetection/whisper_behavior_notes.md line 14 at r6 (raw file):

translate.

Example: First half of audio is in Spanish, `second half in English`

Instead of surrounding the text with back ticks use two asterisks to make it bold. For example: **bold text**

---

python/WhisperDetection/whisper_behavior_notes.md line 19 at r6 (raw file):

language) output:

-   \'Me comunico con diversas personas en inglés y en español o

There are a bunch of \' in this file. Remove the preceding back slashes.

---

python/WhisperDetection/whisper_behavior_notes.md line 1 at r7 (raw file):

Whisper Multi-Lingual Audio Investigation

Change this to: "# Whisper Multi-Lingual Behavior"

---

python/WhisperDetection/whisper_behavior_notes.md line 33 at r7 (raw file):

    su'`

-   `Highlighted part` is English audio, returns different results each

In these sections, I would put the bullets containing your observations and descriptions of the behavior before the output bullets.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 36 at r6 (raw file):

        {
          "name": "WHISPER_MODE",
          "description": "Set to '0' for spoken language detection, '1' for speech-to-text transcription, and '2' for speech translation.",

We should use strings here. For example, we could use LANGUAGE_DETECTION, TRANSCRIPTION, and SPEECH_TRANSLATION. That will make it easier for a user to understand what the property does when looking at a job or pipeline.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 59 at r6 (raw file):

        start_time = int(start_frame / fpms)

        if stop_frame is not None and stop_frame < media_frame_count - 1:

This should be if stop_frame > 0 and stop_frame < media_frame_count - 1:

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 238 at r7 (raw file):

            detected_lang_conf = probs[detected_language]

            if detected_language in self.iso_map:

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(detected_language, 'UNKNOWN')

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 239 at r7 (raw file):

            if detected_language in self.iso_map:
                iso_2 = self.iso_map[detected_language]

Please rename this variable iso_639_3. In the other components ISO_LANGUAGE is the ISO 639-3 language code. ISO 639-3 is a super-set of ISO 639-2, so the language codes in self.iso_map don't need to be changed.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 301 at r7 (raw file):

            result = self.model.transcribe(target_file)

            if result['language'] in self.iso_map:

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(result['language'], 'UNKNOWN')

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 314 at r7 (raw file):

        else:

            if language in self.iso_map:

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(language, 'UNKNOWN')

[mcrensh]

Reviewable status: 8 of 15 files reviewed, 16 unresolved discussions (waiting on @brosenberg42 and @hhuangMITRE)

---

python/WhisperDetection/Dockerfile line 33 at r3 (raw file):

Previously, hhuangMITRE (Howard W Huang) wrote…

I'm not sure why but this line is failing for my Docker builds. I will debug over the week and see if it's an issue on my end.

@mcrensh please double check that the docker build and unit test is working. I'll submit a build on Reviewable as well.

I've updated the unit tests to check for the ISO_LANGUAGE property. Please ensure that this test passes with the change or update the check as needed.

Done.

---

python/WhisperDetection/Dockerfile line 32 at r6 (raw file):

Previously, brosenberg42 wrote…

Add RUN pip install --no-cache-dir openai-whisper==20230314 'numpy<1.24,>=1.18' to prevent the dependencies from being downloaded every time the component code changes.

Done.

---

python/WhisperDetection/Dockerfile line 32 at r6 (raw file):

Previously, brosenberg42 wrote…

After adding the pip install command I mentioned add lines like:

RUN python -c 'import whisper; whisper.load_model("base")'

RUN python -c 'import whisper; whisper.load_model("tiny")'

So the necessary models are downloaded at build time.

Done.

---

python/WhisperDetection/README.md line 23 at r7 (raw file):

Previously, brosenberg42 wrote…

The important information in whisper_behavior_notes.md is too spread out. We need a more concise and easily understandable description of the behavior. I think we should include the tables below. We should explain that the tables contain the results of running an audio file where the beginning was in Spanish and the end was in English. We should also say "See whisper_behavior_notes.md for more details and examples."

### Transcribe ###

Size  | Provided Language | Result for Spanish Part | Result for English Part
------|-------------------|-------------------------|-------------------------
base  | Auto-detected     | Correctly transcribed   | Gibberish
large | Auto-detected     | Correctly transcribed   | Translated to Spanish
base  | English           | Translated to English   | Correctly transcribed 
large | English           | Translated to English   | Correctly transcribed 




### Translate ###

Size  | Provided Language | Result for Spanish Part | Result for English Part
------|-------------------|-------------------------|------------------------
base  | Auto-detected     | Correctly translated    | Not included in output
base  | English           | Correctly translated    | Transcribed
large | Auto-detected     | Correctly translated    | Mostly skipped
large | English           | Correctly translated    | Mostly skipped

Done.

Code quote:

Some quirks in Whisper's behavior when transcribing or translating audio with multiple languages has been observed.

---

python/WhisperDetection/README.md line 30 at r7 (raw file):

Previously, brosenberg42 wrote…

Change the references to "ISO-639-2" to "ISO 639-3". In the other components, ISO_LANGUAGE is the ISO 639-3 language code. The language codes you have listed are the same in ISO 639-2 and ISO 639-3, so don't need to change any of the language codes you listed.

Done.

---

python/WhisperDetection/setup.cfg line 36 at r6 (raw file):

Previously, brosenberg42 wrote…

I needed to add numpy<1.24,>=1.18, otherwise numba will fail to initialize.

Done.

---

python/WhisperDetection/whisper_behavior_notes.md line 14 at r6 (raw file):

Previously, brosenberg42 wrote…

Instead of surrounding the text with back ticks use two asterisks to make it bold. For example: **bold text**

Done.

---

python/WhisperDetection/whisper_behavior_notes.md line 19 at r6 (raw file):

Previously, brosenberg42 wrote…

There are a bunch of \' in this file. Remove the preceding back slashes.

Done.

---

python/WhisperDetection/whisper_behavior_notes.md line 1 at r7 (raw file):

Previously, brosenberg42 wrote…

Change this to: "# Whisper Multi-Lingual Behavior"

Done.

---

python/WhisperDetection/whisper_behavior_notes.md line 33 at r7 (raw file):

Previously, brosenberg42 wrote…

In these sections, I would put the bullets containing your observations and descriptions of the behavior before the output bullets.

Done.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 36 at r6 (raw file):

Previously, brosenberg42 wrote…

We should use strings here. For example, we could use LANGUAGE_DETECTION, TRANSCRIPTION, and SPEECH_TRANSLATION. That will make it easier for a user to understand what the property does when looking at a job or pipeline.

Done.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 59 at r6 (raw file):

Previously, brosenberg42 wrote…

This should be if stop_frame > 0 and stop_frame < media_frame_count - 1:

Done.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 238 at r7 (raw file):

Previously, brosenberg42 wrote…

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(detected_language, 'UNKNOWN')

Done.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 239 at r7 (raw file):

Previously, brosenberg42 wrote…

Please rename this variable iso_639_3. In the other components ISO_LANGUAGE is the ISO 639-3 language code. ISO 639-3 is a super-set of ISO 639-2, so the language codes in self.iso_map don't need to be changed.

Done.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 301 at r7 (raw file):

Previously, brosenberg42 wrote…

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(result['language'], 'UNKNOWN')

Done.

---

python/WhisperDetection/whisper_detection_component/whisper_detection_component.py line 314 at r7 (raw file):

Previously, brosenberg42 wrote…

AzureSpeech uses 'UNKNOWN' when it can't convert a language code. To be consistent with AzureSpeech, please change this to iso_639_3 = self.iso_map.get(language, 'UNKNOWN')

Done.

[brosenberg42]

Reviewable status: 8 of 15 files reviewed, 5 unresolved discussions (waiting on @hhuangMITRE and @mcrensh)

---

python/WhisperDetection/setup.cfg line 29 at r9 (raw file):

[metadata]
name = WhisperDetection
version = 0.1

Please change this to version = 7.2

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 2 at r9 (raw file):

{
  "componentName": "WhisperDetection",

In today's meeting, we were discussing what the name of the component should be. We decided that it should be called "WhisperSpeechDetection".

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 3 at r9 (raw file):

{
  "componentName": "WhisperDetection",
  "componentVersion": "7.1",

Change this to 7.2

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 4 at r9 (raw file):

  "componentName": "WhisperDetection",
  "componentVersion": "7.1",
  "middlewareVersion": "7.1",

Change this to 7.2

[mcrensh]

Reviewable status: 0 of 15 files reviewed, 5 unresolved discussions (waiting on @brosenberg42 and @hhuangMITRE)

---

python/WhisperDetection/setup.cfg line 29 at r9 (raw file):

Previously, brosenberg42 wrote…

Please change this to version = 7.2

Done.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 2 at r9 (raw file):

Previously, brosenberg42 wrote…

In today's meeting, we were discussing what the name of the component should be. We decided that it should be called "WhisperSpeechDetection".

Done.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 3 at r9 (raw file):

Previously, brosenberg42 wrote…

Change this to 7.2

Done.

---

python/WhisperDetection/plugin-files/descriptor/descriptor.json line 4 at r9 (raw file):

Previously, brosenberg42 wrote…

Change this to 7.2

Done.

[brosenberg42]

Reviewed 11 of 15 files at r10, 3 of 3 files at r11.
Reviewable status: 14 of 15 files reviewed, 1 unresolved discussion (waiting on @hhuangMITRE)

[hhuangMITRE]

Reviewable status: 14 of 15 files reviewed, all discussions resolved (waiting on @mcrensh)