asr-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://developer.download.nvidia.com/notebooks/dlsw-notebooks/rivaasrasr-basics/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# How do I use Riva ASR APIs with out-of-the-box models?\n",
    "\n",
    "This tutorial walks you through the basics of Riva Speech Skills ASR Services, specifically covering how to use Riva ASR APIs with out-of-the-box models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NVIDIA Riva Overview\n",
    "\n",
    "NVIDIA Riva is a GPU-accelerated SDK for building Speech AI applications that are customized for your use case and deliver real-time performance. <br/>\n",
    "Riva offers a rich set of speech and natural language understanding services such as:\n",
    "\n",
    "- Automated speech recognition (ASR)\n",
    "- Text-to-Speech synthesis (TTS)\n",
    "- A collection of natural language processing (NLP) services, such as named entity recognition (NER), punctuation, intent classification.\n",
    "\n",
    "In this tutorial, we will interact with the automated speech recognition (ASR) APIs.\n",
    "\n",
    "For more information about Riva, refer to the [Riva developer documentation](https://developer.nvidia.com/riva)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transcription with Riva ASR APIs\n",
    "\n",
    "ASR takes an audio stream or audio buffer as input and returns one or more text transcripts, along with additional optional metadata. Speech recognition in Riva is a GPU-accelerated compute pipeline, with optimized performance and accuracy.  \n",
    "Riva provides state-of-the-art OOTB (out-of-the-box) models and pipelines for multiple languages, like English, Spanish, German, Russian and Mandarin, that can be easily deployed with the Riva Quick Start Scripts. Riva also supports easy customization of the ASR pipeline, in various ways, to meet your specific needs.  \n",
    "Refer to the [Riva ASR documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html) for more information.  \n",
    "\n",
    "Now, let's generate the transcripts using Riva APIs, for some sample audio clips, with an OOTB pipeline, starting with English."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='updated_reqs_and_setup_for_EngASR'></a>\n",
    "#### Requirements and setup\n",
    "\n",
    "1. Start the Riva Speech Skills server.  \n",
    "Follow the instructions in the [Riva Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html#) to deploy OOTB ASR models on the Riva Speech Skills server before running this tutorial. By default, only the English models are deployed.  \n",
    "\n",
    "\n",
    "2. Install the Riva Client library.   \n",
    "Follow the steps in the [Requirements and setup for the Riva Client](https://github.com/nvidia-riva/tutorials#running-the-riva-client) to install the Riva Client library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import the Riva client libraries\n",
    "\n",
    "Let's import some of the required libraries, including the Riva Client libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import IPython.display as ipd\n",
    "import grpc\n",
    "\n",
    "import riva.client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a Riva client and connect to the Riva Speech API server\n",
    "\n",
    "The following URI assumes a local deployment of the Riva Speech API server is on the default port. In case the server deployment is on a different host or via a Helm chart on Kubernetes, use an appropriate URI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "auth = riva.client.Auth(uri='localhost:50051')\n",
    "\n",
    "riva_asr = riva.client.ASRService(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Offline recognition for English\n",
    "\n",
    "You can use Riva ASR in either streaming mode or offline mode. In streaming mode, a continuous stream of audio is captured and recognized, producing a stream of transcribed text. In offline mode, an audio clip of a set length is transcribed to text. <br> \n",
    "Let's look at an example showing offline ASR API usage for English:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a gRPC request to the Riva Speech API server\n",
    "Riva ASR API supports `.wav` files in pulse-code modulation (PCM) format; including `.alaw`, `.mulaw`, and `.flac` formats with single channel. \n",
    "\n",
    "Now, let's make a gRPC request to the Riva Speech server for ASR with a sample `.wav` file in offline mode. Start by loading the audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This example uses a .wav file with LINEAR_PCM encoding.\n",
    "# read in an audio file from local disk\n",
    "path = \"./audio_samples/en-US_sample.wav\"\n",
    "with io.open(path, 'rb') as fh:\n",
    "    content = fh.read()\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, create an audio `RecognizeRequest` object, setting the configuration parameters as required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up an offline/batch recognition request\n",
    "config = riva.client.RecognitionConfig()\n",
    "#req.config.encoding = ra.AudioEncoding.LINEAR_PCM    # Audio encoding can be detected from wav\n",
    "#req.config.sample_rate_hertz = 0                     # Sample rate can be detected from wav and resampled if needed\n",
    "config.language_code = \"en-US\"                    # Language code of the audio clip\n",
    "config.max_alternatives = 1                       # How many top-N hypotheses to return\n",
    "config.enable_automatic_punctuation = True        # Add punctuation when end of VAD detected\n",
    "config.audio_channel_count = 1                    # Mono channel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, submit the request to the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript:\", asr_best_transcript)\n",
    "\n",
    "print(\"\\n\\nFull Response Message:\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Understanding ASR API parameters\n",
    "\n",
    "Riva ASR supports a number of options while making a transcription request to the gRPC endpoint, as shown in the previous section. Let's learn more about these parameters:\n",
    "- `encoding` - Type of audio encoding of the input audio file. Supports (`LINEAR_PCM`, `FLAC`, `MULAW` or `ALAW`). Can be detected from audio file\n",
    "- `sample_rate_hertz` - Sampling rate of the input audio in Hz. Note that the sample rate can be detected automatically from the audio `.wav` file and resampled if needed, making this parameter optional.\n",
    "- `language_code` - Language of the input audio. \"en-US\" represents English (US). Other options include (`es-US`, `de-DE`, `ru-RU`, `zh-CN`). We will explore ASR for non-English languages in the next section.\n",
    "- `max_alternatives` - Determines the number of top alternative transcriptions to return\n",
    "- `enable_automatic_punctuation` - Adds a punctuation at the end of VAD (Voice Activity Detection).\n",
    "- `audio_channel_count` - Number of audio channels. Typical microphones have 1 audio channel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Offline recognition for non-English languages - Spanish example\n",
    "\n",
    "In the previous section, we went through the Riva API usage and understood the different parameters of the ASR API. Now, let's look at using the ASR APIs for non-English languages, like Spanish, in offline mode.\n",
    "\n",
    "Note that we can similarly run Riva ASR for German, Russian, and Mandarin by using their corresponding language codes. We will elaborate on this at the end of this section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='updated_reqs_and_setup_for_nonEngASR'></a>\n",
    "#### Requirements and Setup for Spanish ASR:\n",
    "\n",
    "The requirements and setup steps for non-English ASR (in this case Spanish ASR) is the almost the same as for English ASR. The only difference is before running inference on the Spanish audio, we need to first deploy the Spanish ASR pipeline on the Riva Speech Skills server. \n",
    "\n",
    "Note: The Riva Speech Skills server [Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html#), that we followed in the [Requirements and Setup section](#updated_reqs_and_setup_for_EngASR) above for English ASR, explains how to deploy only English models by default.  \n",
    "\n",
    "1. Start the Riva Speech Skills server, with the Spanish ASR pipeline.  \n",
    "    1.1. Navigate to the Quick Start Guide folder. You downloaded this folder in the [Requirements and setup section](#Requirements-and-setup) above.\n",
    "    \n",
    "    1.2. Run ``bash riva_stop.sh`` to shut down the running Riva Speech Skills server. If Riva Speech Skills server is not currently running, you can skip this step.   \n",
    "    \n",
    "    1.3. [OPTIONAL] Run ``bash riva_clean.sh`` to clean up previous local Riva installation. This stops and removes all Riva-related containers, as well as deletes the Docker volume or directory used to store model files. The Docker images can also be removed, however, you’ll be asked for confirmation before removal.\n",
    "    \n",
    "    1.4. Update the ``config.sh`` file: Update the ``language_code=(\"en-US\")`` line to include the Spanish (language code `\"es-US\"`) model according to the instructions above this line in the ``config.sh`` script.  \n",
    "    \n",
    "    1.5. Rerun ``bash riva_init.sh`` to download and initialize the Spanish models and pipeline. If you see any errors during this step, start again from step 1.3.  \n",
    "    \n",
    "    1.6. Rerun ``bash riva_start.sh`` to restart the Riva Speech Skills server.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a gRPC request to the Riva Speech API server\n",
    "Let's make a gRPC request to the Riva Speech server for ASR with a sample Spanish `.wav` file in offline mode.  \n",
    "\n",
    "Like before, start by loading the audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This example uses a .wav file with LINEAR_PCM encoding.\n",
    "# read in an audio file from local disk\n",
    "path = \"audio_samples/es-US_sample.wav\" #Link to the Spanish sample audio file\n",
    "with io.open(path, 'rb') as fh:\n",
    "    content = fh.read()\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with English, create an audio `RecognizeRequest` object, setting the configuration parameters as required. Notice that we have updated ``language_code`` of the request configuration to the Spanish language code (``\"es-US\"``)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up an offline/batch recognition request\n",
    "config = riva.client.RecognitionConfig()\n",
    "#req.config.encoding = ra.AudioEncoding.LINEAR_PCM    # Audio encoding can be detected from wav\n",
    "#req.config.sample_rate_hertz = 0                     # Sample rate can be detected from wav and resampled if needed\n",
    "config.language_code = \"es-US\"                    # Language code of the audio clip. Set to Spanish\n",
    "config.max_alternatives = 1                       # How many top-N hypotheses to return\n",
    "config.enable_automatic_punctuation = True        # Add punctuation when end of VAD detected\n",
    "config.audio_channel_count = 1                    # Mono channel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, submit the request to the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript:\", asr_best_transcript)\n",
    "\n",
    "print(\"\\n\\nFull Response Message:\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can similarly run Riva ASR for German, Russian, and Mandarin by setting their corresponding language codes (``de-DE``, ``ru-RU``, and ``zh-CN``) in the request configuration. Ensure that these pipelines are deployed on the Riva Speech Skills server as instructed in the [Requirements and Setup for Spanish ASR](#updated_reqs_and_setup_for_nonEngASR)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Go deeper into Riva capabilities\n",
    "\n",
    "Now that you have a basic introduction to the Riva ASR APIs, you can try:\n",
    "\n",
    "### Additional Riva tutorials\n",
    "\n",
    "Checkout more Riva tutorials [here](https://github.com/nvidia-riva/tutorials) to understand how to use some of the advanced features of Riva ASR, including customizing ASR for your specific needs.\n",
    "\n",
    "\n",
    "### Sample Applications\n",
    "\n",
    "Riva comes with various sample applications. They demonstrate how to use the APIs to build various applications. Refer to [Riva Sampple Apps](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/index.html) for more information.  \n",
    "\n",
    "\n",
    "### Additional Resources\n",
    "\n",
    "For more information about each of the Riva APIs and their functionalities, refer to the [documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/reference/protos/protos.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}