Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: 'tokens' #19

Open
syedmustafa54 opened this issue Apr 22, 2023 · 14 comments · Fixed by #39
Open

KeyError: 'tokens' #19

syedmustafa54 opened this issue Apr 22, 2023 · 14 comments · Fixed by #39

Comments

@syedmustafa54
Copy link

Hello @sanchit-gandhi Thanks for sharing this repo.
I installed all the dependencies and ran this command in terminal 1 bash launch_app.sh
in terminal 2 I ran API_URL=http://0.0.0.0:8000/generate/ API_URL_FROM_FEATURES=http://0.0.0.0:8000/gnerate_from_features/ python app.py
when I select you tube url getting this error

File "/home/ubuntu/whisper-jax/app/app.py", line 72, in forward
    outputs["tokens"] = np.asarray(outputs["tokens"])
KeyError: 'tokens'

Complete error

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/ubuntu/whisper-jax/app/app.py", line 185, in transcribe_youtube
    text, runtime = tqdm_generate(inputs, task=task, return_timestamps=return_timestamps, progress=progress)
  File "/home/ubuntu/whisper-jax/app/app.py", line 126, in tqdm_generate
    model_outputs.append(forward(batch, task=task, return_timestamps=return_timestamps))
  File "/home/ubuntu/whisper-jax/app/app.py", line 72, in forward
    outputs["tokens"] = np.asarray(outputs["tokens"])
KeyError: 'tokens'
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ubuntu/anaconda3/envs/whisper/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/ubuntu/whisper-jax/app/app.py", line 185, in transcribe_youtube
    text, runtime = tqdm_generate(inputs, task=task, return_timestamps=return_timestamps, progress=progress)
  File "/home/ubuntu/whisper-jax/app/app.py", line 126, in tqdm_generate
    model_outputs.append(forward(batch, task=task, return_timestamps=return_timestamps))
  File "/home/ubuntu/whisper-jax/app/app.py", line 72, in forward
    outputs["tokens"] = np.asarray(outputs["tokens"])
KeyError: 'tokens'
@sanchit-gandhi
Copy link
Owner

Is port 8000 the right port? Usually gradio defaults to port 7860 - could you check with:

API_URL=http://0.0.0.0:7860/generate/ API_URL_FROM_FEATURES=http://0.0.0.0:7860/generate_from_features/ python app.py

(also generate_from_features was spelled, wrong, which could have been the culprit)

Either way, I agree that we should have a better error message! Will open a PR for this

@sanchit-gandhi sanchit-gandhi mentioned this issue Apr 24, 2023
Merged
@sanchit-gandhi
Copy link
Owner

Using the main branch now should give you a more detailed Gradio error message @syedmustafa54!

@syedmustafa54
Copy link
Author

syedmustafa54 commented Apr 24, 2023 •
edited

@sanchit-gandhi Thanks for the update but now I am facing a different issue on GPU it's not giving output its stuck I have modified demo.launch(server_name="0.0.0.0",server_port=9504, show_api=False) and just ran this command python direct_app.py


To create a public link, set `share=True` in `launch()`.
2023-04-24 14:51:03;INFO;Loading audio file...
2023-04-24 14:51:04;INFO;Pre-processing audio file...

its stuck for long time there

@syedmustafa54
Copy link
Author

lly gradio defaults to port 7860

port 8000 is for sh launch_app.sh and for app.py gradio its 9504 I have modified it

@sanchit-gandhi
Copy link
Owner

Cool! The model is probably JIT compiling (which takes a few minutes the first time you run it, but will be fast thereafter). See https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage for details.

@syedmustafa54
Copy link
Author

syedmustafa54 commented Apr 24, 2023 •
edited

@sanchit-gandhi any guess on T4 I have waited for 25-30 mins but still no results is this normal?

@sanchit-gandhi
Copy link
Owner

Which checkpoint are you using? And how long is the audio file? If it's stuck on the pre-processing stage then it might be that you're maxed out on CPU resource for pre-processing the audio file.

@syedmustafa54
Copy link
Author

I just ran direct_app.py. the audio file is 3 sec. I have checked the CPU usage its only 1 process is using 100% rest is at 0%

@sanchit-gandhi
Copy link
Owner

Hey @syedmustafa54 - could you first check that the Flax Whisper Pipeline works standalone with this audio file, i.e. not as a gradio app? This will give you better error logs for any environment / resource errors. You can follow the instructions from https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage

@syedmustafa54
Copy link
Author

Hey I tried to run the below code by adding audio.mp3 in the folder but still its not giving any error just not showing any output

from whisper_jax import FlaxWhisperPipline

# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")

# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")

# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")```

@syedmustafa54
Copy link
Author

syedmustafa54 commented Apr 25, 2023 •
edited

Hello @sanchit-gandhi I have started a new Instance install cuda-11.8 cudnn-8.8 and install all the requirements
and ran the below code

from whisper_jax import FlaxWhisperPipline

# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")

# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")

# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")
print(text)

Now this code is working fine its taking 1.5 mins to give the output on 3 sec audio on T4 GPU.
After confirming the code is working I ran python direct_app.py it has pass 800 sec still no results output was

2023-04-25 10:40:23;INFO;Loading audio file...

2023-04-25 10:40:23;INFO;Pre-processing audio file...

according to me when I debug the issue is in the below for loop logic it has printed Transcribing... {'stride': [(25728, 0, 0)], 'input_features': array([[[-0.58101606, -0.58101606, -0.38316262, ..., -0.58101606, and stuck at there after that for loop I have also added a print but that is not getting a trigger.
This is the for loop

   for batch, _ in zip(dataloader, progress.tqdm(dummy_batches, desc="Transcribing...")):
            print("Transcribing...", batch.shape)
            model_outputs.append(
                pipeline.forward(batch, batch_size=BATCH_SIZE, task=task, return_timestamps=return_timestamps)
            )

@sanchit-gandhi
Copy link
Owner

Hey @syedmustafa54 - could you maybe try and pinpoint the exact line the app is failing on? Perhaps you can quit the app when it hangs (CTRL + C) and paste the traceback here? For reference, I'm using direct_app.py exactly as is to host the Whisper JAX demo on the HF Hub!

There are some more verbose instructions for creating a demo here: https://github.com/sanchit-gandhi/whisper-jax#creating-an-endpoint

@syedmustafa54
Copy link
Author

@sanchit-gandhi for your case you are just hosting the front end on hugging face and back end at Google TPU I think that's why you are not getting any issues.
this are the logs after CTRL+C

Initialized persistent compilation cache at ./jax_cache
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:319: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your components from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `optional` parameter is deprecated, and it has no effect
  warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:183: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:121: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/outputs.py:22: UserWarning: Usage of gradio.outputs is deprecated, and will not be supported in the future, please import your components from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:27: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `numeric` parameter is deprecated, and it has no effect
  warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_1
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_2
  warnings.warn(
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
2023-04-26 12:00:30;INFO;Loading audio file...
2023-04-26 12:00:31;INFO;Pre-processing audio file...
^CKeyboard interruption in main thread... closing server.
Process ForkPoolWorker-4:
Process ForkPoolWorker-6:
Process ForkPoolWorker-8:
Process ForkPoolWorker-7:
Process ForkPoolWorker-5:
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
    res = self._reader.recv_bytes()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
^CTraceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1984, in block_thread
    time.sleep(0.1)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/whisper-jax/app/direct_app.py", line 236, in <module>
    demo.launch(server_name="0.0.0.0", show_api=False)
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1901, in launch
    self.block_thread()
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1987, in block_thread
    self.server.close()
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/networking.py", line 43, in close
    self.thread.join()
  File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1029, in join
    self._wait_for_tstate_lock()
  File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1045, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^CProcess ForkPoolWorker-9:
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
    res = self._reader.recv_bytes()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 692, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 672, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt
^C

@sanchit-gandhi
Copy link
Owner

Hey @syedmustafa54 - the application is agnostic to whether we host on HF / locally. How many preprocessing workers are you using? Is this more than the number of CPUs you have available? If so, then multiprocessing will hang indefinitely.

Could you check the number of CPUs you have through:

lscpu

And make sure that this number is less than the product of NUM_PROC:

whisper-jax/app/app.py

Line 22 in 7159222

NUM_PROC = 8

And concurrency_count:

whisper-jax/app/app.py

Line 244 in 7159222

demo.queue(concurrency_count=1, max_size=5)

(so lscpu should be less than NUM_PROC * concurrency_count )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[App] Better error message for failed request sanchit-gandhi/whisper-jax
2 participants
@syedmustafa54 @sanchit-gandhi