Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat:add tts-streaming config and future #5492

Merged
merged 20 commits into from
Jul 9, 2024
Merged

feat:add tts-streaming config and future #5492

merged 20 commits into from
Jul 9, 2024

Conversation

ic-xu
Copy link
Contributor

@ic-xu ic-xu commented Jun 21, 2024 •
edited by crazywoola
Loading

Description

Add TTS streaming configuration and feature support, add text to speech in the feature configuration options, click on configure to start

Fixes #5251

Type of Change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Simply configure the streaming TTS option and test it

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 💪 enhancement New feature or request labels Jun 21, 2024
@crazywoola crazywoola requested a review from takatost June 22, 2024 03:54
@crazywoola
Copy link
Member

@charli117 Please review this pr as well.

@charli117
Copy link
Contributor

@crazywoola ic-xu is a colleague of our team, and the version submitted now is the version we are currently using. There is also an optimization point that requires adding a blocking mode output logic for models that are not compatible and do not support streaming tts. We have communicated with each other and merged later

takatost
takatost requested changes Jun 23, 2024
View reviewed changes
api/controllers/web/audio.py Outdated Show resolved Hide resolved
api/core/app/apps/advanced_chat/generate_task_pipeline.py Show resolved Hide resolved
api/core/app/apps/base_app_queue_manager.py Outdated Show resolved Hide resolved
@charli117 charli117 mentioned this pull request Jun 24, 2024
Closed
4 tasks
@ic-xu
feat:Merge blocking and streaming interfaces in TTS, and unify the us…
62b7297 
…e of streaming to implement TTS functionality
@ic-xu
Merge remote-tracking branch 'main/main' into stream_tts
32292f7
@ic-xu
feat :
1188acb 
It has been extracted as a separate module, and processing logic has been added to the corresponding module in GenerateTask Pipeline. process(). At the same time, the expiration time has been set to be small. At the same time, Redis will not store all message objects, but only a portion of the required data, so the data will not be very large
@crazywoola crazywoola requested a review from takatost July 3, 2024 11:31
crazywoola
crazywoola reviewed Jul 3, 2024
View reviewed changes
web/types/app.ts Outdated Show resolved Hide resolved
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jul 4, 2024
crazywoola
crazywoola reviewed Jul 4, 2024
View reviewed changes
web/models/debug.ts Outdated Show resolved Hide resolved
crazywoola
crazywoola reviewed Jul 4, 2024
View reviewed changes
Copy link
Member

@crazywoola crazywoola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

crazywoola
crazywoola previously approved these changes Jul 5, 2024
View reviewed changes
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 5, 2024
@takatost
Copy link
Collaborator

takatost commented Jul 5, 2024

I tested the manual play mode and found it to be unresponsive, with the chrome console reporting the error message as:

Uncaught (in promise) SyntaxError: Unexpected token '�', "�����bt9��"... is not valid JSON

@takatost
Copy link
Collaborator

takatost commented Jul 5, 2024

Maybe it's because they've all switched to streaming mode now, and the web front-end isn't compatible with manual playback mode.

@ic-xu
Copy link
Contributor Author

ic-xu commented Jul 5, 2024

Maybe it's because they've all switched to streaming mode now, and the web front-end isn't compatible with manual playback mode.

Could you provide a test browser? We are using Chrome for testing, which works well and has a fast response speed.

takatost
takatost requested changes Jul 6, 2024
View reviewed changes
Copy link
Collaborator

@takatost takatost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ops, now in the basic chatbot, clicking the play button does not send a request.
Additionally, if the TTS function is enabled for the first time in the Chatflow, clicking the play button will result in an error. This might be due to using a draft version instead of a published one. The error message is as follows:

Traceback (most recent call last):
  File "/Users/takatost/Projects/dify/api/controllers/console/app/audio.py", line 97, in post
    text_to_speech = app_model.workflow.features_dict.get('text_to_speech')
AttributeError: 'NoneType' object has no attribute 'features_dict'

@dosubot dosubot bot removed the lgtm This PR has been approved by a maintainer label Jul 6, 2024
ic-xu added 2 commits July 8, 2024 08:56
@ic-xu
fix : fixed TTS function is enabled for the first time in the Chatflo…
c3b0e14 
…w, clicking the play button will result in an error.
@ic-xu
feat:
6bb76bc 
Fixed audio playback lag, console prompts json parsing error, automatic playback failure, etc. bugs
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 9, 2024
takatost
takatost approved these changes Jul 9, 2024
View reviewed changes
Copy link
Collaborator

@takatost takatost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx!

@takatost takatost merged commit 6ef401a into langgenius:main Jul 9, 2024
5 checks passed
ZhouhaoJiang added a commit that referenced this pull request Jul 9, 2024
@ZhouhaoJiang
Merge branch 'refs/heads/main' into feat/if-elif
d98472c 
* refs/heads/main: (51 commits)
  feat: tailwind related improvement (#6085)
  feat: support AnalyticDB vector store (#5586)
  feat:add tts-streaming config and future (#5492)
  Feat: add index bar to select tool panel of workflow (#6066)
  bump to 0.6.13 (#6078)
  fix: Inconsistency Between Actual and Debug Input Variables (#6055)
  refactor: revamp picker block (#4227)
  chore: remove tsne unused code (#6077)
  fix: relative in overflow div (#5998)
  chore(action): move docker login above Set up QEMU in build-push action workflow (#6073)
  remove clunky welcome message (#6069)
  feat: add request_params field to jina_reader tool (#5610)
  fix azure stream download (#6063)
  chore: hide tracing introduce detail (#6049)
  Address the issue of the absence of poetry in the development container. (#6036)
  Fix authorization header validation to handle bearer types correctly - "authorization config header is required" error (#6040)
  Fix/6034 get random order of categories in explore and workflow is missing in zh hant (#6043)
  Modify slack webhook url validation to allow workflow (#6041) (#6042)
  fix(configs): Update pydantic settings in config files (#6023)
  Fix/incorrect parameter extractor memory (#6038)
  ...
@takatost takatost mentioned this pull request Jul 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@takatost takatost takatost approved these changes

@crazywoola crazywoola crazywoola approved these changes

Assignees
No one assigned
Labels
💪 enhancement New feature or request lgtm This PR has been approved by a maintainer size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
4 participants
@ic-xu @crazywoola @charli117 @takatost