-
Notifications
You must be signed in to change notification settings - Fork 3.1k
(examples revamp): add example walkthroughs #5613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
9ff2eae
revise survey agent readme
tinalenguyen 1fc231e
survey readme
tinalenguyen 5762306
frontdesk readme
tinalenguyen c2b6ba6
drive thru readme
tinalenguyen 2490cc8
Create README for healthcare example
tinalenguyen b877547
format
tinalenguyen 47b39db
update per feedback
tinalenguyen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Healthcare Example | ||
|
|
||
| A full healhcare assistant providing secure appointment management, billing handling, and lab result retrieval. | ||
|
|
||
| For setup instructions and more details, see the [main examples README](https://github.com/livekit/agents/blob/main/examples/README.md). | ||
|
|
||
| ## Overview | ||
|
|
||
| The healthcare agent utilizes a variety of `AgentTasks` to achieve structured workflows to collect information. This example is modality-agnostic, where users can interact via text or voice and switch seamlessly. If the conversation heads out of the scope of the agent, the user will be transfered to a human. | ||
|
|
||
| ### Profile Authentication | ||
|
|
||
| Before any sensitive information is queried, the user must go through an authentication process. This process will only occur once per call. If the user provides a name and birthday existing in the database, the process is fast-forwarded. This is possible via task completion callbacks: | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L574-L588 | ||
| Otherwise, the user will also be asked for their phone number and insurance provider | ||
| to create a profile. | ||
|
|
||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L590-L641 | ||
|
|
||
| After the profile is created, the agent will be given a tool to update the patient's record as needed. | ||
|
|
||
| ### Appointment Management | ||
|
|
||
| Function tools are added dynamically, so the LLM cannot hallucinate parameters or call tools prematurely. The user is given options for compatible doctors based on their insurance: | ||
|
|
||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L306-L333 | ||
|
|
||
| After the doctor is chosen, the appointment scheduling tool is built dynamically with the availabilities. | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L734-L780 | ||
|
|
||
| Finally, the visit reason is collected, and once confirmed the database will be updated accordingly (the doctor's availability will be removed). | ||
|
|
||
| Users are also able to modify existing appointments. If the user wishes to reschedule an appointment, the appointment is canceled and `ScheduleAppointmentTask` is reused. | ||
|
|
||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L506-L543 | ||
|
|
||
| ### Billing Handling | ||
|
|
||
| `GetCreditCardTask()` is showcased here. The user's details are verified, and a balance is generated and connected to their profile. | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L734-L748 | ||
|
|
||
| ### Lab Result Retrieval | ||
|
|
||
| We use OpenAI's provider tool, `FileSearch`, to read through an uploaded lab report (mock_checkup_report.pdf). | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/healthcare/healthcare_agent.py#L734-L780 | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,58 @@ | ||
| # Survey Example | ||
| # Survey Agent | ||
|
|
||
| Example showing how to build an automated voice survey agent. | ||
| Screen a candidate for a software engineer role to see if they meet the prerequisites and are an overall good fit. The responses, summary, and evaluation will be written to a CSV file. | ||
|
|
||
| For setup instructions and more details, see the [main examples README](../README.md). | ||
|
|
||
| ## Overview | ||
|
|
||
| The flow of this agent is flexibly structured, where the specified sequence is maintained but the user is able to regress to a previously visited task if needed. This is possible via `TaskGroup`, which is set up here: https://github.com/livekit/agents/blob/f8efe436afe2470104ce7587f1d89ae383ed619e/examples/survey/survey_agent.py#L285-L315 | ||
|
|
||
| ### IntroTask | ||
| This stage facilitates introductions and collects the candidate’s name. | ||
|
|
||
| ### GetEmailTask | ||
| This task is built in to our framework. By default, it can collect and update emails and mark when a user doesn’t want to give their email. If the input modality is audio, emails are confirmed before the task is marked as complete. See the [docs for GetEmailTask](https://docs.livekit.io/agents/prebuilt/tasks/get-email/). | ||
|
|
||
| ### CommuteTask | ||
| This stage collects whether or not the candidate can commute to the office and their method of transportation. We define the possible commute methods here: | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/survey/survey_agent.py#L32 | ||
|
|
||
| And we pass this to a function tool like so: | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/survey/survey_agent.py#L231-L237 | ||
|
|
||
| ### ExperienceTask | ||
| This stage collects the candidate’s years of experience and a short description of their professional career. It follows a structure similar to `IntroTask` and `CommuteTask`. | ||
|
|
||
| ### BehavioralTask | ||
| For some tasks, you might not want a structured flow of questions. In this stage, we are collecting the candidate’s strengths, weaknesses, and work style. This task incrementally collects answers in no particular order. This allows for a more natural conversation. | ||
|
|
||
| After the candidate answers one of the questions, `self._check_completion()` is called to check if all 3 fields (`”strengths”`, `“weaknesses”`, `“work_style”`) have been collected. If so, then `BehavioralTask` is marked as complete. If not, then the agent will continue prompting for the rest of the answers. | ||
|
|
||
| In practice, this would ensure variability among candidates’ experiences. | ||
|
|
||
| ### Closing out | ||
| Once the interview is concluded and TaskGroup is completed, we extract the summary message (the last inserted message): | ||
|
|
||
| ```python | ||
| summary = self.chat_ctx.items[-1] | ||
| ``` | ||
|
|
||
| And we generate a candidate evaluation based off of the summary: | ||
|
|
||
| ```python | ||
| evaluation = await evaluate_candidate(llm_model=self.session.llm, summary=summary) | ||
| ``` | ||
|
|
||
| The session LLM evaluates the candidate from the given summary: | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/survey/survey_agent.py#L76-L98 | ||
|
|
||
|
|
||
| Finally, the agent hangs up and you can find the results, summary, and evaluation in `results.csv`! | ||
|
|
||
| ### Disqualification | ||
| In each stage after the first, the candidate may be disqualified for unsatisfactory answers or for refusing to answer. We create a function tool that will be passed to the tasks: | ||
| https://github.com/livekit/agents/blob/8283a5a5c9863a07bcf030ee90e8ab780e1e569b/examples/survey/survey_agent.py#L101-L118 | ||
|
|
||
| The candidate will be informed of the interview ending, and then the session will shut down. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
btw not related to the readme, but I'm wondering if we should cleanup this pattern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah.. i feel like working with
chat_ctxis not intuitive