Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Inline Query Functionality with Action Button and Improved Caching #101

Merged
merged 5 commits into from
Apr 15, 2023

Conversation

bugfloyd
Copy link
Contributor

Background

The current inline queries feature functions more like a decorative element, as it does not operate as a true inline query. The shortcomings include:

  • When users select the offered inline result, the final message only includes the user prompt and not the answer.
  • The bot needs to be added to the group to enable it to reply with the actual answer in the chat.
  • It does not work in private chats due to the previous point.

Existing Challenges and Possible Solutions

If we want to generate a response from OpenAI APIs and provide it within the inline result or the final output, there are some clear challenges:

  • Telegram sends inline queries to the bot whenever the user pauses typing. Sending requests to OpenAI for each user input event would waste many tokens and negatively impact bot performance.
  • I tried implementing a solution using Python threads, timers, and the event loop to reduce the number of requests and only send them if we think the user has finished typing. However, this solution was hacky and PTB does not properly support Python threads.
  • I also attempted to implement a queue system in the bot class. The idea was to add new updates to this queue and process them only if we think the user has finished editing the query (by using a timeout). Although it worked for simple user input behaviors, the user had to wait for a long time to see the inline result, and handling edge cases proved to be tricky. For example, when a user types something, takes a long pause, and then continues to write a long question with short pauses, or removes some content. The problem is that each inline query event has a unique ID, and the queue system may not always use the last event in these edge cases.

The Solution

I retained the initial inline results as previews with just the prompt and no answers included, but added an action button (InlineKeyboardButton) in the final output to generate the actual answer to the prompt. This button triggers a callback, in which we read the prompt, get the answer for it, and edit the original message with the answer. While generating the answer, it also adds a status to the message.

I faced challenges in resolving the following issue:
How can we get the original query (prompt) in the action callback function? I found callback_data to be a suitable place to store the prompt and read it in the callback, but unfortunately, Telegram imposes a 64-byte limit on this property. To overcome this limit, I implemented a built-in caching system for each user's inline queries. I ended up using callback_data for short prompts because it is a more reliable solution, and it partially fixes the known bug below.

Known Issues

When a user types a long inline prompt, the query gets cached and, upon clicking the action button and retrieving the answer, that cache is removed. If the user sends the exact same long prompt, the initial response with the action button is generated by the Telegram/bot cache. The action button does not work since we do not have cache available for the unique ID included in the callback_data (which came from the Telegram cache and includes the original unique ID). This bug does not exist for short prompts, as the code uses callback_data to transport the prompt. Considering this bug as a real edge case, I think we are safe to use the current flow.

Note:

For inline queries, I am using the user ID as the chat ID when getting answers from the OpenAI class. Doing so enables users to have a consistent inline query experience across different chats (e.g., asking something in a group and referring to it while having a private conversation at the same time).

Detailed Changes

  • Add 'private' to the list of supported chat types for inline queries
  • Update inline_query method to:
    • Include an action button for fetching the answer
    • Utilize callback_data to store short prompts
    • Implement a built-in cache for longer prompts
  • Introduce handle_callback_inline_query as a callback query handler to:
    • Retrieve the prompt from callback_data or the built-in cache
    • Obtain the answer from OpenAI
    • Update the current message content with the status and answer
  • Implement validate_answering_possibility method to verify user access and budget
  • Create process_used_tokens method to apply token usage
  • Employ validate_answering_possibility in image, transcribe, prompt, and inline_query handlers
  • Extend inline query support to is_allowed and is_within_budget methods
  • Convert error_handler, split_into_chunks, is_user_in_group, and is_group_chat to static methods
  • Address lint warnings and convert unused method parameters to private

Screenshots

Image 1 Image 2 Image 3 Image 3

Future Plans and Enhancements

The current caching system for prompts serves as an initial MVP. We may consider refining or expanding it based on our requirements.
At present, only GPT models are supported in inline queries. We can also explore incorporating image generation using DALL-E, following a similar workflow.

Resolves #41

- Add 'private' to supported chat types for inline queries
- Update inline_query() to:
  * Include an action button to fetch the answer
  * Use callback_data to store short prompts
  * Implement built-in cache for longer prompts
- Add handle_callback_inline_query() as callback query handler to:
  * Read the prompt from callback_data or the built-in cache
  * Fetch the answer from OpenAI
  * Update the current message content with the status and answer
- Add validate_answering_possibility method to verify user access and budget
- Add process_used_tokens method to apply token usage
- Use validate_answering_possibility in image, transcribe, prompt, and inline_query handlers
- Add inline query support to is_allowed and is_within_budget methods
- Convert error_handler, split_into_chunks, is_user_in_group, and is_group_chat to static methods
- Fix lint warnings and convert unused method parameters to private
@k3it
Copy link
Contributor

k3it commented Mar 24, 2023

I've tried this and it worked quite well. there are certainly some constraints in telegram's inline mode that you had to work around. There was a prompt that for some reason did not generate an icon until i added some random extra characters to it.. some sort of edge bug possibly.. but overall i like this, hopefully will get merged.

@n3d1117
Copy link
Owner

n3d1117 commented Mar 25, 2023

Hi @bugfloyd, awesome work and thanks for the writeup! Tested it and I love it

It works well in private chats. The only issue I found is that when used in groups with the bot in it, you will get the answer twice (one inline, and then another one because you sent a query via the bot).

What do you think the best solution would be? Enable this new inline button in private chats only?

@bugfloyd
Copy link
Contributor Author

Hi @bugfloyd, awesome work and thanks for the writeup! Tested it and I love it

It works well in private chats. The only issue I found is that when used in groups with the bot in it, you will get the answer twice (one inline, and then another one because you sent a query via the bot).

What do you think the best solution would be? Enable this new inline button in private chats only?

I am glad to hear that :)

I believe having this feature enabled for group chats would be very handy specially when the bot is not/could not added to the group.
I am currently on a trip. I will continue working on this PR next week to find a better solution to handle group chats.
That edge case bug can also be annoying (using the same long prompt multiple times). I will try to find a fix for it.

@n3d1117 n3d1117 mentioned this pull request Apr 1, 2023
@Pumbaardian
Copy link

Hello, bugfloyd!
Can you upgrade this to the newest (0.2.3) version of n3d1117 bot

@k3it
Copy link
Contributor

k3it commented Apr 13, 2023

This was a nicely done pull request but becoming a bit stale. any chance to whip it into shape for merging @bugfloyd ?

@bugfloyd
Copy link
Contributor Author

My apologies for the delay in responding to this PR. I have had a very busy couple of weeks, but I will do my best to get this PR ready for merge by this weekend.
In the meantime, I want to let you know that I have found a solution to the known bug that was mentioned, by disabling the cache for inline queries.

However, I have some concerns regarding disabling inline queries for group chats, as this feature could be very handy when the bot is not joined to the group. Therefore, I want to explore other possible solutions to prevent duplicate messages while still keeping the inline query feature enabled for group chats.

If either of you have any ideas or suggestions, please feel free to share them with me.

@k3it
Copy link
Contributor

k3it commented Apr 13, 2023

@bugfloyd I agree - definitely keep the inline mode for group chats, since it's not practical to add the bot to every group where it might be useful occasionally. If a duplicate message cannot be easily avoided i think it's a small "side effect" that we can live with for now and fixed at a later time.

Most users would probably realize immediately after receiving a duplicate reply that there is no real reason to use the inline mode if the bot is already a direct member.

# Conflicts:
#	bot/telegram_bot.py
Handle disallowed_message & budget_limit_message for inline queries
@bugfloyd
Copy link
Contributor Author

This PR is now ready for review and merging. I've addressed the known bug by disabling the bot cache for inline queries, ensuring that the internal inline caching system works correctly for all cases. Consequently, even short prompts no longer use the callback button data, and the internal cache is employed for all inline queries.

Additionally, I've handled disallowed and budget_limit messages for inline queries.

In the near future, I plan to submit several more PRs to address:

  • Class cleanup and formatting, which were initially part of this PR but removed due to merge conflicts
  • Streaming support for inline queries
  • Translations support for inline queries
  • Image command support for inline queries

@n3d1117
Copy link
Owner

n3d1117 commented Apr 15, 2023

Thank you @bugfloyd great work!

@n3d1117 n3d1117 merged commit ddd70eb into n3d1117:main Apr 15, 2023
@n3d1117
Copy link
Owner

n3d1117 commented Apr 15, 2023

Done in #230:

  • Translations support for inline queries
  • Fix duplicated answers when inline query is used within a group with bot in it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for using inline mode in private conversations
4 participants