Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor tweaks to retrieval skill and file ingestion #9093

Merged
merged 5 commits into from
Apr 19, 2024

Conversation

dusvyat
Copy link
Contributor

@dusvyat dusvyat commented Apr 18, 2024

Description

  1. The default chunk size and overlap values in file_handler have been increased for better performance.
  2. In langchain_handler, checks have been added to use the same provider and model for embeddings if 'embedding_model_args' are not provided in input params.
  3. Skill_tool's retriever_config has been renamed to 'config' for clarity.
  4. Catch LLM output parser errors and return response if relevant or raise error if genuine error.

Fixes #issue_number

Type of change

(Please delete options that are not relevant)

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ⚡ New feature (non-breaking change which adds functionality)
  • 📢 Breaking change (fix or feature that would cause existing functionality not to work as expected)
  • 📄 This change requires a documentation update

Verification Process

To ensure the changes are working as expected:

  • Test Location: Specify the URL or path for testing.
  • Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

  • I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

  • My code follows the style guidelines(PEP 8) of MindsDB.
  • I have appropriately commented on my code, especially in complex areas.
  • Necessary documentation updates are either made or tracked in issues.
  • Relevant unit and integration tests are updated or added.

The default chunk size and overlap values in file_handler have been increased for better performance. In langchain_handler, checks have been added to use the same provider and model for embeddings if 'embedding_model_args' are not provided in input params. Skill_tool's retriever_config has been renamed to 'config' for clarity.
This update modifies the langchain handler to accommodate errors during the invocation process. Specifically, it implements exception handling for the agent_executor's invoke method. If an error occurs, instead of crashing, it will now return the error message. However, if the error doesn't match a specific format, the exception will still be raised.
@dusvyat
Copy link
Contributor Author

dusvyat commented Apr 18, 2024

cc @ZoranPandovski

This change introduces logging for situations where 'vector_store_config' is not present in the `langchain_handler` tool configuration. It also modifies the condition that checks for the absence of this property to add a persisting directory. Furthermore, it refactors the code in `langchain_handler.py` related to the absence of 'embedding_model_args'.
@dusvyat dusvyat requested a review from ea-rus April 19, 2024 06:11
The default vector store setup was removed from the langchain handler. The 'vector_store_config' is no longer automatically assigned, and an alert for this missing parameter is no longer logged. The code was adjusted to operate without the previously used 'mindsdb_path'.
Copy link
Contributor

@tmichaeldb tmichaeldb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice changes, let's get this merged for the hackathon

@tmichaeldb tmichaeldb merged commit 8abf1c1 into staging Apr 19, 2024
13 checks passed
@hamishfagg hamishfagg deleted the retrieval-skill-minor-tweaks branch June 10, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants