Managing knowledge base columns

[ea-rus]

Description

Fixes RAG-52

Added parameters to KB to specify input columns:

• metadata_columns - optional, default: no metadata columns
• content_columns - optional, default: all columns
• id_column - optional, default: id

Example:

CREATE KNOWLEDGE BASE my_kb
USING
 metadata_columns = ['date', 'creator'],
 content_columns = ['review'],
 id_column='index'   

Logic of different combination of params:
For id column:

• if id_column is defined:
  • use it as id
• elif 'id' column exists:
  • use it
• else:
  • use hash(content)

For content and metadata:

• if content_columns is defined:
  • if len(content_columns) > 1:
    • make text from row (col: value\n col: value)
  • if metadata_columns is defined:
    • use them as metadata
  • else:
    • use all unused columns is metadata
• elif metadata_columns is defined:
  • metadata_columns go to metadata
  • use all unused columns as content (make text if columns>1)
• else:
  • no metadata
  • all unused columns go to content (make text if columns>1)

Compared to #8990: it concatenates columns globally and prepares input to vector storage. And we don't have to adapt every vector storage (or embedding model)

Example on video:
https://www.loom.com/share/8da2d8de7e6e45a9b1fd4036ef9d65ad

Fixes #issue_number

Type of change

(Please delete options that are not relevant)

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ⚡ New feature (non-breaking change which adds functionality)
• 📢 Breaking change (fix or feature that would cause existing functionality not to work as expected)
• 📄 This change requires a documentation update

Verification Process

To ensure the changes are working as expected:

• Test Location: Specify the URL or path for testing.
• Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

• I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

• My code follows the style guidelines(PEP 8) of MindsDB.
• I have appropriately commented on my code, especially in complex areas.
• Necessary documentation updates are either made or tracked in issues.
• Relevant unit and integration tests are updated or added.

[dusvyat]

Overally implementation looks good but let's also include a few unit tests 🙏

[ea-rus]

Overally implementation looks good but let's also include a few unit tests 🙏

Added unit tests for several combination of settings

[dusvyat]

lgtm