Schema refactor #9001

ea-rus · 2024-03-27T08:26:59Z

Description

Changes:

information_schema database

Created information_schema database

Tables visible in GUI: MODELS, DATABASES, ML_ENGINES, HANDLERS, JOBS, TRIGGERS, CHATBOTS, KNOWLEDGE_BASES, SKILLS, AGENTS, VIEWS
Also information_schema has system tables which are not visible in GUI. But it is possible to select from them because they used by database tools to get database objects, for example: TABLES, SCHEMATA, PLUGINS, ...

Syntax update

SHOW:

SHOW PROJECTS support
SHOW <object> displays objects only from current project
To show from other project: SHOW <object> FROM <project name>
Also it is possible to filter output:
- filter by condition show models where engine='openai'
- filter by name show models like 'home%'

DESCRIBE:

Implemented DESCRIBE <object_type> <object_name> command.
- It works the same as query: select * from informational_schema.<object_type> where name=<object_name>

Update the active model version:

was: update models_versions set active=1 where version=11 and name='task_model'
will be: Set active task_model.11

Remove the model version:

was: delete from models_versions where version=2 and name='task_model'
will be: Drop model task_model.2

Changes in project objects

Tables removed from projects (moved to information_schema):

models and models_versions: combined into one table information_schema.MODELS
- it has 'version' column and shows all models versions
mdb_triggers. Moved to information_schema.TRIGGERS
- 'select * from information_schema.triggers' displays different set of columns for sql and http server
  in mysql server columns are extended by system columns required by db tools
jobs and chatbots also removed

For back compatibility: it is possible to select from these tables in project (for example select * from mindsdb.models) but this feature be removed in next updates.

Added to projects:

Agents are visible in project, because they behave like a model

In action
https://www.loom.com/share/5ac2ce8872384293ba4ea48bc09c2081

Dependent on mindsdb/mindsdb_sql#371

Type of change

⚡ New feature (non-breaking change which adds functionality)
📢 Breaking change (fix or feature that would cause existing functionality not to work as expected)

Verification Process

To ensure the changes are working as expected:

Test Location: Specify the URL or path for testing.
Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

My code follows the style guidelines(PEP 8) of MindsDB.
I have appropriately commented on my code, especially in complex areas.
Necessary documentation updates are either made or tracked in issues.
Relevant unit and integration tests are updated or added.

- shows mindsdb triggers 'select * from information_schema.triggers' uses different set of columns for sql and http server

the same as: select * from informational_schema.<obj_type> where name=<obj_name>

'models' left for GUI

# Conflicts: # mindsdb/api/executor/datahub/datanodes/information_schema_datanode.py

# Conflicts: # mindsdb/interfaces/knowledge_base/controller.py

delete model version updated tests

StpMax · 2024-04-25T10:51:07Z

mindsdb/api/executor/datahub/datanodes/mindsdb_tables.py

+from mindsdb.interfaces.database.projects import ProjectController
+
+
+from .system_tables import Table


It is better to use absolute imports where possible

StpMax · 2024-04-25T10:53:21Z

mindsdb/api/executor/datahub/datanodes/mindsdb_tables.py

+        data = triggers_controller.get_list(project_name)
+
+        columns = self.mindsdb_columns
+        if inf_schema.session.api_type == 'sql':


Do we need api_type arg only for this specific case?
Is it a good idea to return a different set of columns for each API?

yes, only for this case.
we use the whole set columns for db managers (like dbeaver) and limited set for web GUI, to make able user see only essential information

Also division between system_tables and mindsdb_tables - it is only for SQL server, to make sql clients working. In GUI we show only mindsb tables (though allow to select from all of them)

StpMax · 2024-04-25T10:56:42Z

mindsdb/api/executor/datahub/datanodes/information_schema_datanode.py

-                self.get_dataframe_funcs[table_name] = partial(
-                    self._get_empty_table, table_name
-                )
+        self.tables = {t.name: t() for t in self.tables_list}


Looks like tables classes do not use self. May be make them static? so we do not need to instantiate them each time.

changed to class_methods

StpMax · 2024-04-25T10:59:54Z

Set model_active = task_model.11 looks strange. In mysql there is few set commands without '=', like
SET CHARACTER SET 'name'
or
SET DEFAULT ROLE admin TO user
May use use something like this?
SET MODEL task_model ACTIVE VERSION TO 11

ea-rus · 2024-04-25T13:44:45Z

Set model_active = task_model.11
changed to
Set active task_model.11

CI will be with error till next midsdb_sql release

ea-rus added 25 commits March 26, 2024 15:21

show projects

ec46f7b

show <obj> from <project name>

6b231b3

models table is equal to models_versions

8aefe89

show tables reduce

66fdfbe

kb: search by project

476e69c

pass 'columns' by name

2eb51cf

unify 'show' for project entities

efcc986

informational_schema.views

a093c36

show triggers:

6b11657

- shows mindsdb triggers 'select * from information_schema.triggers' uses different set of columns for sql and http server

flake8

7b2c337

tests adapted to: models table has versions

ea202fc

information_schema visible in gui for mindsdb tables

94265aa

fix tests

4547938

flake8

cc6b08d

describe obj_type obj_name

7c0486e

the same as: select * from informational_schema.<obj_type> where name=<obj_name>

agents visible in project

8d1a8dd

remove tables which can be select from information_schema

031c284

'models' left for GUI

flake8

56a84f4

Merge branch 'staging' into schema-refactor

b0ab309

# Conflicts: # mindsdb/api/executor/datahub/datanodes/information_schema_datanode.py

fix project name in skills

253cd5a

rename to 'system_tables'

5dade43

fix triggers

fa81e71

unit tests and fixes

71b35e1

Merge branch 'staging' into schema-refactor

58885b6

# Conflicts: # mindsdb/interfaces/knowledge_base/controller.py

flake8

efa6a15

ea-rus requested review from StpMax and martyna-mindsdb April 22, 2024 14:12

ea-rus added 3 commits April 22, 2024 17:20

sql dep

61aeeb7

duplicated 'models' table in gui

49fe8b0

set model active version

1b0a9a7

delete model version updated tests

StpMax requested changes Apr 25, 2024

View reviewed changes

ea-rus added 3 commits April 25, 2024 15:50

Merge branch 'staging' into schema-refactor

28971a4

set active model.version

1826a6e

absolute import

9d6a0a7

schema tables from instance to classes

280c137

StpMax approved these changes Apr 25, 2024

View reviewed changes

sql req

d6420df

ea-rus merged commit 3981834 into staging Apr 26, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schema refactor #9001

Schema refactor #9001

ea-rus commented Mar 27, 2024 •

edited

StpMax Apr 25, 2024

ea-rus Apr 25, 2024

StpMax Apr 25, 2024

ea-rus Apr 25, 2024

StpMax Apr 25, 2024

ea-rus Apr 25, 2024

StpMax commented Apr 25, 2024

ea-rus commented Apr 25, 2024 •

edited

		from mindsdb.interfaces.database.projects import ProjectController


		from .system_tables import Table

Schema refactor #9001

Schema refactor #9001

Conversation

ea-rus commented Mar 27, 2024 • edited

Description

Type of change

Verification Process

Additional Media:

Checklist:

StpMax Apr 25, 2024

Choose a reason for hiding this comment

ea-rus Apr 25, 2024

Choose a reason for hiding this comment

StpMax Apr 25, 2024

Choose a reason for hiding this comment

ea-rus Apr 25, 2024

Choose a reason for hiding this comment

StpMax Apr 25, 2024

Choose a reason for hiding this comment

ea-rus Apr 25, 2024

Choose a reason for hiding this comment

StpMax commented Apr 25, 2024

ea-rus commented Apr 25, 2024 • edited

ea-rus commented Mar 27, 2024 •

edited

ea-rus commented Apr 25, 2024 •

edited