Feat: change the data preprocessing as an interface version

[qidanrui]

The Added Feature

After changing the data preprocessing part as an interface version, users can directly use our sql data preprocessor by using following code:

pip install dbgpt_hub

from dbgpt_hub.data_process import preprocess_sft_data

data_folder = "dbgpt_hub/data"
data_info = [
        {
            "data_source": "spider",
            "train_file": ["train_spider.json", "train_others.json"],
            "dev_file": ["dev.json"],
            "tables_file": "tables.json",
            "db_id_name": "db_id",
            "is_multiple_turn": False,
            "train_output": "spider_train.json",
            "dev_output": "spider_dev.json",
        }
]

preprocess_sft_data(
      data_folder = data_folder,
      data_info = data_info
)

How Do I Test the Feature

1. Test Environment: Google Colab with T4 GPU
2. Test Process:
   • Run poetry install

* Run `poetry build`

* Run `poetry run pip install dist/dbgpt_hub-0.2.1-py3-none-any.whl`

* Run the Python code above

Comments

I still keep Hongyang's previous version for the current soft execution of DB-GPT-Hub. In the future, DB-GPT-Hub can be a package released on PyPI and Conda with APIs for data preprocessing and fine-tuning. This is the first step. I'll continue doing works to achieve this goal.

[wangzaistone]

so great!

[qidanrui]

so great!

I'll complete the test on GPU by this weekend. please donnot merge it before the test is completedThanks

[wangzaistone]

LGTM！GREAT！