Ludwig-based model train and tune support. #935

xzdandy · 2023-08-17T09:41:29Z

Provide support for the following queries.

CREATE UDF IF NOT EXISTS PredictHouseRent FROM
(SELECT * FROM home_rentals)
TYPE Ludwig
'predict' 'rental_price';

SELECT PredictHouseRent(...) FROM home_rentals;

Basic implementation
Documentation on the model training feature.
Set up circleci for ludwig tests

👋 Thanks for submitting a Pull Request to EvaDB!

🙌 We want to make contributing to EvaDB as easy and transparent as possible. Here are a few tips to get you started:

🔍 Search existing EvaDB PRs to see if a similar PR already exists.
🔗 Link this PR to a EvaDB issue to help us understand what bug fix or feature is being implemented.
📈 Provide before and after profiling results to help us quantify the improvement your PR provides (if applicable).

👉 Please see our ✅ Contributing Guide for more details.

This reverts commit 84efe77.

test/integration_tests/test_model_train.py

xzdandy · 2023-08-19T18:13:50Z

The training support is ready. Testcases failed are not related to training. I will fix the fuzzy join one. And Jiashen will fix the native executor. We will set up a new circleci test for training.

xzdandy · 2023-08-19T22:56:27Z

.circleci/config.yml

+              python -c "import yaml;f = open('evadb/evadb.yml', 'r+');config_obj = yaml.load(f, Loader=yaml.FullLoader);config_obj['experimental']['ray'] = True;f.seek(0);f.write(yaml.dump(config_obj));f.truncate();"
+            else
+              if [ $PY_VERSION != "3.11" ]; then
+                pip install ".[dev,ludwig,qdrant]"


@gaurav274 @jiashenC @jarulraj To avoid dependency conflict, I have changed the way that circle ci installs the EvaDB. Let me know if you have a different idea. Thanks.

I am fine with this as well. Another way to rewrite this using the current infrastructure is when the ray is disabled and the python version is not 3.11. We already have both of those flags. If I understand correctly, we just need to combine https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L168 and https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L181?

I am not able to find a way to do when else in circle ci. I am following the suggestions from https://discuss.circleci.com/t/conditional-steps-if-else-without-code-duplication/45030

I am thinking of the following logic.

// we already have those two if ray enabled: install ray if python != 3.11: install qdrant // we add for this PR if ray disabled and python != 3.11: install ludwig

We can use nested logic https://circleci.com/docs/configuration-reference/#logic-statements to specify ray disabled and python != 3.11 in one when statement. Will this work?

jiashenC · 2023-08-21T01:37:38Z

.circleci/config.yml

+              python -c "import yaml;f = open('evadb/evadb.yml', 'r+');config_obj = yaml.load(f, Loader=yaml.FullLoader);config_obj['experimental']['ray'] = True;f.seek(0);f.write(yaml.dump(config_obj));f.truncate();"
+            else
+              if [ $PY_VERSION != "3.11" ]; then
+                pip install ".[dev,ludwig,qdrant]"


I am fine with this as well. Another way to rewrite this using the current infrastructure is when the ray is disabled and the python version is not 3.11. We already have both of those flags. If I understand correctly, we just need to combine https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L168 and https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L181?

jiashenC · 2023-08-21T01:45:13Z

evadb/binder/statement_binder.py

+            inputs, outputs = [], []
+            for column in all_column_list:
+                if column.name in predict_columns:
+                    column.name = column.name + "_predictions"


How do we ensure the UDF always returns something named xxx_predictions?

The ludwig appends the _predictions to the target column. I haven't found a way to specify the output column name for ludwig yet.

jiashenC · 2023-08-21T01:52:17Z

docs/source/reference/evaql/create.rst

+.. code-block:: sql
+
+   CREATE UDF IF NOT EXISTS PredictHouseRent FROM
+   (SELECT * FROM HomeRentals)


I remember you were discussing this. Is * working now?

It works. It will consider _row_id also as an input column. I will fix it in later PRs.

jiashenC · 2023-08-21T13:32:24Z

.circleci/config.yml

+              python -c "import yaml;f = open('evadb/evadb.yml', 'r+');config_obj = yaml.load(f, Loader=yaml.FullLoader);config_obj['experimental']['ray'] = True;f.seek(0);f.write(yaml.dump(config_obj));f.truncate();"
+            else
+              if [ $PY_VERSION != "3.11" ]; then
+                pip install ".[dev,ludwig,qdrant]"


I am thinking of the following logic.

// we already have those two if ray enabled: install ray if python != 3.11: install qdrant // we add for this PR if ray disabled and python != 3.11: install ludwig

We can use nested logic https://circleci.com/docs/configuration-reference/#logic-statements to specify ray disabled and python != 3.11 in one when statement. Will this work?

xzdandy added 15 commits August 15, 2023 05:45

Revert "Revert "Fix benchmark documentation (#931)""

0312c7a

This reverts commit 84efe77.

Merge branch 'master' of github.com:georgia-tech-db/eva

b1c2dc2

Ludwig based model train and tune support

4d888e5

LINTER

1eb6dcd

LINTER

5554e08

LINTER

98d7119

Fix test

92d3030

Fix test

d0f4f75

remove print

2d46ac6

refactor binder for binded column for child select

aec2ae9

LINTER

f4a8f66

pydantic conflict with ray

3aee0e6

Handle inputs and outputs column definition for ludwig UDFs

6be7e2b

LINTER

5b73d5a

Integration test for model train

2937c05

xzdandy marked this pull request as ready for review August 18, 2023 08:18

gaurav274 reviewed Aug 18, 2023

View reviewed changes

test/integration_tests/test_model_train.py Show resolved Hide resolved

xzdandy added 10 commits August 18, 2023 18:50

Merge branch 'master' of github.com:georgia-tech-db/eva

911450a

Merge branch 'master' into model_train

d84edc4

Fix parser for CREATE UDF TYPLE Ludwig

19de275

Fix setup dependency for ludwig

4f2e20b

Fix optimizer for CREATE UDF FROM SELECT

4a6890f

Fix parser

31e619d

Fix test rule

c6fa2eb

Fix column name

cab3c24

Add ludwig skip marker for tests

99deaca

Fix drop column alias

8b97b86

xzdandy requested review from gaurav274 and jiashenC August 19, 2023 18:05

xzdandy added 9 commits August 19, 2023 16:35

Merge branch 'master' of github.com:georgia-tech-db/eva

fa93665

Merge branch 'master' into model_train

a99cfdb

Try enable ludwig test for all non ray test on circlci.

97c2dac

Fix circle ci config

70ab989

Add documentation for model training

03edea1

Try fix the dependency conflict in circle ci

a102550

Fix typo

2d8375a

Modify the yaml after installtion

2a4dc3d

Remove old config

1afa076

xzdandy commented Aug 19, 2023

View reviewed changes

xzdandy added 2 commits August 19, 2023 19:41

Skip the native executor test and skip ludwig and ray for python3.1

8d38fe7

Try fix the inconsistent LINTER between local and circle ci

35eadce

jiashenC reviewed Aug 21, 2023

View reviewed changes

jiashenC approved these changes Aug 21, 2023

View reviewed changes

xzdandy merged commit e8f3221 into master Aug 21, 2023

xzdandy deleted the model_train branch August 21, 2023 15:07

gaurav274 assigned xzdandy Aug 25, 2023

gaurav274 added Integrations 🧩 Pull requests that update an integration Feature Request ✨ New feature or request labels Aug 25, 2023

gaurav274 added this to the v0.3.2 milestone Aug 25, 2023

americast mentioned this pull request Aug 31, 2023

Forecasting in EVA #969

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ludwig-based model train and tune support. #935

Ludwig-based model train and tune support. #935

xzdandy commented Aug 17, 2023 •

edited

Loading

xzdandy commented Aug 19, 2023

xzdandy Aug 19, 2023

jiashenC Aug 21, 2023

xzdandy Aug 21, 2023

jiashenC Aug 21, 2023 •

edited

Loading

jiashenC Aug 21, 2023

jiashenC Aug 21, 2023

xzdandy Aug 21, 2023

jiashenC Aug 21, 2023

xzdandy Aug 21, 2023

jiashenC Aug 21, 2023 •

edited

Loading

Ludwig-based model train and tune support. #935

Ludwig-based model train and tune support. #935

Conversation

xzdandy commented Aug 17, 2023 • edited Loading

xzdandy commented Aug 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiashenC Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiashenC Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

xzdandy commented Aug 17, 2023 •

edited

Loading

jiashenC Aug 21, 2023 •

edited

Loading

jiashenC Aug 21, 2023 •

edited

Loading