Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ludwig-based model train and tune support. #935

Merged
merged 36 commits into from
Aug 21, 2023
Merged

Ludwig-based model train and tune support. #935

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
0312c7a
Revert "Revert "Fix benchmark documentation (#931)""
xzdandy Aug 15, 2023
b1c2dc2
Merge branch 'master' of github.com:georgia-tech-db/eva
xzdandy Aug 17, 2023
4d888e5
Ludwig based model train and tune support
xzdandy Aug 17, 2023
1eb6dcd
LINTER
xzdandy Aug 17, 2023
5554e08
LINTER
xzdandy Aug 17, 2023
98d7119
LINTER
xzdandy Aug 17, 2023
92d3030
Fix test
xzdandy Aug 18, 2023
d0f4f75
Fix test
xzdandy Aug 18, 2023
2d46ac6
remove print
xzdandy Aug 18, 2023
aec2ae9
refactor binder for binded column for child select
xzdandy Aug 18, 2023
f4a8f66
LINTER
xzdandy Aug 18, 2023
3aee0e6
pydantic conflict with ray
xzdandy Aug 18, 2023
6be7e2b
Handle inputs and outputs column definition for ludwig UDFs
xzdandy Aug 18, 2023
5b73d5a
LINTER
xzdandy Aug 18, 2023
2937c05
Integration test for model train
xzdandy Aug 18, 2023
911450a
Merge branch 'master' of github.com:georgia-tech-db/eva
xzdandy Aug 18, 2023
d84edc4
Merge branch 'master' into model_train
xzdandy Aug 18, 2023
19de275
Fix parser for CREATE UDF TYPLE Ludwig
xzdandy Aug 18, 2023
4f2e20b
Fix setup dependency for ludwig
xzdandy Aug 19, 2023
4a6890f
Fix optimizer for CREATE UDF FROM SELECT
xzdandy Aug 19, 2023
31e619d
Fix parser
xzdandy Aug 19, 2023
c6fa2eb
Fix test rule
xzdandy Aug 19, 2023
cab3c24
Fix column name
xzdandy Aug 19, 2023
99deaca
Add ludwig skip marker for tests
xzdandy Aug 19, 2023
8b97b86
Fix drop column alias
xzdandy Aug 19, 2023
fa93665
Merge branch 'master' of github.com:georgia-tech-db/eva
xzdandy Aug 19, 2023
a99cfdb
Merge branch 'master' into model_train
xzdandy Aug 19, 2023
97c2dac
Try enable ludwig test for all non ray test on circlci.
xzdandy Aug 19, 2023
70ab989
Fix circle ci config
xzdandy Aug 19, 2023
03edea1
Add documentation for model training
xzdandy Aug 19, 2023
a102550
Try fix the dependency conflict in circle ci
xzdandy Aug 19, 2023
2d8375a
Fix typo
xzdandy Aug 19, 2023
2a4dc3d
Modify the yaml after installtion
xzdandy Aug 19, 2023
1afa076
Remove old config
xzdandy Aug 19, 2023
8d38fe7
Skip the native executor test and skip ludwig and ray for python3.1
xzdandy Aug 19, 2023
35eadce
Try fix the inconsistent LINTER between local and circle ci
xzdandy Aug 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 17 additions & 25 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,35 +156,27 @@ jobs:

- run:
name: Install EvaDB package from GitHub repo with all dependencies
environment:
PY_VERSION: << parameters.v >>
RAY: << parameters.ray >>
command: |
"python<< parameters.v >>" -m venv test_evadb
pip install --upgrade pip
source test_evadb/bin/activate
pip install ".[dev]"

# Enable Ray (update evadb.yml file and install Ray package)
- when:
condition:
equal: [ ENABLED, << parameters.ray >> ]
steps:
- run:
name: Enable Ray setting in the config.yml file
command: |
source test_evadb/bin/activate
python -c "import yaml;f = open('evadb/evadb.yml', 'r+');config_obj = yaml.load(f, Loader=yaml.FullLoader);config_obj['experimental']['ray'] = True;f.seek(0);f.write(yaml.dump(config_obj));f.truncate();"
pip install ".[ray]"

# Install qdrant only on versions < 3.11
- when:
condition:
not:
equal: [ "3.11", << parameters.v >> ]
steps:
- run:
name: Install Qdrant
command: |
source test_evadb/bin/activate
pip install ".[qdrant]"
if [ $RAY = "ENABLED" ]; then
if [ $PY_VERSION != "3.11" ]; then
pip install ".[dev,ray,qdrant]"
else
pip install ".[dev]" # ray < 2.5.0 does not work with python 3.11 ray-project/ray#33864
fi
python -c "import yaml;f = open('evadb/evadb.yml', 'r+');config_obj = yaml.load(f, Loader=yaml.FullLoader);config_obj['experimental']['ray'] = True;f.seek(0);f.write(yaml.dump(config_obj));f.truncate();"
else
if [ $PY_VERSION != "3.11" ]; then
pip install ".[dev,ludwig,qdrant]"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gaurav274 @jiashenC @jarulraj To avoid dependency conflict, I have changed the way that circle ci installs the EvaDB. Let me know if you have a different idea. Thanks.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with this as well. Another way to rewrite this using the current infrastructure is when the ray is disabled and the python version is not 3.11. We already have both of those flags. If I understand correctly, we just need to combine https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L168 and https://github.com/georgia-tech-db/evadb/blob/master/.circleci/config.yml#L181?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not able to find a way to do when else in circle ci. I am following the suggestions from https://discuss.circleci.com/t/conditional-steps-if-else-without-code-duplication/45030

Copy link
Member

@jiashenC jiashenC Aug 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking of the following logic.

// we already have those two
if ray enabled: install ray
if python != 3.11: install qdrant

// we add for this PR
if ray disabled and python != 3.11: install ludwig

We can use nested logic https://circleci.com/docs/configuration-reference/#logic-statements to specify ray disabled and python != 3.11 in one when statement. Will this work?

else
pip install ".[dev]" # ray < 2.5.0 does not work with python 3.11 ray-project/ray#33864
fi
fi

- run:
name: Test and upload coverage report to coveralls
Expand Down
Loading