Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement scan pandas #2403

Merged
merged 1 commit into from
Nov 14, 2023
Merged

Implement scan pandas #2403

merged 1 commit into from
Nov 14, 2023

Conversation

acquamarin
Copy link
Collaborator

@acquamarin acquamarin commented Nov 14, 2023

This is the first set of PR to implement scan from pandas. It implements the READ_PANDAS(df) table function which can then be called using CALL statement. The READ_PANDAS function is written and implemented in the pybind extension module, which will be loaded and registered to the database during initilizing the database.
TODO:

  1. Implement a complete replacement framework in binder. Implement replacement framework in binder. #2404
  2. Improvement in the pandas scanner scan pandas improvement #2405

Copy link

codecov bot commented Nov 14, 2023

Codecov Report

Attention: 28 lines in your changes are missing coverage. Please review.

Comparison is base (0782f86) 91.04% compared to head (fbae297) 91.24%.
Report is 1 commits behind head on master.

Files Patch % Lines
tools/python_api/src_cpp/pandas/pandas_bind.cpp 84.61% 6 Missing ⚠️
tools/python_api/src_cpp/pandas/pandas_scan.cpp 94.66% 4 Missing ⚠️
src/common/types/value/value.cpp 70.00% 3 Missing ⚠️
src/planner/plan/plan_read.cpp 85.71% 3 Missing ⚠️
tools/python_api/src_cpp/numpy/numpy_type.cpp 95.91% 2 Missing ⚠️
src/common/table_type.cpp 0.00% 1 Missing ⚠️
src/function/cast_string_to_functions.cpp 0.00% 1 Missing ⚠️
.../include/function/table_functions/scan_functions.h 66.66% 1 Missing ⚠️
...operator/persistent/reader/csv/serial_csv_reader.h 50.00% 1 Missing ⚠️
...erator/persistent/reader/parquet/column_reader.cpp 0.00% 1 Missing ⚠️
... and 5 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2403      +/-   ##
==========================================
+ Coverage   91.04%   91.24%   +0.19%     
==========================================
  Files        1015     1022       +7     
  Lines       36012    36273     +261     
==========================================
+ Hits        32788    33096     +308     
+ Misses       3224     3177      -47     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@andyfengHKU andyfengHKU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at coverage report. Also remove src/parser/statement_visitor.cpp

src/common/string_utils.cpp Outdated Show resolved Hide resolved
tools/python_api/test/test_scan_pandas.py Show resolved Hide resolved
src/binder/bind/bind_reading_clause.cpp Show resolved Hide resolved
@acquamarin acquamarin merged commit 3cbef96 into master Nov 14, 2023
12 checks passed
@acquamarin acquamarin deleted the panda-reader branch November 14, 2023 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants