Ndarray UDF Support (ArrayCount, Unnest, etc.) #152
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Abdullahshah
changed the title
Abdullahshah
requested review from
xzdandy and
gaurav274
and removed request for
xzdandy
Use the right sql uri
2. remove more from response 3. add license info to interpreter.py
Added support for uploading a video from the client into a custom location in the server. The syntax is as follows: ``` UPLOAD INFILE 'data/ua_detrac/ua_detrac.mp4' PATH 'test_video.mp4'; LOAD DATA INFILE 'test_video.mp4' INTO MyVideo; SELECT id, data FROM MyVideo WHERE id < 5; ``` The prefix `/tmp` is added to the location before storing the video on the server. Appropriate changes have been made to the parser, planner, and executor. The UPLOAD statement is rewritten by the client as: `UPLOAD PATH 'test_video.mp4' BLOB b'AAAA......';` The server _only_ accepts upload command of the above syntax. The client is responsible for rewriting the command provided by the end-user into a syntax understood by the server.
gaurav274
requested changes
Jul 10, 2021
| class Dimension(Enum): | ||
| ANY_DIM = -1 | ||
|
|
||
|
|
src/udfs/ndarray_udfs/__init__.py
Outdated
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| """user defined functions operating on ndarrays""" |
| """ | ||
|
|
||
| def __call__(self, *args, **kwargs): | ||
| return self.exec(*args, **kwargs) |
src/udfs/ndarray_udfs/array_count.py
Outdated
| count_result = values.apply(lambda x: self.count_in_row(x[0], search_element), axis=1) | ||
|
|
||
| return pd.DataFrame({'count': count_result.values}) | ||
| # return pd.DataFrame({'id': count_result.index, 'count': count_result.values}) |
src/udfs/ndarray_udfs/unnest.py
Outdated
| res = self.xplode(inp, explode) | ||
| res = res.set_index(dummy_idx) | ||
| res.index.name = None | ||
| return res |
src/udfs/udf_bootstrap_queries.py
Outdated
| scores NDARRAY FLOAT32(10)) | ||
| TYPE Classification | ||
| IMPL 'src/udfs/fastrcnn_object_detector.py'; | ||
| """ |
gaurav274
approved these changes
Jul 12, 2021
|
Duplicate #163 |
xzdandy
pushed a commit
to gaurav274/Eva
that referenced
this pull request
Mar 19, 2022
Basic tutorial for using EVA - [x] Need to support Unnest to enable useful queries. georgia-tech-db#152 - [x] Disable concurrent queries for cursor correctness. georgia-tech-db#160 @xzdandy - [x] Upload fails because of missing permissions. georgia-tech-db#162 @pchunduri6 - [x] Better response messages from the server. georgia-tech-db#158
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes the addition of Ndarray UDF support to EVA
In order to add a Ndarray UDF:
CREATEUDF query toudf_bootstrap_queries. This query should have the input specifications, output specifications, type (Ndarray/Classification/etc) and the file location of where the udf is implementedudfs/ndarray_udfs/. Each UDF implementation here should implement theabstract_ndarray_udfs.pyclass which requires that theexeccall of the UDF have the input of a Pandas DataFrame (Ideally only 1-2 columns) and returns one single DataFrame with one column (no need to return anIDcolumn in most cases)This PR adds the
Array_CountandUnnestNdarray UDFs.Some other changes include:
udfs/udf_bootstrap_queries. This is to make it easier to organize all the UDF strings in one file rather than recreating them each time.Catalogfor adding any time of dimension type for NdarraaysStill In Progress:
https://georgia-tech-db.atlassian.net/wiki/spaces/EVA/overview#Detailed-Roadmap