[ADBDEV-5595] - Add support for new types in PXF #100
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xardazzzzzz
commented
Jun 7, 2024
xardazzzzzz
commented
- Refactor read and write parquet process
- Support time, decimal, uuid, interval and their respective array types
- Support list of lists for reading
…ternal readable tables
- Add integration tests for fragments distribution; - Modify test environment adding additional segment host; - Fix other integration tests because of the changing test environment.
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Refactor read and write parquet process - Support time, decimal, uuid, interval and their respective array types - Support list of lists for reading
- Fix integration tests
- Add filtering
RomaZe
reviewed
Jun 13, 2024
server/pxf-api/src/main/java/org/greenplum/pxf/api/filter/SupportedOperatorPruner.java
Outdated
Show resolved
Hide resolved
server/pxf-api/src/main/java/org/greenplum/pxf/api/GreenplumDateTime.java
Show resolved
Hide resolved
server/pxf-s3/src/main/java/org/greenplum/pxf/plugins/s3/S3SelectQueryBuilder.java
Show resolved
Hide resolved
- Fix review comments
RomaZe
previously approved these changes
Jun 14, 2024
- Add tests - Add binary various parsing - Add bson to dependencies
- Add tests - Add binary various parsing - Add bson to dependencies
Refactor ListConstToStr function The original implementation of the list_const_to_str (ListConstToStr) function was intended to extract and format values from array constants into a string buffer, supporting data types like int2[], int4[], int8[], and text[]. It checked for null constants, logged exceptions, extracted the array, processed each supported type with specific code blocks to deconstruct the array, converted each element to a string, and appended the formatted data to a buffer. However, the approach led to code duplication, maintenance challenges, and reduced readability due to scattered, repetitive logic. The refactored function consolidated common logic into reusable, unified steps. It retrieves type information and deconstructs arrays using a single, generalized procedure, eliminating repetitive code. This streamlined array processing is supported by getTypeOutputInfo, which extracts output function from the catalog, facilitating consistent conversion across all data types with OidOutputFunctionCall. Additionally, the refactored function simplifies its logic by centralizing the handling of array elements and reducing the switch case complexity. This enhances readability and maintainability, making it easier to understand and modify. Adding new array types now involves minimal changes, creating a more elegant, less error-prone codebase.
- Fix handling of arrays. Currently they're not supported on parquet library level. Trying <array-column> = array[val1] leads to 0 match
iamlapa
reviewed
Jun 28, 2024
...f-hdfs/src/main/java/org/greenplum/pxf/plugins/hdfs/parquet/ParquetTypeConverterFactory.java
Outdated
Show resolved
Hide resolved
.../main/java/org/greenplum/pxf/plugins/hdfs/parquet/converters/BinaryParquetTypeConverter.java
Outdated
Show resolved
Hide resolved
- Fix review comments
iamlapa
previously approved these changes
Jul 3, 2024
RomaZe
reviewed
Jul 10, 2024
|
|
||
| @Slf4j | ||
| @UtilityClass | ||
| public class ParquetIntervalUtilities { |
There was a problem hiding this comment.
Magic class. Shouldn't we add unit test for this class?
| @@ -1,5 +1,7 @@ | |||
| package org.greenplum.pxf.api.filter; | |||
|
|
|||
| import org.greenplum.pxf.api.io.DataType; | |||
| import org.apache.parquet.schema.Type; | ||
| import org.greenplum.pxf.api.error.UnsupportedTypeException; |
- Optimize imports
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.