-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimized parquet reader does not perform overflow checks for decimals without rescaling #16355
Comments
@397090770 could you share more details about what tool you're using to generate this file ? |
Just as a point of comparison, I ran this same file through Apache Hive and Apache Spark. |
HI @raunaqmorarka, The data is generated by my own program, there is no limit to the size of the data range.
|
I have a hive table with the following structure:
The meta information of the parquet file is as follows
![image](https://user-images.githubusercontent.com/5170878/222628929-d6c9ad98-b352-45eb-bac9-73970fa27399.png)
and I turning on the batch reader of parquet (
set session hive.parquet_optimized_reader_enabled=true
), the execution results are as followsif I turn off the batch reader of parquet, another exception occurred:
It seems that the batch reader of parquet does not detect the validity of the data.
![image](https://user-images.githubusercontent.com/5170878/222629414-cc6197cd-06e7-4577-a0a8-186517b7ef96.png)
If necessary, the following is the download address of the test file:
https://trinodb.slack.com/files/U049BH76GP5/F04S3BNRRGV/parquet-1m
The text was updated successfully, but these errors were encountered: