[SUPPORT] Unable to filter hudi table in hive on partition column #1962
Projects
Comments
|
Did you set hive input format ? Also can you confirm you settings given here are set. |
|
Hi Sivabalan, when I try to set the given settings I get the following error in Hive CLI: 20/08/14 07:19:51 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify hive.input.format at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
Beeline version 3.1.3000.7.2.0.0-237 by Apache Hive |
|
how are you connecting to hive, is it beeline or some other means? if its beeline, you should be able to pass in configs as given in the quick start page. If by some other means, probably you have to find out how to set hive configs when you bring up the tool. Or probably you can paste the command you used to bring up hive(obfuscating any confidential info). we can try to help you out. |
|
also, is it that you are having issues with all columns or just few columns? |
|
I'm running Hive within Cloudera Data Platform Public Cloud. I connected to Hive using HUE and Hive CLI. Beeline Connection: [root@engineeringhub-master3 ~]# beeline --hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat --hiveconf hive.stats.autogather=false
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.3758356/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.3758356/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.3758356/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.2.0-1.cdh7.2.0.p0.3758356/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://engineeringhub-master3.mtag-fas.ft71-kqhq.cloudera.site:2181/default;httpPath=cliservice;principal=hive/_HOST@MTAG-FAS.FT71-KQHQ.CLOUDERA.SITE;serviceDiscoveryMode=zooKeeper;ssl=true;transportMode=http;zooKeeperNamespace=hiveserver2
20/08/14 13:14:12 [main]: INFO jdbc.HiveConnection: Connected to engineeringhub-master3.mtag-fas.ft71-kqhq.cloudera.site:10001
20/08/14 13:14:12 [main]: WARN jdbc.HiveConnection: Failed to connect to engineeringhub-master3.mtag-fas.ft71-kqhq.cloudera.site:10001
20/08/14 13:14:12 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
Error: Could not open client transport for any of the Server URI's in ZooKeeper: Failed to open new session: java.lang.IllegalArgumentException: Cannot modify hive.input.format at runtime. It is not in list of params that are allowed to be modified at runtime (state=08S01,code=0)
Beeline version 3.1.3000.7.2.0.0-237 by Apache Hive
beeline>Later on we would like to use JDBC-Connections from external tools like PowerBI or Dbeaver. I can query and filter all columns excluding the partiton columns: year, month, day, hour, minute. If I apply aggregations on the partition columns I can filter them. This is very odd. SELECT DISTINCT(minute), count(minute)
FROM greentech.iot_device_ro
GROUP BY minute
; but SELECT *
FROM greentech.iot_device_ro
WHERE minute=50
; returns an emtpy result list. Thank you for the quick response. |
|
@bhasudha / @bvaradar : do you folks have any pointers here. Looks like the input format is not getting set. Would that be the issue here? |
|
For the second case, Hive Metastore would be filtering out partitions and only return specific paths. I think there is some inconsistency between the path used in the filesystem and the one that is present in meta-store. @sassai : Sorry for the delay, Can you recursively list your hoodie data set and attach the output. Also please add the file contents of latest .commit or .deltacommit file . Also, add the output for one of the partition with location: |
|
Closing this due to inactivity |
|
@bvaradar: Sorry for the late reply. I was not able to investigate this issue further until now. In the meantime I updated Hudi to 0.6.0 to check if the issue still occurs. Unfortunately yes. I created a table (COPY_ON_WRITE) with testing data for further debugging. Please find the requested information below: Hudi data set: drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/.aux
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/.aux/.bootstrap
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/.aux/.bootstrap/.fileids
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/.aux/.bootstrap/.partitions
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/.temp
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 9133 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171431.commit
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171431.commit.requested
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 999 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171431.inflight
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 1169 2020-10-01 17:18 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171823.commit
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:18 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171823.commit.requested
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 380 2020-10-01 17:18 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001171823.inflight
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 2986 2020-10-01 17:34 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001173346.commit
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:34 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001173346.commit.requested
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 1653 2020-10-01 17:34 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/20201001173346.inflight
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/archived
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 228 2020-10-01 17:14 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/.hoodie/hoodie.properties
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10
drwxr-xr-x - 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 0 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 93 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/.hoodie_partition_metadata
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7095995 2020-10-01 17:34 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/08b5ed87-a749-4a82-a298-59071381dbc9-0_0-89-2258_20201001173346.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7096113 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/08b5ed87-a749-4a82-a298-59071381dbc9-0_8-25-180_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7126955 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/3c49d05f-8a9b-4365-8158-a32f879d674f-0_0-25-172_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7126790 2020-10-01 17:34 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/3c49d05f-8a9b-4365-8158-a32f879d674f-0_1-89-2259_20201001173346.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7144341 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/4f7d1a69-112a-42d1-b0ac-adf8de1e8dad-0_7-25-179_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7120178 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/669c9159-a795-40ac-9827-4551965e1750-0_3-25-175_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7197719 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/ab9c4c36-ea10-47bc-bf05-1555bf07c4ad-0_2-25-174_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7158006 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/ad939d4e-d8dc-4723-b5a5-8ec2c064f3e3-0_6-25-178_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7170312 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/c4c0b379-79e8-4856-9605-e56b1beb1b09-0_4-25-176_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7118844 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/f5fae4bc-396d-4ff6-8b89-01c08559cb50-0_5-25-177_20201001171431.parquet
-rw-r--r-- 1 3d88417a-c602-4b19-b581-ac7265074929 srv_tu_usecase2_producer 7156753 2020-10-01 17:15 abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1/f804937c-8de2-4692-90d9-4b485305d721-0_1-25-173_20201001171431.parquetLatest commit: {
"partitionToWriteStats" : {
"year=2020/month=10/day=1" : [ {
"fileId" : "08b5ed87-a749-4a82-a298-59071381dbc9-0",
"path" : "year=2020/month=10/day=1/08b5ed87-a749-4a82-a298-59071381dbc9-0_0-89-2258_20201001173346.parquet",
"prevCommit" : "20201001171431",
"numWrites" : 110569,
"numDeletes" : 0,
"numUpdateWrites" : 1,
"numInserts" : 0,
"totalWriteBytes" : 7095995,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "year=2020/month=10/day=1",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 7095995
}, {
"fileId" : "3c49d05f-8a9b-4365-8158-a32f879d674f-0",
"path" : "year=2020/month=10/day=1/3c49d05f-8a9b-4365-8158-a32f879d674f-0_1-89-2259_20201001173346.parquet",
"prevCommit" : "20201001171431",
"numWrites" : 111111,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 1,
"totalWriteBytes" : 7126790,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "year=2020/month=10/day=1",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 7126790
} ]
},
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"address_record\",\"namespace\":\"hoodie.address\",\"fields\":[{\"name\":\"id\",\"type\":[\"int\",\"null\"]},{\"name\":\"zipCode\",\"type\":[\"int\",\"null\"]},{\"name\":\"city\",\"type\":[\"string\",\"null\"]},{\"name\":\"street\",\"type\":[\"string\",\"null\"]},{\"name\":\"streetNumber\",\"type\":[\"int\",\"null\"]},{\"name\":\"uuid\",\"type\":[\"string\",\"null\"]},{\"name\":\"start_date\",\"type\":[\"string\",\"null\"]},{\"name\":\"end_date\",\"type\":[\"string\",\"null\"]},{\"name\":\"is_current\",\"type\":\"boolean\"},{\"name\":\"event_time\",\"type\":[\"string\",\"null\"]},{\"name\":\"year\",\"type\":\"int\"},{\"name\":\"month\",\"type\":\"int\"},{\"name\":\"day\",\"type\":\"int\"},{\"name\":\"its\",\"type\":\"string\"}]}"
},
"operationType" : "UPSERT",
"fileIdAndRelativePaths" : {
"08b5ed87-a749-4a82-a298-59071381dbc9-0" : "year=2020/month=10/day=1/08b5ed87-a749-4a82-a298-59071381dbc9-0_0-89-2258_20201001173346.parquet",
"3c49d05f-8a9b-4365-8158-a32f879d674f-0" : "year=2020/month=10/day=1/3c49d05f-8a9b-4365-8158-a32f879d674f-0_1-89-2259_20201001173346.parquet"
},
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 0,
"totalUpsertTime" : 11307,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesCompacted" : 0,
"totalLogFilesSize" : 0
}Hudi properties file: #Properties saved on Thu Oct 01 17:14:31 UTC 2020
#Thu Oct 01 17:14:31 UTC 2020
hoodie.table.name=address
hoodie.archivelog.folder=archived
hoodie.table.type=COPY_ON_WRITE
hoodie.table.version=1
hoodie.timeline.layout.version=1Describe table: +-----------------------------------+----------------------------------------------------+-----------------------+
| col_name | data_type | comment |
+-----------------------------------+----------------------------------------------------+-----------------------+
| _hoodie_commit_time | string | |
| _hoodie_commit_seqno | string | |
| _hoodie_record_key | string | |
| _hoodie_partition_path | string | |
| _hoodie_file_name | string | |
| id | int | |
| zipcode | int | |
| city | string | |
| street | string | |
| streetnumber | int | |
| uuid | string | |
| start_date | string | |
| end_date | string | |
| is_current | boolean | |
| event_time | string | |
| its | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| year | int | |
| month | int | |
| day | int | |
| | NULL | NULL |
| # Detailed Partition Information | NULL | NULL |
| Partition Value: | [2020, 10, 1] | NULL |
| Database: | nyc_taxi | NULL |
| Table: | address | NULL |
| CreateTime: | Thu Oct 01 17:15:51 UTC 2020 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Location: | abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1 | NULL |
| Partition Parameters: | NULL | NULL |
| | numFiles | 9 |
| | totalSize | 64289221 |
| | transient_lastDdlTime | 1601572551 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | NULL |
| InputFormat: | org.apache.hudi.hadoop.HoodieParquetInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-----------------------------------+----------------------------------------------------+-----------------------+ |
|
Update: Using |
|
@sassai : Location is set wrongly. | Location: | abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=1 | NULL | It should have been abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address only. Can you try dropping the table and hive syncing again ? |
|
@bvaradar: I dropped and synced the table again. Issue still exists unless I set Please find the table description with and without partition below. describe formatted address: +-------------------------------+----------------------------------------------------+-----------------------+
| col_name | data_type | comment |
+-------------------------------+----------------------------------------------------+-----------------------+
| _hoodie_commit_time | string | |
| _hoodie_commit_seqno | string | |
| _hoodie_record_key | string | |
| _hoodie_partition_path | string | |
| _hoodie_file_name | string | |
| id | int | |
| zipcode | int | |
| city | string | |
| street | string | |
| streetnumber | int | |
| uuid | string | |
| start_date | string | |
| end_date | string | |
| is_current | boolean | |
| event_time | string | |
| its | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| year | int | |
| month | int | |
| day | int | |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | nyc_taxi | NULL |
| OwnerType: | USER | NULL |
| Owner: | srv_tu_usecase2_producer | NULL |
| CreateTime: | Tue Oct 06 07:06:03 UTC 2020 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | EXTERNAL | TRUE |
| | bucketing_version | 2 |
| | discover.partitions | true |
| | last_commit_time_sync | 20201006065754 |
| | numFiles | 82 |
| | numPartitions | 1 |
| | numRows | 0 |
| | rawDataSize | 0 |
| | totalSize | 642825967 |
| | transient_lastDdlTime | 1601967963 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | NULL |
| InputFormat: | org.apache.hudi.hadoop.HoodieParquetInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-------------------------------+----------------------------------------------------+-----------------------+describe formatted address partition(year=2020,month=10,day=6): +-----------------------------------+----------------------------------------------------+-----------------------+
| col_name | data_type | comment |
+-----------------------------------+----------------------------------------------------+-----------------------+
| _hoodie_commit_time | string | |
| _hoodie_commit_seqno | string | |
| _hoodie_record_key | string | |
| _hoodie_partition_path | string | |
| _hoodie_file_name | string | |
| id | int | |
| zipcode | int | |
| city | string | |
| street | string | |
| streetnumber | int | |
| uuid | string | |
| start_date | string | |
| end_date | string | |
| is_current | boolean | |
| event_time | string | |
| its | string | |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| year | int | |
| month | int | |
| day | int | |
| | NULL | NULL |
| # Detailed Partition Information | NULL | NULL |
| Partition Value: | [2020, 10, 6] | NULL |
| Database: | nyc_taxi | NULL |
| Table: | address | NULL |
| CreateTime: | Tue Oct 06 07:06:04 UTC 2020 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Location: | abfs://xxx@xxx.dfs.core.windows.net/data/hudi/batch/tables/nyc_taxi/address/year=2020/month=10/day=6 | NULL |
| Partition Parameters: | NULL | NULL |
| | numFiles | 82 |
| | totalSize | 642825967 |
| | transient_lastDdlTime | 1601967964 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | NULL |
| InputFormat: | org.apache.hudi.hadoop.HoodieParquetInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | -1 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| Storage Desc Params: | NULL | NULL |
| | serialization.format | 1 |
+-----------------------------------+----------------------------------------------------+-----------------------+ |
|
@bvaradar : guess you missed to follow up on this thread. can you check it out and respond when you can. |
|
I think we should doc better, when |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem you faced
I'm running a spark structured streaming application that reads data from kafka and saves it to a partitioned Hudi MERGE_ON_READ table. Hive sync is enabled and I'm able to query the table with the Hive CLI, e.g.:
SELECT * FROM iot_device_ro LIMIT 5;
However if I want to apply a filter on the partition colum the result is empty:
SELECT * FROM iot_device_ro WHERE day=13 LIMIT 10;
To Reproduce
Steps to reproduce the behavior:
Hudi Datasource Configuration in Spark:
Hudi Table Description in Hive:
Hive Partitions created by HudiHiveSync:
Additional table information:
Strangely enough if I run the same query in Spark everything works as expected.
Expected behavior
Filter on partition columns can be applied on Hudi tables in Hive CLI.
Environment Description
Hudi version : 0.5.3
Spark version : 2.4.0
Hive version : 3.1
Hadoop version : 3
Storage (HDFS/S3/GCS..) : ADLS
Running on Docker? (yes/no) : no
What am I missing? Any help is very much appreciated!
Thank you in advance!
The text was updated successfully, but these errors were encountered: