-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature-wip](iceberg) Step3: Support query iceberg external table #8179
Conversation
<groupId>org.apache.avro</groupId> | ||
<artifactId>avro</artifactId> | ||
<version>1.10.1</version> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add it to fe/pom.xml
and reference it here.
IcebergCatalog icebergCatalog = IcebergCatalogMgr.getCatalog(icebergProperty); | ||
try { | ||
this.icebergTable = icebergCatalog.loadTable(TableIdentifier.of(icebergDb, icebergTbl)); | ||
LOG.info("finished to load table: {}", name); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOG.info("finished to load table: {}", name); | |
LOG.info("finished to load iceberg table: {}", name); |
this.icebergTable = icebergCatalog.loadTable(TableIdentifier.of(icebergDb, icebergTbl)); | ||
LOG.info("finished to load table: {}", name); | ||
} catch (Exception e) { | ||
LOG.warn("failed to load table {} from {}", name, icebergProperty.getHiveMetastoreUris(), e); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOG.warn("failed to load table {} from {}", name, icebergProperty.getHiveMetastoreUris(), e); | |
LOG.warn("failed to load iceberg table {} from {}", name, icebergProperty.getHiveMetastoreUris(), e); |
targetTable = icebergTable; | ||
} | ||
|
||
@Override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a example in comment about the format for returned url
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And what if the iceberg is located on AWS S3 or other oss, instead of hdfs?
fileStatusesList.add(fileStatuses); | ||
filesAdded += fileStatuses.size(); | ||
for (TBrokerFileStatus fstatus : fileStatuses) { | ||
LOG.info("Add file status is {}", fstatus); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOG.info("Add file status is {}", fstatus); | |
LOG.debug("Add file status is {}", fstatus); |
@@ -17,8 +17,8 @@ | |||
|
|||
package org.apache.doris.qe; | |||
|
|||
import avro.shaded.com.google.common.collect.Maps; | |||
import avro.shaded.com.google.common.collect.Sets; | |||
import com.google.common.collect.Maps; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pay attention to import order
Change-Id: I2b7a4ae72df4e24e75d7151ab9644e6adc04d58d
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
1. Add Iceberg scan node 2. Add Iceberg/Hive table type in thrift 3. Support querying Iceberg tables of format types `parquet` and `orc`
Proposed changes
Issue Number: close #8178
Problem Summary:
parquet
andorc
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...