[iceberg] [native] Add support for V1 tables #21584

imjalpreet · 2023-12-20T23:41:38Z

Description

This PR includes the changes required to convert the Presto Iceberg plan and splits into the Velox plan and splits.

Support converting Presto Iceberg Plan to Velox Plan
Support converting Presto Iceberg Split to Velox Split
Register the iceberg catalog as part of Hive Connectors

Motivation and Context

facebookincubator/velox#5977

Impact

With these changes, we will be able to read iceberg v1 tables from Prestissimo

Test Plan

Successfully ran all TPC-H queries with iceberg tables (both partitioned and non-partitioned)

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Introduce Iceberg Connector in Prestissimo

presto-native-execution/presto_cpp/presto_protocol/Connectors.cpp

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

aditi-pandit

Thanks @imjalpreet.

Would you be able to split out the protocol changes from the plan conversion changes in this PR ? That would be easier to review and maintain.

imjalpreet · 2024-01-03T00:27:22Z

Would you be able to split out the protocol changes from the plan conversion changes in this PR ? That would be easier to review and maintain.

@aditi-pandit can you let me know how you want to split the changes? Did you mean in a separate PR? In this PR I had kept separate commits for the protocol and Presto to Velox plan conversion changes.

aditi-pandit · 2024-01-03T04:47:58Z

Would you be able to split out the protocol changes from the plan conversion changes in this PR ? That would be easier to review and maintain.

@aditi-pandit can you let me know how you want to split the changes? Did you mean in a separate PR? In this PR I had kept separate commits for the protocol and Presto to Velox plan conversion changes.

@imjalpreet : Yeah, would be great to have a single PR with just the changes you have in commit Add Iceberg Connector to parse Java Iceberg Plan Fragment. After that checked-in its easier to look at how the protocol pieces are mapped into Prestissimo.

majetideepak · 2024-01-03T11:11:56Z

Please separate the Prestissimo bits into their own PR.

yingsu00 · 2024-01-25T14:22:54Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

@@ -852,6 +853,11 @@ std::vector<std::string> PrestoServer::registerConnectors(
      PRESTO_STARTUP_LOG(INFO) << "Registering catalog " << catalogName
                               << " using connector " << connectorName;

+      // TODO: Temporary Change to use HiveDataSource for Iceberg Connector
+      if (connectorName == "iceberg") {


Can this really happen? Is the connectorName not from presto-native-execution/etc/catalog? There is no iceberg.properties there and in hive.properties there is only connector.name=hive

Yes, we would need to add the iceberg catalog properties file in presto-native-execution/etc/catalog if someone wants to read iceberg tables

If we want to add a default file, I can push one.

@imjalpreet yes please add one

BTW I think we should directly identify the connector name as hive in iceberg.properties. This catalog property file is for Presto Native, which doesn't have Iceberg connector, but just Hive connector.

connector.name=hive

presto-native-execution/presto_cpp/presto_protocol/Connectors.cpp

yingsu00 · 2024-01-25T14:52:10Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

@@ -142,6 +142,18 @@ std::shared_ptr<connector::ColumnHandle> toColumnHandle(
        toRequiredSubfields(hiveColumn->requiredSubfields));
  }

+  if (auto icebergColumn =
+          dynamic_cast<const protocol::IcebergColumnHandle*>(column)) {
+    // TODO(imjalpreet): Modify 'hiveType' argument of the 'HiveColumnHandle'


What needs to be done on hiveType?

In the case of the hive connector, HiveColumnHandle(Presto) also has a parameter of type HiveType which is being used while creating the HiveColumnHandle velox object. We don't have the same in IcebergColumnHandle yet.

The prestissimo change was added as part of this commit in the case of hive connector: 6570b00

majetideepak · 2024-02-14T12:33:13Z

@imjalpreet is this ready for review?

majetideepak · 2024-02-14T12:33:45Z

Can you update the description?

imjalpreet · 2024-02-14T12:53:28Z

@majetideepak Yes, this is ready for review. I have updated the description as well.

yingsu00 · 2024-01-30T13:55:07Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

@@ -852,6 +853,11 @@ std::vector<std::string> PrestoServer::registerConnectors(
      PRESTO_STARTUP_LOG(INFO) << "Registering catalog " << catalogName
                               << " using connector " << connectorName;

+      // TODO: Temporary Change to use HiveDataSource for Iceberg Connector
+      if (connectorName == "iceberg") {


BTW I think we should directly identify the connector name as hive in iceberg.properties. This catalog property file is for Presto Native, which doesn't have Iceberg connector, but just Hive connector.

connector.name=hive

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

yingsu00 · 2024-02-14T14:57:10Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

+    }
+    std::unordered_map<std::string, std::string> customSplitInfo;
+
+    std::shared_ptr<std::string> extraFileInfo;


yingsu00 · 2024-02-14T14:57:17Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

+
+    std::shared_ptr<std::string> extraFileInfo;
+
+    std::optional<int> tableBucketNumber;


yingsu00 · 2024-02-14T14:58:02Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

+            partitionKeys,
+            tableBucketNumber,
+            customSplitInfo,
+            extraFileInfo),


just change to nullptr

yingsu00 · 2024-02-14T14:58:26Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

+            icebergSplit->start,
+            icebergSplit->length,
+            partitionKeys,
+            tableBucketNumber,


Use Presto IcebergColumnHandle to create Velox HiveColumnHandle Use Presto IcebergTableLayoutHandle/IcebergTableHandle to create Velox HiveTableHandle

yingsu00

@tdcmeehan @majetideepak @aditi-pandit @mbasmanova Do you want to review this PR again?

majetideepak · 2024-02-15T16:41:47Z

Successfully ran all TPC-H queries with iceberg tables (both partitioned and non-partitioned)

@imjalpreet, @yingsu00 can you share some details of the tests that were run? What scale factor was used? How many nodes? Was there any comparison made with the Hive Connector?

majetideepak · 2024-02-15T16:43:47Z

I believe only Copy-on-Write tables are supported correct? Can you clarify the scope?

presto-native-execution/etc/catalog/iceberg.properties

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

presto-native-execution/presto_cpp/main/types/PrestoToVeloxSplit.cpp

Co-authored-by: Jalpreet Singh Nanda (:imjalpreet) <jalpreetnanda@gmail.com>

imjalpreet · 2024-02-15T19:07:51Z

can you share some details of the tests that were run? What scale factor was used? How many nodes? Was there any comparison made with the Hive Connector?

@majetideepak Currently, we have only done correctness verifications with smaller scale factors on local machines. We are in the process of testing it with sf1k and sf10k and working on generating the datasets. If you would like to see performance comparison for small scale factor, I can share that or we can wait for the tests with sf1k and sf10k.

I believe only Copy-on-Write tables are supported correct? Can you clarify the scope?

Yes, currently this PR would enable Prestissimo to be able to read iceberg v1 tables. Since v2 is not fully supported yet, merge on read tables won't be supported with this PR since they are only available with iceberg v2 tables.

yingsu00 · 2024-02-16T07:11:31Z

@imjalpreet, @yingsu00 can you share some details of the tests that were run? What scale factor was used? How many nodes? Was there any comparison made with the Hive Connector?

@majetideepak Reetika has a branch with the TPCH/DS tiny e2e tests here https://github.com/agrawalreetika/prestodb/commits/icebergPrestissimo-tests-tpch-v1/ . Since it depends on this PR, we will submit a PR after this one is merged.

majetideepak · 2024-02-16T07:15:45Z

presto-native-execution/etc/catalog/iceberg.properties

@@ -0,0 +1,4 @@
+# The Presto "iceberg" catalog is handled by the hive connector in Presto native execution.
+connector.name=hive


I think we can add a mapping from iceberg name to hive on the Velox side. Similar to how hive-hadoop2 is mapped. Not a blocker for now.

Created a PR here facebookincubator/velox#8765

majetideepak

Thanks, @imjalpreet

tdcmeehan · 2024-02-20T15:37:05Z

Just in case: if this doesn't fail hard on V2 tables, can we add a small followup to do that until V2 support is finished?

imjalpreet · 2024-02-21T08:30:38Z

Just in case: if this doesn't fail hard on V2 tables, can we add a small followup to do that until V2 support is finished?

@tdcmeehan Yes, sure. The follow-up PR for V2 support should also be ready soon. But I will raise the above fail-fast change in the meantime.

imjalpreet requested a review from yingsu00 December 20, 2023 23:41

imjalpreet self-assigned this Dec 20, 2023

imjalpreet requested review from a team as code owners December 20, 2023 23:41

imjalpreet requested a review from presto-oss December 20, 2023 23:41

imjalpreet marked this pull request as draft December 20, 2023 23:41

yingsu00 added the iceberg Apache Iceberg related label Dec 21, 2023

yingsu00 reviewed Dec 25, 2023

View reviewed changes

presto-native-execution/presto_cpp/presto_protocol/Connectors.cpp Outdated Show resolved Hide resolved

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp Show resolved Hide resolved

aditi-pandit reviewed Dec 28, 2023

View reviewed changes

imjalpreet mentioned this pull request Jan 9, 2024

Refactor Java Hive and Iceberg Connector for Prestissimo Iceberg Connector #21662

Merged

6 tasks

imjalpreet force-pushed the icebergPrestissimoChanges branch 2 times, most recently from 62163c3 to 70146b3 Compare January 24, 2024 11:11

imjalpreet marked this pull request as ready for review January 24, 2024 12:22

yingsu00 reviewed Jan 25, 2024

View reviewed changes

yingsu00 mentioned this pull request Jan 30, 2024

[native] Add support for Iceberg connector in presto_protocol #21765

Merged

6 tasks

imjalpreet force-pushed the icebergPrestissimoChanges branch 2 times, most recently from f0ff681 to 2e229ac Compare February 5, 2024 10:32

imjalpreet force-pushed the icebergPrestissimoChanges branch 8 times, most recently from 5f71cfe to 51050fc Compare February 14, 2024 07:52

yingsu00 reviewed Feb 14, 2024

View reviewed changes

[native] Support converting Presto Iceberg Plan to Velox Plan

de7170a

Use Presto IcebergColumnHandle to create Velox HiveColumnHandle Use Presto IcebergTableLayoutHandle/IcebergTableHandle to create Velox HiveTableHandle

imjalpreet force-pushed the icebergPrestissimoChanges branch from 51050fc to f9dfb73 Compare February 15, 2024 08:28

yingsu00 approved these changes Feb 15, 2024

View reviewed changes

yingsu00 requested review from mbasmanova and majetideepak February 15, 2024 10:36

majetideepak changed the title ~~[iceberg] [native] Introduce Iceberg Connector in Prestissimo~~ [iceberg] [native] Add support for Iceberg Connector in Prestissimo Feb 15, 2024

majetideepak reviewed Feb 15, 2024

View reviewed changes

[native] Support converting Presto Iceberg Split to Velox Split

96f4b50

Co-authored-by: Jalpreet Singh Nanda (:imjalpreet) <jalpreetnanda@gmail.com>

imjalpreet force-pushed the icebergPrestissimoChanges branch from f9dfb73 to 1ec8e47 Compare February 15, 2024 18:59

majetideepak changed the title ~~[iceberg] [native] Add support for Iceberg Connector in Prestissimo~~ [iceberg] [native] Add support for Iceberg Connector (V1) in Prestissimo Feb 16, 2024

majetideepak changed the title ~~[iceberg] [native] Add support for Iceberg Connector (V1) in Prestissimo~~ [iceberg] [native] Add support for Iceberg Connector (V1) Feb 16, 2024

majetideepak changed the title ~~[iceberg] [native] Add support for Iceberg Connector (V1)~~ [iceberg] [native] Add support for Iceberg Connector V1 Feb 16, 2024

majetideepak changed the title ~~[iceberg] [native] Add support for Iceberg Connector V1~~ [iceberg] [native] Add support for V1 tables Feb 16, 2024

majetideepak reviewed Feb 16, 2024

View reviewed changes

imjalpreet added 2 commits February 16, 2024 13:25

[native] Register iceberg catalog as part of Hive Connectors

162dd51

[native] Remove defunct config from hive catalog

7aa5ab1

imjalpreet force-pushed the icebergPrestissimoChanges branch from 1ec8e47 to 7aa5ab1 Compare February 16, 2024 08:06

majetideepak approved these changes Feb 16, 2024

View reviewed changes

yingsu00 merged commit e5ba4bb into prestodb:master Feb 19, 2024
59 checks passed

wanglinsong mentioned this pull request May 1, 2024

Add release notes for 0.287 #22647

Merged

48 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[iceberg] [native] Add support for V1 tables #21584

[iceberg] [native] Add support for V1 tables #21584

imjalpreet commented Dec 20, 2023 •

edited

Loading

aditi-pandit left a comment

imjalpreet commented Jan 3, 2024

aditi-pandit commented Jan 3, 2024

majetideepak commented Jan 3, 2024

yingsu00 Jan 25, 2024

imjalpreet Jan 25, 2024

imjalpreet Jan 25, 2024

yingsu00 Jan 25, 2024

yingsu00 Jan 30, 2024

yingsu00 Jan 25, 2024

imjalpreet Jan 25, 2024

majetideepak commented Feb 14, 2024

majetideepak commented Feb 14, 2024

imjalpreet commented Feb 14, 2024

yingsu00 Jan 30, 2024

yingsu00 Feb 14, 2024

yingsu00 Feb 14, 2024

yingsu00 Feb 14, 2024

yingsu00 Feb 14, 2024

yingsu00 left a comment

majetideepak commented Feb 15, 2024

majetideepak commented Feb 15, 2024

imjalpreet commented Feb 15, 2024 •

edited

Loading

yingsu00 commented Feb 16, 2024

majetideepak Feb 16, 2024

majetideepak Feb 16, 2024

majetideepak left a comment

tdcmeehan commented Feb 20, 2024

imjalpreet commented Feb 21, 2024


		std::shared_ptr<std::string> extraFileInfo;

		std::optional<int> tableBucketNumber;

		@@ -0,0 +1,4 @@
		# The Presto "iceberg" catalog is handled by the hive connector in Presto native execution.
		connector.name=hive

[iceberg] [native] Add support for V1 tables #21584

[iceberg] [native] Add support for V1 tables #21584

Conversation

imjalpreet commented Dec 20, 2023 • edited Loading

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

aditi-pandit left a comment

Choose a reason for hiding this comment

imjalpreet commented Jan 3, 2024

aditi-pandit commented Jan 3, 2024

majetideepak commented Jan 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

majetideepak commented Feb 14, 2024

majetideepak commented Feb 14, 2024

imjalpreet commented Feb 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yingsu00 left a comment

Choose a reason for hiding this comment

majetideepak commented Feb 15, 2024

majetideepak commented Feb 15, 2024

imjalpreet commented Feb 15, 2024 • edited Loading

yingsu00 commented Feb 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

majetideepak left a comment

Choose a reason for hiding this comment

tdcmeehan commented Feb 20, 2024

imjalpreet commented Feb 21, 2024

imjalpreet commented Dec 20, 2023 •

edited

Loading

imjalpreet commented Feb 15, 2024 •

edited

Loading