Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add register_partition, unregister_partition procedures #2692

Closed
wants to merge 3 commits into from
Closed

Add register_partition, unregister_partition procedures #2692

wants to merge 3 commits into from
+686 −25

Conversation

wendigo
Copy link
Contributor

@wendigo wendigo commented Jan 31, 2020

Hive: Add partition manipulation procedures:

  • system.register_partition(schema_name, table_name, location, partition_columns, partition_values)

    Registers existing location as a partition in metastore for specified table.

  • system.unregister_partition(schema_name, table_name, partition_columns, partition_values)

    Unregisters given, existing partition in metastore for specified table.
    The referenced partition data location is not deleted.

@cla-bot cla-bot bot added the cla-signed label Jan 31, 2020
@findepi findepi requested a review from aalbu January 31, 2020 12:32
@wendigo wendigo force-pushed the hive_add_partition branch 4 times, most recently from 1c69c94 to 98ccb99 Compare January 31, 2020 15:42
aalbu
aalbu reviewed Jan 31, 2020
View reviewed changes
presto-hive/src/main/java/io/prestosql/plugin/hive/RegisterPartitionProcedure.java Outdated Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/RegisterPartitionProcedure.java Show resolved Hide resolved
presto-product-tests/src/main/java/io/prestosql/tests/hive/TestHivePartitionProcedures.java Outdated Show resolved Hide resolved
presto-product-tests/src/main/java/io/prestosql/tests/hive/TestHivePartitionProcedures.java Outdated Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/metastore/file/FileHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive/src/main/java/io/prestosql/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/RegisterPartitionProcedure.java Outdated Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/UnregisterPartitionProcedure.java Outdated Show resolved Hide resolved
@wendigo wendigo force-pushed the hive_add_partition branch 6 times, most recently from 3df9ff4 to f94b928 Compare February 3, 2020 15:05
@findepi findepi changed the title Hive add partition Add register_partition, unregister_partition procedures Feb 4, 2020
findepi
findepi requested changes Feb 4, 2020
View reviewed changes
presto-hive/src/main/java/io/prestosql/plugin/hive/CreateEmptyPartitionProcedure.java Outdated Show resolved Hide resolved
presto-docs/src/main/sphinx/connector/hive.rst Show resolved Hide resolved
presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved
presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved
presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/RegisterPartitionProcedure.java Outdated Show resolved Hide resolved
...to-hive/src/main/java/io/prestosql/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive/src/main/java/io/prestosql/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved
...to-hive/src/main/java/io/prestosql/plugin/hive/metastore/SemiTransactionalHiveMetastore.java Outdated Show resolved Hide resolved
presto-hive/src/test/java/io/prestosql/plugin/hive/TestHiveIntegrationSmokeTest.java Outdated Show resolved Hide resolved
@wendigo wendigo force-pushed the hive_add_partition branch 5 times, most recently from 06cedb5 to 0fec4d2 Compare February 5, 2020 11:04
@wendigo
Copy link
Contributor Author

wendigo commented Feb 5, 2020

Failed due to: 2020-02-05T12:03:40.6380315Z java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.testcontainers.containers.ContainerLaunchException: Container startup failed

@wendigo wendigo force-pushed the hive_add_partition branch 8 times, most recently from 56d0033 to 29dfb6d Compare February 5, 2020 20:37
findepi
findepi approved these changes Feb 6, 2020
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some editorial fixups. Thought this will more efficient this way.

presto-hive/src/main/java/io/prestosql/plugin/hive/metastore/file/FileHiveMetastore.java Show resolved Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/RegisterPartitionProcedure.java Show resolved Hide resolved
@wendigo wendigo force-pushed the hive_add_partition branch 2 times, most recently from 1c1e215 to 2e66901 Compare February 6, 2020 11:23
@wendigo
Copy link
Contributor Author

wendigo commented Feb 6, 2020

Failed due to #2616

@findepi
Copy link
Member

findepi commented Feb 6, 2020 

2020-02-06T12:10:50.7843778Z ptl-tests           | 2020-02-06 17:55:50 INFO: FAILURE     /    io.prestosql.tests.hive.TestHivePartitionProcedures.testRegisterPartitionFromAnyLocation (Groups: smoke, hive_partitioning) took 2.8 seconds
2020-02-06T12:10:50.7844184Z ptl-tests           | 2020-02-06 17:55:50 SEVERE: Failure cause:
2020-02-06T12:10:50.7844495Z ptl-tests           | java.lang.AssertionError: 
2020-02-06T12:10:50.7844887Z ptl-tests           | Actual and expected have the same elements but not in the same order, at index 0 actual element was:
2020-02-06T12:10:50.7845177Z ptl-tests           |   <42>
2020-02-06T12:10:50.7845462Z ptl-tests           | whereas expected element was:
2020-02-06T12:10:50.7845733Z ptl-tests           |   <1>
2020-02-06T12:10:50.7845973Z ptl-tests           | 
2020-02-06T12:10:50.7846379Z ptl-tests           | 	at io.prestosql.tests.hive.TestHivePartitionProcedures.testRegisterPartitionFromAnyLocation(TestHivePartitionProcedures.java:196)
2020-02-06T12:10:50.7846729Z ptl-tests           | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

org.assertj.core.api.IterableAssert#containsExactly checks order too, even for assertThat(set)

@findepi
Copy link
Member

findepi commented Feb 7, 2020

Merged as 783cb68, thanks!

@findepi findepi closed this Feb 7, 2020
@findepi findepi added this to the 330 milestone Feb 7, 2020
@findepi findepi added the enhancement New feature or request label Feb 7, 2020
@findepi findepi mentioned this pull request Feb 7, 2020
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aalbu aalbu aalbu left review comments

@findepi findepi findepi approved these changes

Assignees
No one assigned
Labels
cla-signed enhancement New feature or request
Milestone
330
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@wendigo @findepi @aalbu