Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
Summary: Enable explicit table-partitioning (i.e. using the `PARTITION BY` clause) in YSQL. See this architecture doc: https://github.com/yugabyte/yugabyte-db/blob/master/architecture/design/ysql-row-level-partitioning.md for more details. Test Plan: Adds new test: org.yb.pgsql.TestPgRegressPartitions containing the following regress tests: yb_pg_partition_prune yb_pg_partition_join yb_pg_partition_aggregate yb_pg_partition_update Individual changes and fixes are detailed below: * removed all the modifications made to the grammar that disable any partitions related syntax. * Fixed a bug where StoreCatalogInheritance was called twice during table creation. This causes a bug in the following manner: i) Child table is created on the heap, but no partition information about it has been updated in the catalog yet. ii) StoreCatalogInheritance is called, creating an entry for it in pg_inherits, but no information about its partition bounds is stored in pg_class iii) When StorePartitionBound is called, postgres tries to check whether the parent table has any child with an overlapping range iv) POstgres sees the newly created child table in pg_inherits v) However, when it tries to look up its bounds information in the catalog, there is understandably no bounds information for the new table (as we haven't updated it yet). An assertion fires, stating that there is a child table with no bounds information. To fix this issue, removed the extraneous StoreCatalogInheritance call. Also noticed that postgres 11.2 source code actually does not have this extraneous call. * YBPreloadRelCache fix Today YBPreloadRelCache during initialization, has two phases: 1) Scan pg_class table 2) Scan pg_attribute table This means the first scan populates some relations into the cache without their corresponding attribute information. This is usually okay, but this breaks once partitions are enabled. This is because, in the first phase, there was a special handling for partitioned tables that tried to update the partition key and bounds information. This requires the attribute info for pg_partitioned_table and pg_type catalog tables. However, since we were still in the first phase of YBPreloadRelCache, we will hit assertion here as these catalog tables will be present in the cache without any attribute information. To fix this, this patch does not update the partition information in the first phase. Since we need the attribute info populated for multiple catalog tables, and we cannot be sure whether these info were all populated when we try to process the partitioned table in the second phase, this patch creates a third phase. In the third phase, we scan the pg_partitioned_table, and update the cache for any partitioned tables with partition information. * For queries on partitioned tables, when indices were created on the tables, any join queries where failing. This is because postgres was picking a strategy MergeAppend to combine values from multiple child tables. MergeAppend expects the input streams to be merged to be sorted. However, in YB, by default, any single column indices are hash based and unordered. Still, MergeAppend was treating the values returned by IndexScans as ordered. This is because we try to pick hash index if there is an equality comparison in the equivalence class. However, the corresponding pathkey is created with strategy BTLessStrategyNumber or BTGreaterStrategyNumber always depending on the value passed for reverse_sort. Thus, when hash indices were picked, the merge append would assume that the output returned by this pathkey was already sorted appropriately. Fixed this by ensuring that when we pick a hash index, the corresponding pathkey should always have strategy BTEqualStrategyNumber. This way, the output from this pathkey is not considered to be ordered, and partition queries start returning correct results. * Add tests taken from partition_join.sql and partition_aggregate.sql in two new files yb_partition_join.sql and yb_partition_aggregate.sql. These files contain the same queries as the original postgres tests, but queries of the form "EXPLAIN.." and "ALTER TABLE" have been omitted as they are expected to fail. All other queries testing partition aggregates and partition joins work as expected and are included in the new files. * For partitioned tables, when the partition key of a row is updated such that the partition constraint is no longer satisfied, PostgreSQL converts the operation into 2 operations - a DELETE from the current partition, and an INSERT into the new partition, however this is not a transaction. The check for row movement, and conversion into a DELETE+INSERT was disabled for Yugabyte tables, so the update was returning success without actually doing anything. In this patch, returning failure if partition constraints are being violated. In a later patch, this row movement will be performed as a transaction between the two partition tables. Additionally there were a few bugs in the UPDATE path owing to the fact that there were some occurrences of FirstLowInvalidHeapAttributeNumber were not replaced by the YB equivalent in the partitions code path. Fixed these bugs and added more tests in the update code path in this commit.
- Loading branch information
Showing
32 changed files
with
9,665 additions
and
427 deletions.
There are no files selected for viewing
33 changes: 33 additions & 0 deletions
33
java/yb-pgsql/src/test/java/org/yb/pgsql/TestPgRegressPartitions.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| // Copyright (c) YugaByte, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except | ||
| // in compliance with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software distributed under the License | ||
| // is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express | ||
| // or implied. See the License for the specific language governing permissions and limitations | ||
| // under the License. | ||
| // | ||
| package org.yb.pgsql; | ||
|
|
||
| import org.junit.Test; | ||
| import org.junit.runner.RunWith; | ||
| import org.yb.util.YBTestRunnerNonTsanOnly; | ||
|
|
||
| /** | ||
| * Runs the pg_regress test suite on YB code. | ||
| */ | ||
| @RunWith(value=YBTestRunnerNonTsanOnly.class) | ||
| public class TestPgRegressPartitions extends BasePgSQLTest { | ||
| @Override | ||
| public int getTestMethodTimeoutSec() { | ||
| return 1800; | ||
| } | ||
|
|
||
| @Test | ||
| public void testPgRegressPartitions() throws Exception { | ||
| runPgRegressTest("yb_pg_partitions"); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.