Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add support for partitioning #1295

Merged
merged 5 commits into from
Nov 14, 2024
Merged

feat: add support for partitioning #1295

merged 5 commits into from
Nov 14, 2024

Conversation

elucidsoft
Copy link
Contributor

@elucidsoft elucidsoft commented Nov 12, 2024 •
edited
Loading

Added support for partitioning for tables.

pgm.createTable('mytable', {
  id: 'serial',
  title: 'text',
  language: 'text'
},
{
  partition: {
    strategy: 'RANGE',
    columns: 'timestamp'
  }
}

Supports the follow partition types: 'RANGE', 'LIST', 'HASH'.
Supports collation: PARTITION BY LIST ("language" COLLATE en_US)
Supports opclass:

pgm.createTable('mytable', {
  id: 'uuid',
  title: 'text',
  language: 'text'
},
{
  partition: {
    strategy: 'RANGE',
    columns: { name: 'id', opclass: 'hash_extension.uuid_ops' }
  }
}

Supports multiple columns:

pgm.createTable('mytable', {
  id: 'serial',
  title: 'text',
  language: 'text'
},
{
  partition: {
    strategy: 'RANGE',
    columns: ['id', 'title']
  }
}

Supports multiple columns with collate:

pgm.createTable('mytable', {
  id: 'serial',
  title: 'text',
  language: 'text'
},
{
  partition: {
    strategy: 'RANGE',
    columns: [{ name: 'title', collate: 'en_US'}, { name: 'language', collate: 'fr-FR'}]
  }
}

I tried my best to interpret the Postgresql standard for how this is suppose to work. I wrote a bunch of unit tests and added them to the existing ones. All unit tests are passing.

Also addresses #667

@Shinigami92 Shinigami92 added c: feature Request for new feature p: 1-normal Nothing urgent labels Nov 12, 2024
@Shinigami92 Shinigami92 added this to the v7.x milestone Nov 12, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 12, 2024 •
edited
Loading

Coverage Report

Status Category Percentage Covered / Total
🟢 Lines 92.12% (🎯 90%)
⬆️ +0.09%		 3230 / 3506
🟢 Statements 92.12% (🎯 90%)
⬆️ +0.09%		 3230 / 3506
🟢 Functions 95.91% (🎯 90%)
⬆️ +0.03%		 258 / 269
🟢 Branches 90.56% (🎯 85%)
⬆️ +0.22%		 835 / 922
File Coverage
File Stmts % Branch % Funcs % Lines Uncovered Lines
Changed Files
src/operations/tables/createTable.ts 88.7% 76.47% 100% 88.7% 50-54, 86-87
src/operations/tables/shared.ts 82.31% 78.46% 80% 82.31% 155-156, 159-160, 213-217, 242, 246-247, 266-267, 292-293, 296-303, 306-307, 444-449, 451-454, 456-466, 468-469
src/utils/formatPartitionColumns.ts 100% 100% 100% 100%
src/utils/index.ts 100% 100% 100% 100%
Unchanged Files
src/db.ts 84.29% 89.28% 87.5% 84.29% 75-77, 107, 110-119, 121-122, 162-164
src/index.ts 100% 100% 100% 100%
src/migration.ts 73.33% 86.79% 66.66% 73.33% 137-140, 161-163, 165-177, 216-222, 227-231, 234, 236-240, 242-249, 252, 255-260, 262-263, 320-321, 347-349, 357-358, 375-378, 407-408
src/migrationBuilder.ts 95.43% 92.85% 88.88% 95.43% 354-358, 497-499, 501-502, 526-527
src/runner.ts 75.43% 59.25% 90% 75.43% 45, 65-66, 75-76, 125-126, 168-171, 180-184, 197, 201-202, 204-206, 208-214, 233, 235-241, 244, 257, 269, 271-277, 279-282, 285-288, 298-299, 308-310, 319, 321, 323-330
src/sqlMigration.ts 90% 100% 80% 90% 45-46, 48-49
src/types.ts 100% 100% 100% 100%
src/operations/sql.ts 100% 100% 100% 100%
src/operations/casts/createCast.ts 100% 100% 100% 100%
src/operations/casts/dropCast.ts 100% 100% 100% 100%
src/operations/casts/index.ts 100% 100% 100% 100%
src/operations/domains/alterDomain.ts 100% 100% 100% 100%
src/operations/domains/createDomain.ts 100% 100% 100% 100%
src/operations/domains/dropDomain.ts 100% 100% 100% 100%
src/operations/domains/index.ts 100% 100% 100% 100%
src/operations/domains/renameDomain.ts 100% 100% 100% 100%
src/operations/domains/shared.ts 100% 100% 100% 100%
src/operations/extensions/createExtension.ts 100% 100% 100% 100%
src/operations/extensions/dropExtension.ts 100% 100% 100% 100%
src/operations/extensions/index.ts 100% 100% 100% 100%
src/operations/extensions/shared.ts 100% 100% 100% 100%
src/operations/functions/createFunction.ts 95.58% 94.44% 100% 95.58% 71-73
src/operations/functions/dropFunction.ts 100% 100% 100% 100%
src/operations/functions/index.ts 100% 100% 100% 100%
src/operations/functions/renameFunction.ts 100% 100% 100% 100%
src/operations/functions/shared.ts 100% 100% 100% 100%
src/operations/grants/grantOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/grantOnTables.ts 100% 100% 100% 100%
src/operations/grants/grantRoles.ts 100% 100% 100% 100%
src/operations/grants/index.ts 100% 100% 100% 100%
src/operations/grants/revokeOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/revokeOnTables.ts 100% 100% 100% 100%
src/operations/grants/revokeRoles.ts 100% 100% 100% 100%
src/operations/grants/shared.ts 95.65% 85.71% 100% 95.65% 71
src/operations/indexes/createIndex.ts 96.29% 95.23% 100% 96.29% 74-75
src/operations/indexes/dropIndex.ts 100% 100% 100% 100%
src/operations/indexes/index.ts 100% 100% 100% 100%
src/operations/indexes/shared.ts 88% 86.95% 100% 88% 22, 32-35, 47
src/operations/materializedViews/alterMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/createMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/dropMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/index.ts 100% 100% 100% 100%
src/operations/materializedViews/refreshMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedViewColumn.ts 100% 100% 100% 100%
src/operations/materializedViews/shared.ts 100% 87.5% 100% 100%
src/operations/operators/addToOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/createOperator.ts 100% 100% 100% 100%
src/operations/operators/createOperatorClass.ts 100% 83.33% 100% 100%
src/operations/operators/createOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/dropOperator.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/index.ts 100% 100% 100% 100%
src/operations/operators/removeFromOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/shared.ts 85.71% 75% 100% 85.71% 24-25, 38
src/operations/policies/alterPolicy.ts 100% 100% 100% 100%
src/operations/policies/createPolicy.ts 100% 100% 100% 100%
src/operations/policies/dropPolicy.ts 100% 100% 100% 100%
src/operations/policies/index.ts 100% 100% 100% 100%
src/operations/policies/renamePolicy.ts 100% 100% 100% 100%
src/operations/policies/shared.ts 100% 100% 100% 100%
src/operations/roles/alterRole.ts 100% 100% 100% 100%
src/operations/roles/createRole.ts 100% 75% 100% 100%
src/operations/roles/dropRole.ts 100% 100% 100% 100%
src/operations/roles/index.ts 100% 100% 100% 100%
src/operations/roles/renameRole.ts 100% 100% 100% 100%
src/operations/roles/shared.ts 98.07% 76.92% 100% 98.07% 78
src/operations/schemas/createSchema.ts 100% 100% 100% 100%
src/operations/schemas/dropSchema.ts 100% 100% 100% 100%
src/operations/schemas/index.ts 100% 100% 100% 100%
src/operations/schemas/renameSchema.ts 100% 100% 100% 100%
src/operations/sequences/alterSequence.ts 94.11% 75% 100% 94.11% 23
src/operations/sequences/createSequence.ts 100% 100% 100% 100%
src/operations/sequences/dropSequence.ts 100% 100% 100% 100%
src/operations/sequences/index.ts 100% 100% 100% 100%
src/operations/sequences/renameSequence.ts 100% 100% 100% 100%
src/operations/sequences/shared.ts 78.57% 68.75% 100% 78.57% 41, 43-44, 49-50, 63-64, 69-70
src/operations/tables/addColumns.ts 100% 80% 100% 100%
src/operations/tables/addConstraint.ts 100% 100% 100% 100%
src/operations/tables/alterColumn.ts 88.57% 76.47% 100% 88.57% 75, 82-85, 87-89
src/operations/tables/alterTable.ts 100% 100% 100% 100%
src/operations/tables/dropColumns.ts 100% 100% 100% 100%
src/operations/tables/dropConstraint.ts 100% 100% 100% 100%
src/operations/tables/dropTable.ts 100% 100% 100% 100%
src/operations/tables/index.ts 100% 100% 100% 100%
src/operations/tables/renameColumn.ts 100% 100% 100% 100%
src/operations/tables/renameConstraint.ts 100% 100% 100% 100%
src/operations/tables/renameTable.ts 100% 100% 100% 100%
src/operations/triggers/createTrigger.ts 86.41% 68.18% 100% 86.41% 53-54, 66-67, 70-71, 74-77, 113
src/operations/triggers/dropTrigger.ts 100% 100% 100% 100%
src/operations/triggers/index.ts 100% 100% 100% 100%
src/operations/triggers/renameTrigger.ts 100% 100% 100% 100%
src/operations/triggers/shared.ts 100% 100% 100% 100%
src/operations/types/addTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/addTypeValue.ts 100% 100% 100% 100%
src/operations/types/createType.ts 100% 100% 100% 100%
src/operations/types/dropType.ts 100% 100% 100% 100%
src/operations/types/dropTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/index.ts 100% 100% 100% 100%
src/operations/types/renameType.ts 100% 100% 100% 100%
src/operations/types/renameTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/renameTypeValue.ts 100% 100% 100% 100%
src/operations/types/setTypeAttribute.ts 100% 100% 100% 100%
src/operations/views/alterView.ts 100% 100% 100% 100%
src/operations/views/alterViewColumn.ts 100% 100% 100% 100%
src/operations/views/createView.ts 100% 100% 100% 100%
src/operations/views/dropView.ts 100% 100% 100% 100%
src/operations/views/index.ts 100% 100% 100% 100%
src/operations/views/renameView.ts 100% 100% 100% 100%
src/operations/views/shared.ts 100% 66.66% 100% 100%
src/utils/PgLiteral.ts 88.88% 100% 75% 88.88% 37-38
src/utils/StringIdGenerator.ts 100% 100% 100% 100%
src/utils/createSchemalize.ts 100% 100% 100% 100%
src/utils/createTransformer.ts 100% 100% 100% 100%
src/utils/decamelize.ts 100% 100% 100% 100%
src/utils/escapeValue.ts 100% 100% 100% 100%
src/utils/formatLines.ts 100% 100% 100% 100%
src/utils/formatParams.ts 100% 100% 100% 100%
src/utils/getMigrationTableSchema.ts 100% 100% 100% 100%
src/utils/getSchemas.ts 100% 100% 100% 100%
src/utils/identity.ts 100% 100% 100% 100%
src/utils/intersection.ts 100% 100% 100% 100%
src/utils/makeComment.ts 100% 100% 100% 100%
src/utils/quote.ts 100% 100% 100% 100%
src/utils/toArray.ts 100% 100% 100% 100%
src/utils/types.ts 100% 100% 100% 100%
Generated in workflow #1317 for commit 698952c by the Vitest Coverage Report Action

@Shinigami92
Copy link
Collaborator

The implementation looks quite clean to me 😃
FFR: https://www.postgresql.org/docs/current/ddl-partitioning.html

Please add an integration test in https://github.com/salsita/node-pg-migrate/tree/main/test/migrations
Either find one of the existing tests that can be updated, or add a new 092_partition.js (or something like that)

@elucidsoft
Copy link
Contributor Author

Made the changes, please check.

@elucidsoft
Copy link
Contributor Author

Ahh, I see problem with integration tests. Ok I will fix that.

@Shinigami92
Copy link
Collaborator

Ahh, I see problem with integration tests. Ok I will fix that.

Beside the failing integration tests, this PR looks awesome and I would wish that every PR is like this. Straight forward and clean.

I will approve and merge this when CI is green and will target a release within today or before weekend.

@Shinigami92
Copy link
Collaborator

you can run pnpm run preflight locally, and than commit all missing changes

@Shinigami92 Shinigami92 merged commit f3013e0 into salsita:main Nov 14, 2024
34 checks passed
@Shinigami92 Shinigami92 mentioned this pull request Nov 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Shinigami92 Shinigami92 Shinigami92 approved these changes

Assignees

@elucidsoft elucidsoft

Labels
c: feature Request for new feature p: 1-normal Nothing urgent
Projects
None yet
Milestone
v7.x
Development

Successfully merging this pull request may close these issues.
2 participants
@elucidsoft @Shinigami92