Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(database)!: add db identifier shortening and migration #19732

Merged
merged 92 commits into from
Mar 25, 2024
Merged
Show file tree
Hide file tree
Changes from 90 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
1956c5e
fix: adapt review workflows to long identifiers
christiancp100 Feb 29, 2024
715e4ba
fix: hash long table names too:
christiancp100 Feb 29, 2024
e6288aa
fix: allow options in all methods
innerdvations Mar 4, 2024
4c6c052
enhancement: enable db name shortening
innerdvations Mar 4, 2024
96b5e16
fix: do not snake case table names
innerdvations Mar 4, 2024
3e0c983
chore: fix typo in createJoinColumn
innerdvations Mar 4, 2024
4c2d688
fix: prettier whitespace
innerdvations Mar 5, 2024
8aa1f8c
Merge branch 'v5/main' into feat/shorten-db-names
innerdvations Mar 5, 2024
c7b7f22
chore: add comments about re-shortening in index names
innerdvations Mar 6, 2024
be6c30c
chore: remove useless jsdoc
innerdvations Mar 6, 2024
aa997c6
fix: allow metadata to shorten table names
innerdvations Mar 8, 2024
527cb92
fix: disable feature by default
innerdvations Mar 8, 2024
f50a690
chore: test long and hashed identifiers
christiancp100 Mar 8, 2024
e56fff7
Merge branch 'feat/shorten-db-names' of github.com:strapi/strapi into…
christiancp100 Mar 8, 2024
49d42e5
fix: remove unnecessary code
christiancp100 Mar 8, 2024
e014264
Merge branch 'v5/main' into feat/shorten-db-names
innerdvations Mar 11, 2024
ef6145d
fix: shorten indexes that weren't being shortened
innerdvations Mar 11, 2024
4f5ffa0
feat(database): add migration
innerdvations Mar 11, 2024
2e7ac17
chore: merge
innerdvations Mar 11, 2024
09f47cf
enhancement: shorten suffixes
innerdvations Mar 11, 2024
5aa630f
chore: remove the gaslighting comment
innerdvations Mar 11, 2024
7e8c948
chore: finish incomplete thought
innerdvations Mar 11, 2024
d98aaff
fix: migration should succeed on new projects
innerdvations Mar 11, 2024
f301347
chore: remove dev file
innerdvations Mar 11, 2024
f80a87d
chore: test morphToMany relations
christiancp100 Mar 11, 2024
988f190
enhancement: remove duplicate metadata objects; fix replacement suffixes
innerdvations Mar 12, 2024
75afb30
chore: clean up comment
innerdvations Mar 12, 2024
6591456
Merge branch 'v5/main' into feat/shorten-db-names
innerdvations Mar 12, 2024
417acb9
fix: import only type
innerdvations Mar 12, 2024
bbc0f3a
chore: import only type
innerdvations Mar 12, 2024
ef4683c
chore: import only type
innerdvations Mar 12, 2024
e8b89b4
fix: do not snake case column name
innerdvations Mar 12, 2024
c63f76f
chore: comment explaining max length values
innerdvations Mar 12, 2024
19e474a
Merge branch 'v5/main' into feat/shorten-db-names
innerdvations Mar 12, 2024
c8ffd0c
chore: merge branch 'feat/shorten-db-names' into enhancement/short-db…
innerdvations Mar 12, 2024
99f8427
fix: only add replacements when shortening is enabled
innerdvations Mar 12, 2024
ec3c724
fix: match snake case behavior to v4 table and index name generation
innerdvations Mar 12, 2024
f5c1887
Merge branch 'v5/main' into feat/shorten-db-names
innerdvations Mar 12, 2024
0ee9fec
update from v5 but with test failures
innerdvations Mar 13, 2024
b86684a
fix: revert the changes to referencedtable shortening
innerdvations Mar 13, 2024
4ac36cc
fix: don't run shortener on raw meta tablename
innerdvations Mar 13, 2024
c79aee7
Merge branch 'feat/shorten-db-names' into enhancement/short-db-names-…
innerdvations Mar 13, 2024
87952ad
fix: expected metadata with referencedTable
christiancp100 Mar 13, 2024
8ceda3a
fix: make short forms of incompressible tokens work
innerdvations Mar 13, 2024
d139fab
fix: serialize names in map and remove metadataoptions from db
innerdvations Mar 13, 2024
63030af
fix: store unshortened name in map
innerdvations Mar 13, 2024
88561a0
fix: revert suffix shortening and unshortened name map
innerdvations Mar 13, 2024
71b664c
fix: do not snake case names in shortener
innerdvations Mar 14, 2024
f896d28
fix: snake case attribute on order columns
innerdvations Mar 14, 2024
d9d2aa4
fix: do not shorten relation names twice
innerdvations Mar 14, 2024
074f9bc
Merge branch 'feat/shorten-db-names' into enhancement/short-db-names-…
innerdvations Mar 14, 2024
c210001
fix: update tests
innerdvations Mar 14, 2024
67cee22
fix: add missing component_type column
innerdvations Mar 14, 2024
e0f1b8f
fix: suppress index rename errors
innerdvations Mar 14, 2024
a03bb0e
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 14, 2024
cf961a5
fix: skip index recreate if names are the same
innerdvations Mar 14, 2024
2635d0f
chore: clean up debug logs
innerdvations Mar 14, 2024
3054ba5
chore: comments and renaming
innerdvations Mar 14, 2024
31b6d51
chore: don't set columnName unless needed
innerdvations Mar 15, 2024
3bb25a5
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 15, 2024
274ee91
chore: add debug logging
innerdvations Mar 15, 2024
1368fff
chore: remove log
innerdvations Mar 15, 2024
f268193
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 15, 2024
870ac5f
chore: prettier
innerdvations Mar 15, 2024
75df0e5
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 15, 2024
64f06a7
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 18, 2024
fce452e
enhancement: remove option to change max length
innerdvations Mar 19, 2024
7053dd9
fix: remove debugging code
innerdvations Mar 19, 2024
011072d
chore: remove import
innerdvations Mar 19, 2024
4db0345
chore: add comment for future improvement
innerdvations Mar 20, 2024
2a645b6
chore: improve comment
innerdvations Mar 20, 2024
30cc873
chore: improve typings
innerdvations Mar 20, 2024
1a78eec
enhancement(database)!: shorten prefixes and suffixes used in identif…
innerdvations Mar 20, 2024
207f690
chore: clarify comment
innerdvations Mar 20, 2024
439e6c1
chore: improve comment
innerdvations Mar 20, 2024
e70e352
chore: remove unused args
innerdvations Mar 20, 2024
3147f98
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 20, 2024
66a41ab
chore: remove unused var
innerdvations Mar 20, 2024
4123324
chore: types cleanup
innerdvations Mar 20, 2024
8c7aeb7
Merge branch 'enhancement/short-db-names-migration' of https://github…
innerdvations Mar 20, 2024
615e484
fix: column name renamed synchronously
innerdvations Mar 21, 2024
80cf2c7
Merge branch 'enhancement/short-db-names-migration' of https://github…
innerdvations Mar 21, 2024
0928335
Merge branch 'v5/main' into enhancement/short-db-names-migration
innerdvations Mar 22, 2024
f58b836
fix: use knex.alterTable instead of knex.table for mysql
innerdvations Mar 22, 2024
433b1ed
test: add tests for shortname
innerdvations Mar 22, 2024
427bf71
chore: correct debug log
innerdvations Mar 22, 2024
73e4e97
chore: instantiate an identifiers object
innerdvations Mar 25, 2024
e06d15c
chore: database holds its own identifiers object
innerdvations Mar 25, 2024
50bed03
test: remove log
innerdvations Mar 25, 2024
d2a1c86
chore: identifiers moved to metadata
innerdvations Mar 25, 2024
1b62a64
fix: use trx properly and skip indexes that can't be renamed
innerdvations Mar 25, 2024
5978630
chore: update comments
innerdvations Mar 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { Core } from '@strapi/types';
import { filter, forEach, pipe, map, stubTrue, cond, defaultsDeep } from 'lodash/fp';
import { getService } from '../../utils';
import { getVisibleContentTypesUID, hasStageAttribute } from '../../utils/review-workflows';
Expand All @@ -14,14 +15,6 @@ import {
import { persistTables, removePersistedTablesWithSuffix } from '../../utils/persisted-tables';
import webhookEvents from '../../constants/webhookEvents';

import type { Core } from '@strapi/types';

const MAX_DB_TABLE_NAME_LEN = 63; // Postgres limit
// The longest index name that Strapi can create is prefixed with '_strapi_stage_links_inv_fk', so the content type name should be no longer than this.
const MAX_JOIN_TABLE_NAME_SUFFIX =
1 /* _ */ + ENTITY_STAGE_ATTRIBUTE.length + '_links_inv_fk'.length;
const MAX_CONTENT_TYPE_NAME_LEN = MAX_DB_TABLE_NAME_LEN - MAX_JOIN_TABLE_NAME_SUFFIX;

const DEFAULT_OPTIONS = {
numberOfWorkflows: MAX_WORKFLOWS,
stagesPerWorkflow: MAX_STAGES_PER_WORKFLOW,
Expand Down Expand Up @@ -71,22 +64,7 @@ const setReviewWorkflowAttributes = (contentType: any) => {

function extendReviewWorkflowContentTypes({ strapi }: { strapi: Core.LoadedStrapi }) {
const extendContentType = (contentTypeUID: any) => {
const assertContentTypeCompatibility = (contentType: any) =>
contentType.collectionName.length <= MAX_CONTENT_TYPE_NAME_LEN;

const incompatibleContentTypeAlert = (contentType: any) => {
strapi.log.warn(
`Review Workflow cannot be activated for the content type with the name '${contentType.info.displayName}' because the name exceeds the maximum length of ${MAX_CONTENT_TYPE_NAME_LEN} characters.`
);
return contentType;
};

const extendContentTypeIfCompatible = cond([
[assertContentTypeCompatibility, setReviewWorkflowAttributes],
[stubTrue, incompatibleContentTypeAlert],
]);

strapi.get('content-types').extend(contentTypeUID, extendContentTypeIfCompatible);
strapi.get('content-types').extend(contentTypeUID, setReviewWorkflowAttributes);
};

pipe([
Expand All @@ -102,7 +80,10 @@ function persistStagesJoinTables({ strapi }: { strapi: Core.LoadedStrapi }) {
// Persist the stage join table
const { attributes, tableName } = strapi.db.metadata.get(contentTypeUID) as any;
const joinTableName = attributes[ENTITY_STAGE_ATTRIBUTE].joinTable.name;
return { name: joinTableName, dependsOn: [{ name: tableName }] };
return {
name: joinTableName,
dependsOn: [{ name: tableName }],
};
};

const joinTablesToPersist = pipe([
Expand Down
8 changes: 4 additions & 4 deletions packages/core/core/src/Strapi.ts
Original file line number Diff line number Diff line change
Expand Up @@ -493,10 +493,10 @@ class Strapi extends Container implements Core.Strapi {

async bootstrap() {
const models = [
...utils.transformContentTypesToModels([
...Object.values(this.contentTypes),
...Object.values(this.components),
]),
...utils.transformContentTypesToModels(
[...Object.values(this.contentTypes), ...Object.values(this.components)],
this.db.metadata.identifiers
),
...this.get('models').get(),
];

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,25 @@
import { cloneDeep, merge } from 'lodash';
import { Database } from '@strapi/database';
import { transformContentTypesToModels } from '..';
import { LoadedContentTypeModel } from '../transform-content-types-to-models';

const db = new Database({
settings: {
migrations: {
dir: 'fakedir',
},
},
connection: {
client: 'sqlite',
useNullAsDefault: true,
connection: {
filename: 'fake.db',
},
},
});

const identifiers = db.metadata.identifiers;

// We want to match exactly with the exception that document_id.default should be any function
expect.extend({
toMatchModels(received, expected) {
Expand Down Expand Up @@ -263,41 +281,48 @@ function patchContentTypes(
}

describe('transformContentTypesToModels', () => {
test('converts valid content types to models', () => {
const models = transformContentTypesToModels(contentTypes);
describe('full length identifiers', () => {
// mock the options so that the 'global' identifiers created for use by createMetadata uses 0 for maxLength
Object.defineProperty(identifiers, 'options', {
get: jest.fn(() => ({ maxLength: 0 })),
});

expect(models).toMatchModels(expectedModels);
});
test('converts valid content types to models', () => {
const models = transformContentTypesToModels(contentTypes, identifiers);

expect(models).toMatchModels(expectedModels);
});

test.each(['id', 'document_id', 'ID', 'documentId'])(
'throws on restricted attribute name: %s',
(restrictedName) => {
const changes = {
attributes: {
[restrictedName]: {
type: 'string',
test.each(['id', 'document_id', 'ID', 'documentId'])(
'throws on restricted attribute name: %s',
(restrictedName) => {
const changes = {
attributes: {
[restrictedName]: {
type: 'string',
},
},
},
};
const modifiedContentTypes = patchContentTypes('countries', changes);
};
const modifiedContentTypes = patchContentTypes('countries', changes);

expect(() => transformContentTypesToModels(modifiedContentTypes)).toThrow(
`The attribute "${restrictedName}" is reserved`
);
}
);
expect(() => transformContentTypesToModels(modifiedContentTypes, identifiers)).toThrow(
`The attribute "${restrictedName}" is reserved`
);
}
);

test.each(['collectionName', 'uid', 'modelName'])(
'throws on missing name: %s',
(restrictedName) => {
const changes = {
[restrictedName]: null,
};
const modifiedContentTypes = patchContentTypes('countries', changes);
test.each(['collectionName', 'uid', 'modelName'])(
'throws on missing name: %s',
(restrictedName) => {
const changes = {
[restrictedName]: null,
};
const modifiedContentTypes = patchContentTypes('countries', changes);

expect(() => transformContentTypesToModels(modifiedContentTypes)).toThrow(
`"${restrictedName}" is required`
);
}
);
expect(() => transformContentTypesToModels(modifiedContentTypes, identifiers)).toThrow(
`"${restrictedName}" is required`
);
}
);
});
});