Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transformer workaround memory leak when processing large models #3253

Merged
merged 14 commits into from
Feb 22, 2022
Merged

transformer workaround memory leak when processing large models #3253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
ecb5017
use async query on primary connection for detect deletions to prevent…
MichaelBelousov Feb 16, 2022
21879d8
save changes in the target every 500mgb, this will be configurable ev…
MichaelBelousov Feb 16, 2022
47d25bd
add sessionChangesetSizeSaveThreshold configuration of transformer, a…
MichaelBelousov Feb 18, 2022
cd338f1
created memory issue reproducing test
MichaelBelousov Feb 19, 2022
284b48e
tighten mv reproduction
MichaelBelousov Feb 19, 2022
c620c63
deterministic end and don't need primary connection
MichaelBelousov Feb 19, 2022
0cf9afc
separate getInstanceProps loop
MichaelBelousov Feb 20, 2022
5252423
more leak test changes
MichaelBelousov Feb 21, 2022
fe2e0d3
restore working process size measure
MichaelBelousov Feb 21, 2022
e835f1b
remove test since there is now an mvr here: https://github.com/nodejs…
MichaelBelousov Feb 22, 2022
ae29ed6
revert this branch
MichaelBelousov Feb 22, 2022
053b08d
add manual setImmediates to known large loops
MichaelBelousov Feb 22, 2022
2debe9d
rush change
MichaelBelousov Feb 22, 2022
de7666e
revert loop style change
MichaelBelousov Feb 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions common/changes/@itwin/core-transformer/transformer-handle-large-models_2022-02-22-18-07.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"changes": [
{
"packageName": "@itwin/core-transformer",
"comment": "mitigate memory leak in large model processing",
"type": "none"
}
],
"packageName": "@itwin/core-transformer"
}
2 changes: 2 additions & 0 deletions core/transformer/src/IModelExporter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,7 @@ export class IModelExporter {
}
while (DbResult.BE_SQLITE_ROW === statement.step()) {
await this.exportElement(statement.getValue(0).getId());
await new Promise(setImmediate); // workaround for: https://github.com/nodejs/node-addon-api/issues/1140
}
});
}
Expand Down Expand Up @@ -638,6 +639,7 @@ export class IModelExporter {
const relInstanceId: Id64String = statement.getValue(0).getId();
const relProps: RelationshipProps = this.sourceDb.relationships.getInstanceProps(baseRelClassFullName, relInstanceId);
await this.exportRelationship(relProps.classFullName, relInstanceId); // must call exportRelationship using the actual classFullName, not baseRelClassFullName
await new Promise(setImmediate); // workaround for: https://github.com/nodejs/node-addon-api/issues/1140
}
});
}
Expand Down
3 changes: 2 additions & 1 deletion core/transformer/src/IModelTransformer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -709,7 +709,7 @@ export class IModelTransformer extends IModelExportHandler {
}
const aspectDeleteIds: Id64String[] = [];
const sql = `SELECT ECInstanceId,Identifier,JsonProperties FROM ${ExternalSourceAspect.classFullName} aspect WHERE aspect.Scope.Id=:scopeId AND aspect.Kind=:kind`;
this.targetDb.withPreparedStatement(sql, (statement: ECSqlStatement): void => {
await this.targetDb.withPreparedStatement(sql, async (statement: ECSqlStatement) => {
statement.bindId("scopeId", this.targetScopeElementId);
statement.bindString("kind", ExternalSourceAspect.Kind.Relationship);
while (DbResult.BE_SQLITE_ROW === statement.step()) {
Expand All @@ -722,6 +722,7 @@ export class IModelTransformer extends IModelExportHandler {
}
aspectDeleteIds.push(statement.getValue(0).getId());
}
await new Promise(setImmediate); // workaround for: https://github.com/nodejs/node-addon-api/issues/1140
}
});
this.targetDb.elements.deleteAspect(aspectDeleteIds);
Expand Down