Increased importer perf by 50% and allow adding nested relations by foreign key#9431
Merged
kirrg001 merged 2 commits intoTryGhost:masterfrom Feb 20, 2018
Merged
Increased importer perf by 50% and allow adding nested relations by foreign key#9431kirrg001 merged 2 commits intoTryGhost:masterfrom
kirrg001 merged 2 commits intoTryGhost:masterfrom
Conversation
kirrg001
commented
Jan 28, 2018
|
|
||
| tags: function tags() { | ||
| return this.belongsToMany('Tag').withPivot('sort_order').query('orderBy', 'sort_order', 'ASC'); | ||
| return this.belongsToMany('Tag', 'posts_tags', 'post_id', 'tag_id').withPivot('sort_order').query('orderBy', 'sort_order', 'ASC'); |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
kirrg001
commented
Jan 28, 2018
| "created_by": 1, | ||
| "updated_at": null, | ||
| "updated_by": null | ||
| "updated_by": 10 |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
Contributor
Author
|
@cobbspur Could you please reserve a little time to test my PR? Thanks 👍 No code review needed. |
1e4e22a to
c133b46
Compare
no issue
- change behaviour from updating user references after the actual import to update the user reference before the actual import
- updating user references after the import is way less case intense
- that was the initial decision for updating the references afterwards
- but that does not play well with adding nested relations by identifier
- the refactoring is required for multiple authors
- if we e.g. store invalid author id's, we won't be able to add a belongs-to-many relation for multiple authors
- bookshelf-relations is generic and always tries to find a matching target before attching a model
- invalid user references won't work anymore
- this change has a very good side affect
- 17mb takes on master ~1,5seconds
- on this branch it takes ~45seconds
- also the memory usage is way lower and stabler
- 40mb takes 1,6s (times out on master)
no issue - replace logic for preparing nested tags - if you have nested tags in your file, we won't update or update the target tag - we simply would like to add the relationship to the database - use same approach as base class - add `posts_tags` to target post model - update identifiers - insert relation by foreign key `tag_id` - bump bookshelf-relations to 0.1.10
c133b46 to
956c90a
Compare
Contributor
Author
|
Was tested by @kevinansfield, @AileenCGN and @cobbspur. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
no issue
Speed improvement
author_id, we have to detect that and insert the correctpost_authorsrelationposts_authorswith an invalidauthor_id, bookshelf-relations will complain, because it ensures that adding nested relations requires a valid and existent target (it does not allow adding broken references to the database)This change has a very good side affect
Insert nested tags by foreign key
tagsruns already through bookshelf-relationsnamepost.tags=[{name: 'x'}]doesn't even make use of the target tag idnamecan also trouble, becausenameis not even unique (!)posts_tagsto target post modeltag_ide.g.post.tags=[{tag_id: '...'}]@cobbspur I've assigned you. To A: share knowledge that we refactored a piece of the importer and B: it would be great if you could do a second browser test and compare the result 👍 The importer still works like the same, it's just faster. You don't have to review the code.