Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve InboundTableMapper performance with bulk inserts #1626

Closed
davidnich opened this issue Feb 1, 2017 · 4 comments
Closed

Improve InboundTableMapper performance with bulk inserts #1626

davidnich opened this issue Feb 1, 2017 · 4 comments

Comments

@davidnich
Copy link
Contributor

InboundTableMapper bulk inserts should be made without converting the data to a list of hashes and then back again to a hash of lists - a hash of lists should be mapped in place and any constant or identify mappings should be made on the source list directly, which should bring performance up closed to that of BulkInsertOperation

@davidnich davidnich added this to the 0.8.13 milestone Feb 1, 2017
@davidnich davidnich self-assigned this Feb 1, 2017
@pavelkveton
Copy link
Contributor

how this affects the user code? Will we have to change the existing user code when the grouping will change?

@davidnich
Copy link
Contributor Author

the change will be 100% transparent - it's just about changing the internal implementation to use more efficient data structures - which should allow the same SQL approach as with BulkInsertOperation with mappers in common use cases (such as used now in user code in the Rossini project for example)

@pavelkveton
Copy link
Contributor

ok then - nevertheless, as will have big impact on Rossini, I'd like to be also in reviewers to see what happened... thanks

@pavelkveton
Copy link
Contributor

if we want it for Rossini, the milestone should change, right?

@davidnich davidnich modified the milestones: 0.8.12.7, 0.8.13 Feb 13, 2017
@davidnich davidnich changed the title Improve InboundTableMapper performance with bulk selects and inserts Improve InboundTableMapper performance with bulk inserts Feb 13, 2017
davidnich added a commit that referenced this issue Feb 14, 2017
…e *Mapper modules to improve the performance of InboundTableMapper::queueData() with hashes of lists, to keep data in the same format as it will be inserted with bulk DML
davidnich added a commit that referenced this issue Feb 14, 2017
@omusil24 omusil24 added the fixed label Feb 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants