Improve InboundTableMapper performance with bulk inserts #1626

Closed
davidnich opened this Issue Feb 1, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@davidnich
Contributor

davidnich commented Feb 1, 2017

InboundTableMapper bulk inserts should be made without converting the data to a list of hashes and then back again to a hash of lists - a hash of lists should be mapped in place and any constant or identify mappings should be made on the source list directly, which should bring performance up closed to that of BulkInsertOperation

@davidnich davidnich added this to the 0.8.13 milestone Feb 1, 2017

@davidnich davidnich self-assigned this Feb 1, 2017

@pavelkveton

This comment has been minimized.

Show comment
Hide comment
@pavelkveton

pavelkveton Feb 13, 2017

Contributor

how this affects the user code? Will we have to change the existing user code when the grouping will change?

Contributor

pavelkveton commented Feb 13, 2017

how this affects the user code? Will we have to change the existing user code when the grouping will change?

@davidnich

This comment has been minimized.

Show comment
Hide comment
@davidnich

davidnich Feb 13, 2017

Contributor

the change will be 100% transparent - it's just about changing the internal implementation to use more efficient data structures - which should allow the same SQL approach as with BulkInsertOperation with mappers in common use cases (such as used now in user code in the Rossini project for example)

Contributor

davidnich commented Feb 13, 2017

the change will be 100% transparent - it's just about changing the internal implementation to use more efficient data structures - which should allow the same SQL approach as with BulkInsertOperation with mappers in common use cases (such as used now in user code in the Rossini project for example)

@pavelkveton

This comment has been minimized.

Show comment
Hide comment
@pavelkveton

pavelkveton Feb 13, 2017

Contributor

ok then - nevertheless, as will have big impact on Rossini, I'd like to be also in reviewers to see what happened... thanks

Contributor

pavelkveton commented Feb 13, 2017

ok then - nevertheless, as will have big impact on Rossini, I'd like to be also in reviewers to see what happened... thanks

@pavelkveton

This comment has been minimized.

Show comment
Hide comment
@pavelkveton

pavelkveton Feb 13, 2017

Contributor

if we want it for Rossini, the milestone should change, right?

Contributor

pavelkveton commented Feb 13, 2017

if we want it for Rossini, the milestone should change, right?

@davidnich davidnich modified the milestones: 0.8.12.7, 0.8.13 Feb 13, 2017

@davidnich davidnich changed the title from Improve InboundTableMapper performance with bulk selects and inserts to Improve InboundTableMapper performance with bulk inserts Feb 13, 2017

davidnich added a commit that referenced this issue Feb 14, 2017

refs #1626 added a new performance test script and made updates to th…
…e *Mapper modules to improve the performance of InboundTableMapper::queueData() with hashes of lists, to keep data in the same format as it will be inserted with bulk DML

davidnich added a commit that referenced this issue Feb 14, 2017

@omusil24 omusil24 added the fixed label Feb 15, 2017

@omusil24 omusil24 closed this Feb 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment