Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store.append() very slow #255

Closed
alexis-bvq opened this issue Mar 23, 2021 · 0 comments
Closed

store.append() very slow #255

alexis-bvq opened this issue Mar 23, 2021 · 0 comments
Assignees
Labels
💾 loading related to a table loading operation
Milestone

Comments

@alexis-bvq
Copy link

Description

Hello,

I was playing with the .append of a store and I saw that it was very slow compared to a load_pandas when there are a lot of tuples.

image

Here the tuples list's length is 100 000.

Context

The use case would be a quants in a front office retrieving tuples from its internal message service and who would try to load group of tuples directly in the store.

Other information (if relevant)

I put it as a feature request because you could use pandas in the back end of .append method or change the documentation to tell users to use load_pandas. It is not really a bug but a performance issue. I think it is better to make .append efficient (else what would be its use case?) but I let you decide what solutions is the best as you have the overview of the product.

@Skyscimitar Skyscimitar added the 💾 loading related to a table loading operation label Mar 23, 2021
@Skyscimitar Skyscimitar self-assigned this Apr 1, 2021
@Skyscimitar Skyscimitar added this to the Next release milestone Apr 6, 2021
@patachoux patachoux bot closed this as completed Apr 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💾 loading related to a table loading operation
Projects
None yet
Development

No branches or pull requests

2 participants