Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance with intermediate table #5

Open
LaggAt opened this issue Mar 13, 2018 · 1 comment
Open

Performance with intermediate table #5

LaggAt opened this issue Mar 13, 2018 · 1 comment
Assignees
@LaggAt
Labels
performance Performance improvment

Comments

@LaggAt
Copy link
Owner

LaggAt commented Mar 13, 2018

I'm currently working on a live system (with low resources) and do profiling the code on a large dataset. This issue is just a reminder to review the cProfile results and optimize performance as soon as the past data is done. Seems biggest issues are lot of SQL's when working on intermediate tables - this is a less optimized portion of code for now.

So - stay tuned for cProfile / SnakeViz analysis.
@LaggAt LaggAt added the performance Performance improvment label Mar 13, 2018
@LaggAt LaggAt self-assigned this Mar 13, 2018
@LaggAt
Copy link
Owner Author

LaggAt commented Mar 13, 2018

Ideas for performance improvements are:

  • thread for every Dimension we work on (still getting new data while working on old once)
  • producer/consumer for intermediate table updating - doing it later but more performant using bulk updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LaggAt LaggAt

Labels
performance Performance improvment
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@LaggAt