Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: At-risk dashboard issues #781

Open
bmtcril opened this issue May 8, 2024 · 1 comment
Open

Perf: At-risk dashboard issues #781

bmtcril opened this issue May 8, 2024 · 1 comment
Assignees
Labels
aspects v1 Issues blocking Aspects v1 release

Comments

@bmtcril
Copy link
Contributor

bmtcril commented May 8, 2024

Tested with the 300M row dataset on ClickHouse Cloud I've found the following issues:

OOM:

  • Watched Video Segments (at-risk)
  • Problem Interactions (at-risk)
  • Partial and Full Views Per Video (at-risk)
  • Problem Results (at-risk)

Slow queries (> 3 seconds):

  1. learner-groups - Problems attempted per section/subsection (at-risk)
    Superset time: 20.888254 (s).
    Query duration: 20.439 (s).
    Result rows: 44
    Memory Usage (MB): 4332.254327774048
    Row count (superset) 44
    Filters: [{'column': 'course_key'}]

  2. learner-groups - Video Views per Section/Subsection (at-risk)
    Superset time: 11.049571 (s).
    Query duration: 10.62 (s).
    Result rows: 25
    Memory Usage (MB): 3735.4167289733887
    Row count (superset) 25
    Filters: [{'column': 'course_key'}]

  3. learner-groups - Page views per section/subsection (at-risk)
    Superset time: 10.707674 (s).
    Query duration: 10.328 (s).
    Result rows: 37
    Memory Usage (MB): 3852.6872720718384
    Row count (superset) 37
    Filters: [{'column': 'course_key'}]

  4. learner-groups - Cumulative Interactions (at-risk)
    Superset time: 10.587901 (s).
    Query duration: 8.761 (s).
    Result rows: 5
    Memory Usage (MB): 2674.5562772750854
    Row count (superset) 5
    Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

  5. learner-groups - Subsection Summary (at-risk)
    Superset time: 10.024011 (s).
    Query duration: 9.625 (s).
    Result rows: 30
    Memory Usage (MB): 2833.420473098755
    Row count (superset) 30
    Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

  6. learner-groups - Section Summary (at-risk)
    Superset time: 9.862307 (s).
    Query duration: 9.21 (s).
    Result rows: 7
    Memory Usage (MB): 2681.8192348480225
    Row count (superset) 7
    Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

  7. learner-groups - Video Views by Section/Subsection (at-risk)
    Superset time: 9.843626 (s).
    Query duration: 9.401 (s).
    Result rows: 40
    Memory Usage (MB): 2461.7836952209473
    Row count (superset) 40
    Filters: [{'column': 'emission_time'}, {'column': 'course_key'}]

  8. learner-groups - Evolution of Engagement (at-risk)
    Superset time: 9.206902 (s).
    Query duration: 8.761 (s).
    Result rows: 5
    Memory Usage (MB): 2674.5562772750854
    Row count (superset) 5
    Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

  9. learner-groups - Last visit date
    Superset time: 6.424135 (s).
    Query duration: 5.688 (s).
    Result rows: 8114
    Memory Usage (MB): 1887.9568014144897
    Row count (superset) 8114
    Filters: [{'column': 'course_key'}]

  10. learner-groups - At-risk learners
    Superset time: 5.895613 (s).
    Query duration: 5.317 (s).
    Result rows: 1000
    Memory Usage (MB): 1881.2105741500854
    Row count (superset) 1000
    Filters: [{'column': 'course_key'}]

  11. learner-groups - At-risk Enrollment Dates
    Superset time: 5.881555 (s).
    Query duration: 5.076 (s).
    Result rows: 8119
    Memory Usage (MB): 1886.271502494812
    Row count (superset) 8119
    Filters: [{'column': 'course_key'}]

  12. learner-groups - At-risk Enrollees per Enrollment Track
    Superset time: 5.200881 (s).
    Query duration: 4.766 (s).
    Result rows: 3
    Memory Usage (MB): 1881.2149724960327
    Row count (superset) 3
    Filters: [{'column': 'course_key'}]

  13. learner-groups - Distribution of Course Grades
    Superset time: 5.158168 (s).
    Query duration: 4.801 (s).
    Result rows: 10
    Memory Usage (MB): 1883.7704000473022
    Row count (superset) 10
    Filters: [{'column': 'course_key'}]

@bmtcril bmtcril added the aspects v1 Issues blocking Aspects v1 release label May 8, 2024
@bmtcril
Copy link
Contributor Author

bmtcril commented May 22, 2024

There should be decent improvements in these after #822 and the work on problems in dbt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aspects v1 Issues blocking Aspects v1 release
Projects
Status: Doing
Development

No branches or pull requests

2 participants