Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add accuracy tests against a known-good dataset #102

Open
bmtcril opened this issue May 31, 2024 · 1 comment
Open

Add accuracy tests against a known-good dataset #102

bmtcril opened this issue May 31, 2024 · 1 comment

Comments

@bmtcril
Copy link
Contributor

bmtcril commented May 31, 2024

As the schema settles, we should add more testing around making sure the calculations performed in this repo are good / don't accidentally break with future updates. I think the following is a good start, but am open to better ideas:

  • Create a fairly substantial dataset using the xapi-db-load tool's CSV output, maybe 100k rows
  • Store those in a test folder here, in a folder like test-seeds
  • As part of CI, rename that folder to seeds and load those files as seeds to the appropriate tables
  • Write new dbt tests tagged with ci so we can explicitly check the outputs of views and MVs but turn it off by default
  • In addition to the normal dbt test run, we should also run a dbt test --select tag:ci (I'm not 100% sure this will work with default off) and ensure the additional tests all come out correctly
  • It would be nice to reload the seeds and re-run the tests to check that all of our deduplication is working as well

Where possible it would be great if the tests can be assumed to work outside of the known-good dataset, in which case we should exclude marking them with the ci tag.

@saraburns1
Copy link
Contributor

examples of queries to test:
A simple one would be something like the fact_enrollments model, which should just have the most recent enrollment status joined with the user data.
The far other end is fact_video_watched_segments which tries to find how many times each 5 second section of a video was watched

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

2 participants