Build tests from dashing output#45
Conversation
The defaultdict output does not appear to be stable, so we sort the keys. Also, it doesn't account for existing exclusion tests.
michaelblyons
force-pushed
the
test/from-dashing
branch
from
04b2620 to
fd461dc
Compare
|
Generating a test case/assertion for each index item seems a bit much imo. I understand that the purpose is to have regression tests, but it still needs to be updated periodically whenever changes to the docs are made because we are not verifying extra index entries. A better workflow I have in mind is to instead store a snapshot of all currently known index elements in either a tab- or comma-separated value table file with a deterministic order (and with duplicates). We would then have a single test case that validates whether the dashing output (or the sqlite db) matches this file and fail if it does not. This is accompanied by a github action that runs on PRs opened by dependabot where this snapshot file is re-generated and committed on the same branch (if it differs). This would allow us to immediately review the diff for the automatically opened PR as a simple sanity check without requiring any updating of tests while also being a more thorough verification (catches additions and duplicates alike). What do you think? |
|
No objection in principle. I do think that this schema makes more sense for the format than one line per duplicate:
Do you want this before merging 4205 to get the diff, or send that through? Footnotes
|
|
I do not need to see the diff, so we can merge 4205 first and then work on the diff. It would have been be a good opportunity to test the entire setup since the docs do not change that often, but the PR also already exists, so I do not really mind. In order to test the action more easily, we may want to run the action that updates the snapshot on all merge requests and not only those opened by dependabot. |
2 participants
Closes #44