Update license scripts to support YAML frontmatter changes - #5259
Open
sarthak-shubham wants to merge 3 commits into
Open
Update license scripts to support YAML frontmatter changes#5259sarthak-shubham wants to merge 3 commits into
sarthak-shubham wants to merge 3 commits into
Conversation
Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
Signed-off-by: Sarthak Shubham <sarthakshubham23@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3138
What
Updated
report_license_rules.pyandbuildrules.pyinetc/scripts/licenses/to work correctly with the YAML frontmatter format introduced in #3100, which
added new fields (
is_builtin,is_generic,is_continuous,length) to theLicenseandRulemodels.report_license_rules.py
extrasaction='ignore'to thecsv.DictWritercalls so the script nolonger crashes when the models gain fields not present in the CSV column
headers — also avoids needing to update this script every time a new model
field is added going forward.
encoding="utf-8"to theopen()call inwrite_data_to_csv(), sincethe default Windows encoding (cp1252) can't handle non-ASCII characters
(e.g. in author names) present in the license database.
click.echo()(start, periodic progress every5,000 rules, completion message) since exporting 35,000+ rules takes several
minutes with no prior feedback that the script is working.
enumerate()instead of loading all rules into alist upfront, to support the progress logging without changing the output.
buildrules.py
click.echo()message before the duplicate-check step(
all_rule_by_tokens()). This step tokenizes all 35,000+ existing rules tocheck the new rule for duplicates, which takes roughly 10 minutes on a full
run — expected given the dataset size, but silent without the log.
Testing
Ran both scripts manually against the current license/rule database —
report_license_rules.py completes end-to-end; buildrules.py correctly parses
an example rule file and generates a new .RULE file (deleted afterward to
avoid committing test artifacts).
I wasn't able to find existing automated tests covering these two scripts —
they appear to be maintainer-facing utility scripts outside the main src/
detection engine and its test suite. Is there a preferred location/pattern
for tests on scripts like these, or should I add something under tests/
following a different structure?
Note
Saw #3150 referenced in the issue as possibly covering part of this — happy
to adjust if there's overlap.
Also included
Added an AUTHORS.rst entry and a CHANGELOG.rst note per CONTRIBUTING.rst's
PR guidelines.