Add license dump option #3100

AyanSinhaMahapatra · 2022-09-12T08:57:18Z

Modifies the License and Rule files to have their data as YAML frontmatter.

.RULE files and .LICENSE files now also contain their data.
git history is preserved
loading/dumping methods are modified to support YAML frontmatter
License/Rule now has different methods to point to their file.

Adds a new command line option: --get-license-data to:

Dump license data in JSON, YAML and HTML formats.
Also dumps the .LICENSE file with text and data as YAML frontmatter.
Generates an index and a static website to view the data.

This is reusing code originally located at:
https://github.com/nexB/scancode-licensedb

Reference: #2738
Reference: #3049

This also contains PR: #3063 btw.

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁

Signed-off-by: Thomas Druez <tdruez@nexb.com>

…o main

Signed-off-by: Thomas Druez <tdruez@nexb.com>

* Expose spdx_license_key, other_spdx_license_keys, and is_exception Signed-off-by: Jono Yang <jyang@nexb.com>

Expose SPDX license info in index.json #7

Signed-off-by: tdruez <tdruez@nexb.com>

Signed-off-by: Jono Yang <jyang@nexb.com>

Expose is_deprecated in index.json

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

This commit ends the process of merging .RULE and .yml files into a single .RULE file which has YAML frontmatter storing the rule metadata present in the .yml file previously. This renaming and merging has been done to preserve line-history for both the files. Reference: #3049 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

* Import and use functions/classes from frontmatter * Copy and modify class to use saneyaml * Copy and modify load/parse functions * Add .ABOUT and LICENSE files. Signed-off-by: AyanSinhaMahapatra <ayansmahapatra@gmail.com>

Reference: nexB/scancode-licensedb#25 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Regen test expectations for this and also fix another test. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

pombredanne

LGTM... Please do one last check wrt. spaces:

from licensedcode.models import *
gpl=db["gpl-2.0"]
bottom = gpl.text.strip("\n") + "\n"
top = saneyaml.dump(gpl.to_dict())
fm=f"---\n{top}\n---\n{bottom}"

IMHO redumping all licenses as frontmatter this way should be to help that we are not damaging space-based formatting. In particualr watch out for the GPL -2.0.

src/licensedcode/models.py

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra · 2022-11-01T17:32:38Z

@pombredanne post your approval I've pushed the following changes:

Fix whitespace in License texts
Merge the latest changes in develop including external licenses work and make tests pass.
Update changelog

Should I go ahead and merge?

AyanSinhaMahapatra · 2022-11-01T17:40:17Z

Some scripts need update but I opened an issue: #3138 to track that separately to not block this anymore.

AyanSinhaMahapatra · 2023-01-10T20:41:51Z

Note that the command line option was later renamed from --get-license-data to --dump-license-data later.

ScanCode 32.0.0 started to prepend its `*.LICENSE` files with YAML-encoded metadata, see [1]. This is a hot fix to remove this header, if present, from the license files. A better solution will be implemented later as part of a larger refactoring of license providers. Different ScanCode versions also differ in whether license files come with a final newline or not. Align on not having a final newline to make tests pass either way. [1]: nexB/scancode-toolkit#3100 Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.org>

tdruez and others added 30 commits December 4, 2020 16:30

Initial commit

07b0269

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Add the generated static website

27c255c

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Create CNAME

b39763d

Upgrade ScanCode-toolkit to latest 21.2.9 version

3bf65b9

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Do not display "LicenseRef-" SPDX key as links

e97dc5d

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Create CNAME

76c145d

Inject the search value in the browser href

07316b6

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Merge branch 'main' of https://github.com/nexB/scancode-licensedb int…

3e56341

…o main

Do not remove the CNAME file on Make clean

4e8389b

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Upgrade ScanCode-toolkit to latest version

9164413

Add "publish" command in Makefile

cf386c8

Signed-off-by: Thomas Druez <tdruez@nexb.com>

Expose SPDX license info in index.json #7

efdb626

* Expose spdx_license_key, other_spdx_license_keys, and is_exception Signed-off-by: Jono Yang <jyang@nexb.com>

Merge pull request #8 from nexB/7-expose-spdx-information

c6a64d1

Expose SPDX license info in index.json #7

Fix header layout for smaller display sizes

f98b976

Signed-off-by: tdruez <tdruez@nexb.com>

Merge branch

8c69070

Upgrade ScanCode-toolkit to latest version

d2f3920

Upgrade ScanCode-toolkit to latest version

16dd721

Enhance the details view UI and display URLs as link #5

13e9166

Signed-off-by: tdruez <tdruez@nexb.com>

Do not display deprecated licenses in main listing by default #2

2a4e4cb

Signed-off-by: tdruez <tdruez@nexb.com>

Expose is_deprecated in index.json #9

40c5a8d

Signed-off-by: Jono Yang <jyang@nexb.com>

Merge pull request #10 from nexB/9-expose-is_deprecated

d7fa06b

Expose is_deprecated in index.json

Upgrade ScanCode-toolkit to latest version

3069232

Upgrade ScanCode-toolkit to latest version

5b0f829

Upgrade ScanCode-toolkit to latest version

1d81315

Upgrade ScanCode-toolkit to latest version

838f364

Rename .yml files to .RULE.new

8d234ca

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Rename the .RULE files to .RULE.new

0156b3a

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Merge branch '3049-too-many-files-rename' into 3049-too-many-files

e755274

Use modified functions and classes from frontmatter

bd3545c

* Import and use functions/classes from frontmatter * Copy and modify class to use saneyaml * Copy and modify load/parse functions * Add .ABOUT and LICENSE files. Signed-off-by: AyanSinhaMahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra requested a review from pombredanne September 12, 2022 10:07

AyanSinhaMahapatra added 2 commits September 28, 2022 12:51

dump sorted license data by key

12355c5

Reference: nexB/scancode-licensedb#25 Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Add license dump tests

5366e68

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra mentioned this pull request Sep 28, 2022

Use scancode-license-data option and build automatically nexB/scancode-licensedb#26

Merged

2 tasks

pombredanne mentioned this pull request Oct 10, 2022

Add licenses from scancode license list fsfe/reuse-tool#526

Open

AyanSinhaMahapatra force-pushed the add-license-dump-option branch from 719fabb to 84f8a01 Compare October 13, 2022 14:41

Merge branch 'develop' into add-license-dump-option

a7cc2d5

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the add-license-dump-option branch from 84f8a01 to a7cc2d5 Compare October 13, 2022 16:01

Fix binary gibberish bug on extra newlines

8a8fba0

Regen test expectations for this and also fix another test. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the add-license-dump-option branch from 6843a32 to 8a8fba0 Compare October 13, 2022 20:32

pombredanne mentioned this pull request Oct 28, 2022

Introduce Rules with YAML frontmatter #3049 #3063

Closed

pombredanne approved these changes Oct 28, 2022

View reviewed changes

src/licensedcode/models.py Outdated Show resolved Hide resolved

Reinstate whitespace in license text

8b2bbfe

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the add-license-dump-option branch 2 times, most recently from 8b1d58e to 093b8b4 Compare November 1, 2022 11:09

Merge branch 'develop' into add-license-dump-option

bc6d6a5

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra force-pushed the add-license-dump-option branch from 093b8b4 to bc6d6a5 Compare November 1, 2022 11:26

Update changelog

c857fa4

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

AyanSinhaMahapatra mentioned this pull request Nov 1, 2022

Update license scripts for yaml frontmatter #3138

Open

AyanSinhaMahapatra mentioned this pull request Nov 1, 2022

Creating a executable of ScanCode-toolkit #2427

Open

7 tasks

AyanSinhaMahapatra merged commit 9f91bf5 into develop Nov 2, 2022

pombredanne deleted the add-license-dump-option branch November 14, 2022 16:53

sschuberth mentioned this pull request Nov 16, 2023

fix: Remove any YAML front matter from ScanCode license files oss-review-toolkit/ort#7883

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add license dump option #3100

Add license dump option #3100

AyanSinhaMahapatra commented Sep 12, 2022 •

edited

Loading

pombredanne left a comment

AyanSinhaMahapatra commented Nov 1, 2022

AyanSinhaMahapatra commented Nov 1, 2022 •

edited

Loading

AyanSinhaMahapatra commented Jan 10, 2023

Add license dump option #3100

Add license dump option #3100

Conversation

AyanSinhaMahapatra commented Sep 12, 2022 • edited Loading

Tasks

pombredanne left a comment

Choose a reason for hiding this comment

AyanSinhaMahapatra commented Nov 1, 2022

AyanSinhaMahapatra commented Nov 1, 2022 • edited Loading

AyanSinhaMahapatra commented Jan 10, 2023

AyanSinhaMahapatra commented Sep 12, 2022 •

edited

Loading

AyanSinhaMahapatra commented Nov 1, 2022 •

edited

Loading