Add identifiers and splits #3

goneall · 2017-10-21T22:21:30Z

Made some progress going through the fsf web pages and adding SPDX identifiers.

The attached file has a list of FSF ID's that do not have obvious matches in SPDX followed by a list of code snippets that need to be investigated and competed.
list.txt

Signed-off-by: Gary O'Neall <gary@sourceauditor.com>

goneall · 2017-10-22T00:24:24Z

Ammended PR with identifiers up through the letter N - I'll add some more later tonight or tomorrow

wking · 2017-10-22T03:17:48Z

pull.py

    'Condor': {'spdx': 'Condor-1.1'},
    'ECL2.0': {'spdx': 'ECL-2.0'},
+    'eCos11': {'spdx': 'RHeCos-1.1'},
+    'eCos2.0': {'spdx': 'eCos-2.0'},


This SPDX ID is deprecated in favor of GPL-2.0+ WITH eCos-exception-2.0. I'd rather use the modern license expression.

Do we want to allow license expressions for the 'spdx' tag or restrict it to the license ID's only?

Do we want to allow license expressions for the 'spdx' tag...

Yes, I think we do. If someone finds GPL-2.0+ WITH eCos-exception-2.0 content, they should be able to determine its FSF tags without going through a deprecated SPDX ID.

OK - I'll add it to the amended PR - coming up soon

I changed eCos2.0, but left ecos11 since RHeCos-1.1 is not marked as deprecated.

wking · 2017-10-22T03:54:51Z

pull.py

+    'ModifiedBSD': {'spdx': 'BSD-3-Clause'},
+    'MPL': {'spdx': 'MPL-1.1'},
+    'MPL-2.0': {'spdx':'MPL-2.0'},
+    'ms-pl': {'spdx': 'MPL-1.1'},


This is FSF's Microsoft Public License (Ms-PL), which is SPDX's MS-PL. SPDX's MPL-1.1 is the Mozilla Public License 1.1.

Oops - that must have been a typo. Good catch.

wking · 2017-10-22T04:02:53Z

pull.py

+    'NCSA': {'spdx':'NCSA'},
+    'newOpenLDAP': {'spdx': 'OLDAP-2.7'},
+    'Nokia': {'spdx': 'Nokia'},
+    'NoLicense': {'spdx': 'UNSPECIFIED'},


I read the FSF's NoLicense entry as "all rights reserved". NOASSERTION is different, and I don't see UNSPECIFIED in the spec. ~~I expect NoLicense has no SPDX equivalent at the moment.~~ I think the equivalent SPDX expression is NONE.

Agree - I'll amend the PR

wking · 2017-10-22T05:25:41Z

Also, can you drop the Signed-off-by from your commit messages? (or I can squash when I merge). For those to mean anything, you need repository-local documentation for them (e.g. see https://github.com/wking/signed-off-by), and I expect this project to be small enough that that doesn't matter.

…ment

…e license IDs as some of these required judgement as to the associated SPDX license ID

… into add-identifiers-go

goneall · 2017-10-22T18:57:24Z

@wking Hold off on merging the PR - I made a mistaking and I'm currently fighting git to get the fix added

goneall · 2017-10-22T19:03:56Z

OK - Everything should be added and all review comments addressed.

Please review - especially those SPDX identifiers with comments.

I also added a file to track the FSF licenses which do not have associated SPDX license ID's or expressions.

goneall · 2017-10-22T19:04:48Z

BTW - you may want to squash the commits - I added quite a few commits as part of the PR to make it easier to find specific changes during the review.

wking · 2017-10-22T19:34:26Z

unassociated-license-ids.md

@@ -0,0 +1,47 @@
+# The following FSF license tags did not have any obvious match to an SPDX license


I don't think we need this in version control, because you can generate it from licenses.json with jq. Can we remove it here? And if you like open an issue with a list of unchecked boxes?

jq for this is:

$ jq -r 'to_entries[] | select(.value.identifiers.spdx | not) | .key' licenses.json

I'll replace it with an issue since some of the items have comments that would not be retained by using jq.

wking · 2017-10-22T21:49:35Z

pull.py

+    'Zend': {'spdx': 'Zend-2.0'},
+    'Zimbra': {'spdx': 'Zimbra-1.3'},
+    'ZLib': {'spdx': 'Zlib'},
+    'Zope': {'spdx': 'ZPL-1.1'}, # Note the FSF refers to version 1.0 and SPDX uses version 1.1 - it should be verified that 1.1 should be included


I'd rather leave these inexact matches off. If someone is looking up GPL-compat in this API, for example, false negatives are much less problematic than false positives.

If we want to take a more conservative approach, I would remove the SPDX ID's for Zope and Ruby. I consider SISSL to be an internal inconsistency on the FSF website and I believe they intend it to match the referenced SPDX license.

In your review comment, you included Zlib, Zimbra and Zend. I'm assuming your comment only referred to Zope - let me know that is incorrect or if there are other license matches that concern you.

In your review comment, you included Zlib, Zimbra and Zend. I'm assuming your comment only referred to Zope - let me know that is incorrect or if there are other license matches that concern you.

I'd rather have this bulk PR only cover exact matches. Can you drop anything you consider questionable and file those in per-license(-group) PRs?

Done - I already removed the matches I consider questionable and added them to the list of FSF license ids without an associated SPDX license ID.

wking · 2017-10-22T23:05:00Z

pull.py

@@ -47,6 +47,20 @@
        'CC-BY-ND-3.0',
        'CC-BY-ND-4.0',
    ],
+    'CC-BY': [ # any version


I don't see any CC-BY slug on their page. This link should not take you to any specific entry: https://www.gnu.org/licenses/license-list.html#CC-BY

I've added checks in master's pull.py to watch out for this sort of thing.

I just pushed a correction for this and CC-BY-SA to the pull request branch

…se matches to the spdx license list and remove the unassociated-license-ids.md file per pull request review

goneall · 2017-10-22T23:23:16Z

Amended PR to address last 2 review comments.

wking · 2017-10-23T16:38:12Z

pull.py

+        'OSL-2.1',
+        'OSL-3.0',
+    ],
+    'RPL': [ # any version - Note that FSF website does not state any version, but references version 1.3 in the URL.  It is assumed that it also covers version 1.1 and 1.5, but this should be verified with FSF.


The FSF at least occasionally says “any version” when that's their intention. Until we get clarity on this, can we drop it from this PR (and the associated RPL-* IDENTIFIERS entries) and come back to them in follow-up work?

wking · 2017-10-23T16:46:33Z

pull.py

+    ],
+    'Unicode': [ # any version
+        'Unicode-DFS-2015',
+        'Unicode-DFS-2016',


Here is another case where the FSF doesn't say “any version”. The version they link is from 2012 (based on the copyright) and differs from Unicode-DFS-2015 at least in:

the above copyright notice(s) and this permission notice appear with all copies of the Data Files or Software,

vs. Unicode-DFS-2015's

this copyright and permission notice appear with all copies of the Data Files or Software,

Until we get clarity on this, can we drop it from this PR (and the associated Unicode-DFS-* IDENTIFIERS entries) and come back to them in follow-up work?

wking · 2017-10-23T16:55:55Z

pull.py

+    ],
+    'W3C': [ # any version
+        'W3C',
+        'W3C-20150513',


Here is another case where the FSF doesn't say “any version”. The version they link is from 2002-12-31 (based on the URI) which matches our W3C. Our W3C-19980720, on the other hand, has “This W3C Work”, and our W3C-20150513 has “Permission to copy, modify, and distribute this work”. Until we get clarity on this, can we drop the SPLITS entry from this PR (and all but the W3C IDENTIFIERS entries) and come back to them in follow-up work?

wking · 2017-10-23T16:58:09Z

pull.py

+        'W3C-20150513',
+        'W3C-19980720',
+    ],
+    'Zope2.0': [ # Versions 2.0 and later


The FSF wording for this isn't “Versions 2.0 and later”, it's “versions 2.0 and 2.1”.

wking · 2017-10-23T17:00:04Z

pull.py

    'FreeBSDDL': ['FreeBSD'],  # unify (multi-tag)
-    # FIXME: still working through this
+    'NPL': [ #any version


The FSF wording for this isn't “any version”, it's “versions 1.0 and 1.1”.

wking · 2017-10-23T17:05:02Z

pull.py

@@ -57,8 +57,40 @@
        'FDLv1.2',
        'FDLv1.3',
    ],
+    'FreeArt': [ # any version
+        'LAL-1.2',


Here is another case where the FSF doesn't say “any version”. The version they link is 1.3. Until we get clarity on this, can we drop the SPLITS entry (and the associated LAL-1.2 IDENTIFIERS entry) and come back to them in follow-up work?

They also link the English translation, while we currently include only the canonical French version. Until we sort out translations (spdx/license-list-XML#438), we should remove the LAL-1.3 IDENTIFIERS entry as well.

goneall · 2017-10-23T18:48:56Z

@wking I'm not going to have much time to work on this today - do you mind taking the PR and adding your recommended changes above? They all seem reasonable.

wking · 2017-10-23T22:02:04Z

… do you mind taking the PR and adding your recommended changes above?

Done (with a few more as well) in my splits-identifiers branch. Differences vs. your current tip:

$ git diff -U0 origin/pr/3..splits-identifiers
diff --git a/pull.py b/pull.py
index b37d874..6edc611 100755
--- a/pull.py
+++ b/pull.py
@@ -60,6 +60,2 @@ SPLITS = {
-    'FreeArt': [ # any version
-        'LAL-1.2',
-        'LAL-1.3',
-    ],
-    'FreeBSDDL': ['FreeBSD'],  # unify (multi-tag)
-    'NPL': [ #any version
+    'FreeBSDDL': ['FreeBSD'], # unify (multi-tag)
+    'NPL': [ # versions 1.0 and 1.1
@@ -76,17 +72,3 @@ SPLITS = {
-    'RPL': [ # any version - Note that FSF website does not state any version, but references version 1.3 in the URL.  It is assumed that it also covers version 1.1 and 1.5, but this should be verified with FSF.
-        'RPL-1.1',
-        'RPL-1.3',
-        'RPL-1.5',
-    ],
-    'Unicode': [ # any version
-        'Unicode-DFS-2015',
-        'Unicode-DFS-2016',
-    ],
-    'W3C': [ # any version
-        'W3C',
-        'W3C-20150513',
-        'W3C-19980720',
-    ],
-    'Zope2.0': [ # Versions 2.0 and later
-        'ZPL-2.0',
-        'ZPL-2.1',
+    'Zope2.0': [ # versions 2.0 and 2.1
+        'Zope2.0',
+        'Zope2.1',
@@ -147,3 +129,3 @@ IDENTIFIERS = {
-    'FDL1.1': {'spdx': 'GFDL-1.1'},
-    'FDL1.2': {'spdx': 'GFDL-1.2'},
-    'FDL1.3': {'spdx': 'GFDL-1.3'},
+    'FDLv1.1': {'spdx': 'GFDL-1.1'},
+    'FDLv1.2': {'spdx': 'GFDL-1.2'},
+    'FDLv1.3': {'spdx': 'GFDL-1.3'},
@@ -165,2 +146,0 @@ IDENTIFIERS = {
-    'LAL-1.2': {'spdx':'LAL-1.2'},
-    'LAL-1.3': {'spdx':'LAL-1.3'},
@@ -196 +175,0 @@ IDENTIFIERS = {
-    'Python': {'spdx': 'Python-2.0'}, # Note: references 'later versions which are not in the SPDX license list
@@ -198,2 +176,0 @@ IDENTIFIERS = {
-    'RPL-1.1': {'spdx': 'RPL-1.1'},
-    'RPL-1.5': {'spdx': 'RPL-1.5'},
@@ -203 +179,0 @@ IDENTIFIERS = {
-    'SISSL': {'spdx': 'SISSL'}, # Note that the header on the 'FSF website states version 1.0, but the link points to 'version 1.1.  The SPDX license is version 1.1
@@ -206,2 +181,0 @@ IDENTIFIERS = {
-    'Unicode-DFS-2015': {'spdx': 'Unicode-DFS-2015'},
-    'Unicode-DFS-2016': {'spdx': 'Unicode-DFS-2016'},
@@ -212,2 +185,0 @@ IDENTIFIERS = {
-    'W3C-20150513': {'spdx': 'W3C-20150513'},
-    'W3C-19980720': {'spdx': 'W3C-19980720'},
@@ -223,4 +195,2 @@ IDENTIFIERS = {
-    'ZPL-2.0': {'spdx': 'ZPL-2.0'},
-    'ZPL-2.1': {'spdx': 'ZPL-2.1'},
-
-    # FIXME: still working through this
+    'Zope2.0': {'spdx': 'ZPL-2.0'},
+    'Zope2.1': {'spdx': 'ZPL-2.1'},
@@ -236,0 +207 @@ def extract(root, base_uri=None):
…

If that looks acceptable to you, let me know, and I'll push it and close this PR. Then we can open follow-up issues/PRs for the questionable splits and identifiers.

wking · 2017-10-23T22:37:39Z

I pushed 50e1556 → ef88572 to my branch fixing the oldOpenLDAP association to be SPDX's OLDAP-2.3.

wking · 2017-11-11T21:42:44Z

@goneall, have you had time to look over my splits-identifiers reroll yet? It's still at ef88572.

goneall · 2017-11-12T00:04:59Z

@wking Just reviewed it, looks fine to me.

wking · 2017-11-12T00:43:06Z

Merged and published the ef88572 reroll.

goneall added 3 commits October 21, 2017 15:13

Update identifiers and splits

6f84754

Signed-off-by: Gary O'Neall <gary@sourceauditor.com>

Merged master

94eb095

Signed-off-by: Gary O'Neall <gary@sourceauditor.com>

Added identifiers up through the letter N

87db910

Signed-off-by: Gary O'Neall <gary@sourceauditor.com>

wking reviewed Oct 22, 2017

View reviewed changes

goneall added 8 commits October 22, 2017 10:44

Merge branch 'master' into add-identifiers-go

c7aface

Fix error for MS-PL spdx ID per pull request review

eb672a3

Change NoLicense from NOASSERTION to NONE per pull request review com…

baf1bad

…ment

Add remaining license IDs and splits - please note the comments on th…

74f3965

…e license IDs as some of these required judgement as to the associated SPDX license ID

Add a list of unassociated SPDX license IDs

5e63316

Change ecos 2.0 SPDX identifier to a non-deprecated license expression

8cde30a

Change ecos 2.0 SPDX identifier to a non-deprecated license expression

e679630

Merge branch 'add-identifiers-go' of https://github.com/goneall/fsf-api…

6bfab5a

… into add-identifiers-go

trying to clean up last commit - should fix incorrect ECL2.0

2056ef4

wking reviewed Oct 22, 2017

View reviewed changes

Take a more conservative approach and remove the more ambiguous licen…

e9b2687

…se matches to the spdx license list and remove the unassociated-license-ids.md file per pull request review

Correct tags for ccbysa and ccby

eb9e7c3

wking reviewed Oct 23, 2017

View reviewed changes

wking mentioned this pull request Oct 23, 2017

Add FsfLibre flag to GPL-2.0 spdx/license-list-XML#453

Closed

wking closed this Nov 12, 2017

This was referenced Nov 14, 2017

Review FSF referenced licenses which do not have an associated SPDX license ID #4

Open

FSF mentions 1.0 for SISSL but links to the 1.1 text #8

Open

wking mentioned this pull request Apr 21, 2018

Repository license should be one of the GNU family of licenses #18

Open

		@@ -0,0 +1,47 @@
		# The following FSF license tags did not have any obvious match to an SPDX license

Add identifiers and splits #3

Add identifiers and splits #3

Conversation

goneall commented Oct 21, 2017

goneall commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking commented Oct 22, 2017

goneall commented Oct 22, 2017

goneall commented Oct 22, 2017

goneall commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goneall commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goneall commented Oct 23, 2017

wking commented Oct 23, 2017 • edited Loading

wking commented Oct 23, 2017

wking commented Nov 11, 2017

goneall commented Nov 12, 2017

wking commented Nov 12, 2017

wking Oct 22, 2017 •

edited

Loading

wking commented Oct 23, 2017 •

edited

Loading