Skip to content

GSOC 2020: Validate license cross references

Jump to bottom
Smith Tanjong Agbor edited this page Aug 30, 2020 · 2 revisions

Validate license cross references

Project abstract

The SPDX license list is generated from a git repository of XML files. One of the fields maintained in the XML is the crossRef which is a URL cross reference for the license which may be valid or it may also be a "dead link". The LicenseListPublisher is the tool that generates the web pages and the output formats. The output formats can be found in the SPDX license list data git repository. Issue #60 for the LicenseListPublisher describes a request to include a validity attribute.

What is the project about? Why is it important?

This project’s main focus is on establishing whether a given license url is valid, and checking its license text to determine whether it matches the current license text. It is important because license urls could be annotated with their validity status, and having licenses that do not have a license text could be a problem and must be identified easily, to better provide the license text.

Project description

The target repository is License list publisher, which is the source code repository for the tool that generates license data found in the license-list-data repository. The source for the data is located in the license-list-XML repository.

This project initially was two fold, but after discussing with the mentor, there are a couple of things I needed to work on.

Link validity

Check if a license url is valid. This should be seamless and using a timeout will be too slow, so, the best solution will be evaluated based on the performance.

Url live status;

Check if the url in the license (crossref) is live or not.

Url live status;

Check if the url in the license (crossref) is live or not.

Url is wayback link status;

Check if the url in the license (crossref) is from the wayback machine or not.

License text match status;

Compare the license text obtained from the url(if the url is valid) to the current license text and determine whether they match.

Timestamp;

The timestamp in UTC at which all the above operations were done

All the above items are fields to be added to the licenses generated by the LicenseListPublisher repo, having as source of the xml files the LicenseListXML repo.

Every xml license in the repo has a list of urls in the “crossRefs” tag. The urls listed in this tag are our focus. This project seeks to sort of expand those urls with details that can be of help. So for each of those urls, we should get details as mentioned in the parameters above and add them to the generated license.

There are 7 formats of licenses after generation by the LicenseListPublisher repo. They are:

  • html
  • json
  • jsonld
  • rdfa
  • rdfnt
  • rdfturtle
  • rdfxml

Initially I went for a unified format of the expanded license parameters, and this format is of the type:

"{
    url: https://creativecommons.org/publicdomain/zero/1.0/legalcode,
    isValid: true,
    isLive: false,
    isWayBackLink: false,
    match: null,
    timestamp: 2020-08-25 - 11:36:35
}"

Which is a string.

Requirements:

Check for license link validity

This stackoverflow answer gives a concise solution to this issue, by making use of a stream and just checking the http head returned, to establish if the url is valid or not. Write a java function to make use of the answer above to find link validity.

If link is valid, check if license from url matches

Parse the xml file of the license

Read the xml file

Validate the xml before parsing

Examine elements: Parse

Get the complete license text from the xml file. This was gotten from parsing the xml.

Get the license url(s)

Use the url to execute (1) above. If the link is valid, as established by (1) above, get the content of the html by making another http request. This time, the content of the html was gotten, not just the HEAD as in the previous request.

Parse the html content to get the complete license text. I used jsoup for this.

Compare the license text from the html to that from the xml file

Observe the performance of the above tasks, and optimize

Test the entire process manually, and write tests

Pull requests made:

Work done:

  • Discuss project requirements
  • Establish work plan moving forward
  • Do necessary research on tools that could be of help(jsoup)
  • Propose display format with tech and legal teams
  • Changed the design for more decoupling from the tools repo
  • Use URLValidator to check url validity
  • Remove whitespace in files in repo
  • Write methods for the following checks:
  • url validity
  • url live status
  • url is wayback url?
  • timestamp
  • url match?
  • Changed license match check for more convenience to the legal team
  • Improve performance by use of multi-threading
  • Optimize performance improvements to account for errors
  • Write class for Crossrefs details(in case there are changes in the future) with string and json possible formats

Current status:

  • Working on the method to handle the comparison between the license text from the html and that from the xml file; to populate the “match” parameter in the crossref details. Given that there could be some optional text in the license xml and not in the html(from the url), the main task is to identify this optional text and remove it from the license text, prior to any text comparison. This entails obtaining the start and end text of the license from the html(gotten from the crossref url) without the optional text.

Remaining work to be done:

Clone this wiki locally