Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate dependencies licenses #836

Merged

Conversation

daniel-shuy
Copy link
Contributor

@daniel-shuy daniel-shuy commented Oct 18, 2020

Closes #449

This PR adds Makefile tasks to validate go-mod and npm dependency licenses to ensure they are compatible with the project's license (Apache-2.0).

I used go-licenses for validating go-mod dependencies, and diligent for validating npm dependencies.

Currently, I have only added the tasks to the Makefile, but I haven't added them to the GitHub Actions workflow file(s). Which workflow should I bind it to?

Ideally I would've liked to use diligent for both go-mod and npm dependencies, as it claims to be able to do so, but unfortunately I couldn't get it to work with go-mod.

Since I couldn't use a single tool for both go-mod and npm, I could've used a different tool for validating npm dependencies, however I couldn't find another one than diligent that allowed specifying the license type (eg. permissive). diligent also has the advantage of being a Go module.

Also, I had to update base-64 to 1.0.0 because it had the license information in the wrong place, which was causing diligent to detect it as license: none (see mathiasbynens/base64#24).

Also, note that there is an issue with diligent that prevents it from working with NPM scoped packages (eg. @primer/octicons-react). I have created an issue for this (senseyeio/diligent/issues/77). As a workaround, I have configured diligent to ignore scoped packages.

Note that go-licenses ignores Go modules without any license (or modules that it fails to detect a license for), whereas diligent will fail validation if it cannot find an NPM package or a license for an NPM package (which is why the workaround above to ignore scoped packages is required).

Example make check-licenses-go-mod output (this is considered passing despite all the warnings):

/home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/bin/go-licenses check ./cmd/lakefs
E1019 00:27:15.115029   31731 library.go:108] Failed to find license for github.com/josharian/intern: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/josharian/intern@v1.0.0
E1019 00:27:20.032481   31731 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0
E1019 00:27:20.048400   31731 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/cwriter: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/cwriter
E1019 00:27:20.059830   31731 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/decor: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/decor
E1019 00:27:20.078421   31731 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/internal: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/internal
E1019 00:27:20.686697   31731 library.go:108] Failed to find license for github.com/cznic/mathutil: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/cznic/mathutil@v0.0.0-20180504122225-ca4c9f2c1369
/home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/bin/go-licenses check ./cmd/lakectl
E1019 00:27:33.077803   32116 library.go:108] Failed to find license for github.com/josharian/intern: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/josharian/intern@v1.0.0
E1019 00:27:40.884138   32116 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0
E1019 00:27:40.898762   32116 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/cwriter: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/cwriter
E1019 00:27:40.910413   32116 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/decor: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/decor
E1019 00:27:40.925479   32116 library.go:108] Failed to find license for github.com/vbauerster/mpb/v5/internal: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/vbauerster/mpb/v5@v5.3.0/internal
E1019 00:27:41.558124   32116 library.go:108] Failed to find license for github.com/cznic/mathutil: no file/directory matching regexp "^(LICEN(S|C)E|COPYING|README|NOTICE)(\\..+)?$" found for /home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/pkg/mod/github.com/cznic/mathutil@v0.0.0-20180504122225-ca4c9f2c1369

Example check-licenses-npm output:

# The -i arg is a workaround to ignore NPM scoped packages until https://github.com/senseyeio/diligent/issues/77 is fixed
/home/daniel-shuy/.asdf/installs/golang/1.15.3/packages/bin/diligent check -w permissive -i ^@[^/]+?/[^/]+ webui
acorn                      MIT License
base-64                    MIT License
bootstrap                  MIT License
bootswatch                 MIT License
http-proxy-middleware      MIT License
moment                     MIT License
react                      MIT License
react-bootstrap            MIT License
react-bootstrap-typeahead  MIT License
react-dom                  MIT License
react-redux                MIT License
react-router-dom           MIT License
react-scripts              MIT License
redux                      MIT License
redux-thunk                MIT License

@CLAassistant
Copy link

CLAassistant commented Oct 18, 2020

CLA assistant check
All committers have signed the CLA.

@daniel-shuy daniel-shuy force-pushed the feature/validate-dependencies-licenses branch from a222016 to de80b2a Compare October 18, 2020 17:11
@codecov-io
Copy link

codecov-io commented Oct 18, 2020

Codecov Report

Merging #836 into master will decrease coverage by 0.05%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #836      +/-   ##
==========================================
- Coverage   42.89%   42.84%   -0.06%     
==========================================
  Files         135      135              
  Lines       10574    10574              
==========================================
- Hits         4536     4530       -6     
- Misses       5448     5451       +3     
- Partials      590      593       +3     
Impacted Files Coverage Δ
catalog/cataloger_commit.go 75.75% <0.00%> (-6.07%) ⬇️
catalog/cataloger_create_entry.go 94.73% <0.00%> (-5.27%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d56db9e...de80b2a. Read the comment docs.

Copy link
Contributor

@arielshaqed arielshaqed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @daniel-shuy ! This would be a nice safety net for us.

I'll ask @ozkatz to have a look at it, it does make sense but we need to know we're checking for the right things.

check-licenses-npm:
go get github.com/senseyeio/diligent/cmd/diligent
# The -i arg is a workaround to ignore NPM scoped packages until https://github.com/senseyeio/diligent/issues/77 is fixed
$(GOBINPATH)/diligent check -w permissive -i ^@[^/]+?/[^/]+ $(UI_DIR)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ozkatz for "permissive" here: described here as:

Open Source software that is made available under "non-copyleft" licenses. These generally require
attribution of the included open source and may include other obligations
and you get to pick any other category you like from there


check-licenses-go-mod:
go get github.com/google/go-licenses
$(GOBINPATH)/go-licenses check ./cmd/$(LAKEFS_BINARY_NAME)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ozkatz here we could also add a check category, I guess.

@ozkatz ozkatz merged commit 0b2f3ba into treeverse:master Oct 20, 2020
@arielshaqed
Copy link
Contributor

Thanks!

@daniel-shuy daniel-shuy deleted the feature/validate-dependencies-licenses branch October 20, 2020 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI step: validate dependencies comply with license
5 participants