Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] licenses and generate-disclaimer #1164

Closed
matzeeable opened this issue Apr 4, 2020 · 12 comments
Closed

[Question] licenses and generate-disclaimer #1164

matzeeable opened this issue Apr 4, 2020 · 12 comments

Comments

@matzeeable
Copy link

Hi yarn team,

Awesome work on berry! I am working on an open source project (https://github.com/devowlio/wp-react-starter) and want to migrate yarn to berry. I want to benefit from faster install times also on my CI pipeline. Generally, I had a look at "issues" which can be caused by upgrading. The first thing I note:

yarn licenses
yarn generate-disclaimer

What happened to this commands? They seem not to be available / ported. Are they on the roadmap? Are there better alternatives?

Best regards,
Matthew 🙂

@arcanis
Copy link
Member

arcanis commented Apr 4, 2020

Hello 👋

We felt like those commands had a too fringe use case to be part of the core offering. Additionally there are various ways such tools could be expanded and configured, and trying to support them (and keep up with feature requests) would put an unwanted additional burden on our team.

The good news is, those commands are exactly the kind of thing we had in mind when designing the plugin API, so it's entirely possible to implement them in userspace (and share your implementation with the community if you so wish).

The plugin documentation is admittedly a bit sparse at the moment as we're working on improving that. In the meantime, you can check the plugin tutorial on the website and the plugin sources in this repo to get an idea of how it works.

@matzeeable
Copy link
Author

Hi again!

Thanks for your fast response. I had a look at the codebase unfortunately I do not really understand how to achieve the following thing:

Get all dependencies for a given package.json including deep dependencies including the metadata (license, version, ...). Can you give me a tip? Consider that yarn classic had an issue with workspaces: yarnpkg/yarn#5174 - is this resolved in berry? I dived into this a bit deeper and the following code snippet gives me all dependencies if I run it inside a package (also respects workspaces):

import { resolve } from "path";

class ProductionDependencies {
    private all: string[] = [];

    constructor(packageJsonPath: string) {
        this.fetchAll(Object.keys(require(packageJsonPath).dependencies));
    }

    public fetchAll(deps: string[]) {
        for (const dep of deps) {
            if (this.all.indexOf(dep) === -1) {
                // eslint-disable-next-line @typescript-eslint/no-var-requires
                const pkg = require(require.resolve(`${dep}/package.json`));
                this.fetchAll(Object.keys(pkg.dependencies || [])); // devDependencies is not needed for license check and disclaimer
                console.log(pkg.name, pkg.license || pkg.licenses || "UNKNOWN");
            }
        }
        this.all.push(...deps);
    }
}

new ProductionDependencies(resolve("./package.json"));

I am thinking of creating a public package - I also do not promise if I create the package because perhaps someone maintaining a similar package is open to create a yarn plugin. I do not know yet if it will be a yarn plugin or a standalone package so it can be used with npm, too. The most important part for me is the workspace-compatibility.

So, I will note further things in this issue (just for myself, perhaps it can be interesting for others):

  1. Obtain a list of all dependencies with their metadata of a package.json (see above code snippet)
  2. Configure a whitelist for packages and skip their license check ["create-react-context@0.2.3"]: string[]
  3. Configure a whitelist for allowed licenses (string[]) for the license check
  4. Check the license meta and fallback to UNKNOWN
  5. If the found license is not UNKNOWN:
    1. Check if license starts with ( and pass it through spdx-expression-parse. Afterwards do the license check based on the parsed expression
    2. Otherwise check if license is a string[] and if not, parse it to a string[]
    3. Pass the license array to spdx-correct and do the license check based on the result
  6. Otherwise obtain the license from other files (mark "guess" with e. g. MIT*, and print a warning)
    1. scripts/generate-licenses-js.js generates a list of unique parts of LICENSE texts so they can be infered when package.json#license cannot be resolved correctly. The generated RegEx parts are found here: scripts/licenses
    2. src/util/normalize-manifest/infer-license.js is the programmatically way to infer a license from a string (e. g. LICENSE, NOTICE, COPYING or README file)
  7. Throw an error of not found licenses. The developer has three options to correct the issue
    • Add the package to the whitelist for packages (the developer opts-in to skip that license)
    • Contact the developer of the package and ask to create a valid license in their package.json file
    • Contribute to this package' compat table
  8. Throw an error if license check failed for a package. The developer has three options:
    • Add the license of the package to the allowed licenses
    • Add the package to the whitelist for packages (the developer opts-in to skip that license)
    • Search an alternative package

The generate-disclaimer I think is more easier, but I will dive into this later.

This are just thoughts how it can be implemented - I got inspired by the already implemented yarn licenses list and license-checker.

@mhassan1
Copy link
Contributor

I've just pushed up a plugin that adds yarn licenses: https://github.com/mhassan1/yarn-plugin-licenses

It only does yarn licenses list right now, inspired by Yarn v1. We can add more commands in the future. Also, it only works with the latest Berry sources from master (it uses treeUtils), so it's not quite production ready.

Check it out and let me know if you have any feedback.

@mhassan1
Copy link
Contributor

@arcanis I've written a plugin that provides yarn licenses list: https://github.com/mhassan1/yarn-plugin-licenses. Assuming there's interest in the community, what would be the best way to share it?

@arcanis
Copy link
Member

arcanis commented Oct 15, 2020

I think we should add a "community plugins" page on the website to list those 🤔

@mhassan1
Copy link
Contributor

Did you have something in mind? I'd be happy to contribute that to the website, but I'm not sure where you would want it, or how you would want to manage it.

@noahnu
Copy link
Contributor

noahnu commented Feb 26, 2021

I also created an open source plugin for auditing licenses: https://github.com/tophat/yarn-plugin-licenses. It's still early stage, and needs some cleanup to the project (mainly tests), but as it's written right now, it can complement @mhassan1's plugin, as the commands don't conflict (yarn licenses audit).

@matzeeable
Copy link
Author

@noahnu thanks for your reply. Does the plugin you provide support per-package audit? We need to manage multiple licenses per project. :-)

@noahnu
Copy link
Contributor

noahnu commented Feb 27, 2021

@noahnu thanks for your reply. Does the plugin you provide support per-package audit? We need to manage multiple licenses per project. :-)

@matzeeable I plan on cleaning it up a bit. At the moment it lets you specify a license validator (RegEx/predicate function) in a config file and then runs your license validator against all licenses for all installed dependencies (including transitive dependencies) in your project. It runs against the entire project at the moment, but wouldn't be much work to constrain the scope to a specific workspace, and allow finer filtering on dev dependencies vs. prod depepndencies vs. peer, etc.

It doesn't generate disclaimers. You can use @mhassan1's plugin for that.

@Eli-Black-Work
Copy link

Adding to the list:

We use license-checker-webpack-plugin. It's a license checker that's deeply integrated with Webpack, so it shows you only the licenses of packages that are still left after tree shaking 🙂

@merceyz
Copy link
Member

merceyz commented Jul 16, 2022

Closing as the question has been answered and the documentation now links to various license related plugins.

@merceyz merceyz closed this as completed Jul 16, 2022
@jeroenhabets
Copy link

For people, stumbling upon this ticket via Getting Started > CLI Commands > Removed from core, please note this plugin is listed on Features > Plugins > Contrib plugins

P.S. Especially for people like me who overlook Kurt-von-Laven's mention on Apr 29, 2021 🤭

@yarnpkg yarnpkg locked and limited conversation to collaborators Jun 14, 2023
@merceyz merceyz converted this issue into discussion #5503 Jun 14, 2023

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants