Run check of dependency licences

[sebbASF]

What should happen

The skill should determine the licences for all dependencies and whether or not these dependencies are optional.

Why

For Apache projects, there are restrictions on which licences are compatible, see
https://apache.org/legal/resolved.html

Similar considerations may apply to projects with other licenses.

Which layer

Skill (.claude/skills/<name>/)

Boundary conditions (optional)

No response

Out of scope (optional)

No response

References (optional)

No response

[justinmclean]

ASF policy says https://www.apache.org/legal/resolved.html#optional, so optional dependencies IMO do not need to be checked or rather Category X optional dependencies are allowed.

[justinmclean]

Are you referring to the license-compliance-audit skill? If so, it doesn't scan dependencies at all, it checks the root LICENSE file, the NOTICE file, and SPDX headers, etc. Or perhaps you mean the dependency-audit skill? If so, it checks dependencies, but for known vulnerabilities, not licenses. A skill to scan dependency licenses doesn't exist yet.

[justinmclean]

Is this something you'd be willing to contribute? Happy to point you to the recently created education resources [1] to help you do this. Otherwise, I'll add it to the backlog.

1. https://magpie.apache.org/docs/education/readme/

[sebbASF]

In any case, all dependencies will have to be discovered initially.

Also, in the case of optional dependencies, the user still needs to know what license they use in case their organisation disallows some licenses.

The user needs to be given the information so they can make an informed choice.

[potiuk]

In any case, all dependencies will have to be discovered initially.

Yeah, I think @sebbASF you misunderstood the purpose of this one- as Justin explained.

This is a generic - not ASF specific SKILL - it's in non-ASF specific repo health family of skills. We might want to - indeed - add some specific asf-nlicence-compliance in the ASF-specific family of skills and such contributions are welcome (and will most likely happen).

The families of skills are described here - https://magpie.apache.org/#skill-families - with ASF association where it's present.

Just to explain - Magpie is not a tool to enforce "Apache Way" - that is out-of-the scope of our work. We provide mostly generic "OSS" skills that any repo can use, so we even "cannot" (and we even have tests for it) - add ASF-specific rules in generic rules (we are both Organization-Neutral and Vendor-Neutral). The ASF-specific SKILLs are currently experimental and we can channel them from Comdev/ Incubator - discussion is open about this - and there, yes, absolutley discussion about shape of SKILLs for ASF specific things is most welcome- and if Incubator/Comdev would like to provide those SKILLs from their repos - we are prepared for that and discussion is on-going.

Nothing we have here - including in ASF families - is normative or "should" be done - we can discuss what is in the generic skils and have different opinions on those of course.

And you are absolutely welcome to propose ASF-related skill for that in the corresponding ASF family of skill. PRs are most welcome.

Let me close this one. But if you think there is somethig to improve in the generic skills - also PRs are most welcome. But "Apache Way" of doing things there is not relevant - they are 'generic" skills.

BTW. This is one of the reasons we have Magpie as separate PMC - because our goal is mostly to produce "generic" SKILLs that are usable by anyone - no matter if they are ASF projects or not. The "ASF" or (soon i hope) "PSF" types of skills are more of "extensions" that we keep as a convenient way for organizations to have their own SKILLS - for now committed in Magpie for ASF, but I would really like them to be eventually managed in ComDev or Incubator or both.

Maybe @sebbASF - you would like to take on the task on agreeing that split and taking leadership in moving those SKILLs if others agree it's a good idea.

Detailed discussion about it here: https://lists.apache.org/thread/4t0wcz073f9owf35t4yhsh34zn47nrhm - also we have support for that in https://magpie.apache.org/docs/adapters/registry/#community--external-adapters

It would be good idea if someone takes leadership on clarifying it.

[sebbASF]

See #814 which does most of this enhancement suggestion. However it does not currently appear to cover optionality.

[justinmclean]

Yep, that is a known limitation. Optional deps don't appear in the resolved tree at all, so the skill never sees them. Discovering them means going beyond what the dependency manager resolves, and they can be declared almost anywhere, so discovery can range from hard to basically impossible. If you have any suggestions, I'll try to implement them.

[sebbASF]

Doesn’t that depend on why the dependency is optional?
For example, if a product has an optional component, won’t its dependencies be discoverable?
I think the hard part here may be in determining which components are optional, and which are essential.

[justinmclean]

Not always, no, well not easily anyway, dependencies can be installed out of band: by a script, at runtime, or by asking the user to install them. Things like that never show up in any manifest, so it's hard to find them. It's possible to declared-optional ones in some build tools, but that varies by the tool and the project.

Spark is a good example, connectors like spark-sql-kafka-0-10 are pulled in at job submit with spark-submit --packages, so they're in no build manifest and no simple audit can see them.

Superset deliberately ships with no database drivers. From its own docs: "Superset does not ship bundled with connectivity to databases... the main step in connecting Superset to a database is to install the proper database driver(s) in your environment." So you install the driver for your database yourself: pip install psycopg2 for Postgres, pip install mysqlclient for MySQL, and so on. As of 4.1.0 they even removed the MySQL driver from the default Docker images. MySQL is is GPL-licensed.

[potiuk]

Yes. This is really human-judgment. But yes - likely it woudl be useful if we had .asf.yaml or other features that would allow ASF project to define which dependencies are optional.

But this is not something of the concern of this PMC. The 'asf` skills are not normative herem they are helpful, but if we seek for something more precise - I guess work on it by someone to introduce and deploy some kind of standard approach there should be very welcome I think.

[sebbASF]

It is not just the ASF who care about licenses.

Other organisations wishing to use a product will also have certain prohibited licences as some licences are much stricter than AL 2.0.

In which case it is vital to know which licences are optional, as they may not need the part that relies on a prohibited licence.

Again: the user needs to be given the information needed to make an informed choice.

[justinmclean]

I agree, but there's no easy way to know, for a given piece of software, which optional parts could be added. As you can see from the Superset example, no tool can easily determine what that project's optional software is, let alone how it is licensed.

You can specify what licenses are allowed and which are not, so this skill can be used outside the ASF. (See the allowed_licenses and forbidden_licenses policy selection config.)

[sebbASF]

Superset is actually a good example of where the user does have full visibility of the licence, precisely because the user is responsible for providing the optional software.

My main concern is where the product ships with code that requires downloading a 3rd party product.
Is that code optional, or is it fundamental to the operation of the product?