Draft licenses CLI feature #1433

JohnHardy · 2022-10-13T11:21:22Z

Pull Request Check List

A news fragment is added in news/ describing what is new.
Test cases added for changed code.

Describe what you have changed in this PR.

Added a new CLI command licenses that allows you to output the license documentation for all the deps. This usually requires 3rd party projects that may or may not be compatible with whatever packaging system is used. This CLI command implements common use cases that I have seen in the wild.

🙏🙏 If you are interested in merging this PR in, please let me know so I will invest the time to lint / write test cases. However, before that, I wanted to make sure that you were happy with: (1) the existence of the feature, (2) the implementation details, my first time working on a package manager, (3) the ux. 🙏🙏

Features are:

Licenses can be gathered from the working set or resolved using the resolver/lockfile.
Output to CSV
Output to JSON
Output to console in the fancy PDM tables
Output as markdown (including license text files included in dist-info)
If a License field is empty, it will try to recover it from the Trove classifiers.
Highlight the group a requirement is defined in (eg. dev deps)
Distinguish between specified deps and sub-dependencies

Here are some use cases:

pdm licenses

pdm licenses --resolve --skip default,tox,workflow,sub --fields group,identifier,version,licenses --sort licenses

pdm licenses --sort licenses

pdm licenses --sort licenses --csv

pdm licenses --sort licenses --markdown > mylicenses.md

for more information, see https://pre-commit.ci

codecov-commenter · 2022-10-13T11:32:45Z

Codecov Report

Base: 84.97% // Head: 85.11% // Increases project coverage by +0.14% 🎉

Coverage data is based on head (b1815a1) compared to base (1e5dc62).
Patch coverage: 89.79% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1433      +/-   ##
==========================================
+ Coverage   84.97%   85.11%   +0.14%     
==========================================
  Files          88       88              
  Lines        7662     7836     +174     
  Branches     1798     1852      +54     
==========================================
+ Hits         6511     6670     +159     
- Misses        766      779      +13     
- Partials      385      387       +2

Flag	Coverage Δ
unittests	`84.92% <89.79%> (+0.14%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/pdm/cli/actions.py	`79.46% <ø> (-1.05%)`	⬇️
src/pdm/cli/commands/list.py	`90.19% <89.74%> (-9.81%)`	⬇️
src/pdm/cli/utils.py	`81.26% <100.00%> (+2.11%)`	⬆️
src/pdm/models/repositories.py	`78.04% <0.00%> (-0.82%)`	⬇️
src/pdm/termui.py	`89.85% <0.00%> (-0.73%)`	⬇️
src/pdm/models/requirements.py	`95.10% <0.00%> (+0.30%)`	⬆️
src/pdm/resolver/providers.py	`94.57% <0.00%> (+0.77%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

frostming · 2022-10-13T13:30:09Z

The output is very similar to pdm list and it is preferred to extend that command rather than adding a new one. Or is it better to live in its own plugin?

JohnHardy · 2022-10-13T14:02:04Z

@frostming sounds good to me, but it will impact other functions inside pdm like show_dependency_graph , actions.py, etc. I was reluctant to pollute do_list since I don't know the codebase. What do you prefer?

Shall I keep the switches for selecting fields, skipping dep groups, changing the sort column, etc?

EDIT: It could also live in a plugin - happy to follow your guidance and suggestions from others.

frostming · 2022-10-14T02:23:35Z

@frostming sounds good to me, but it will impact other functions inside pdm like show_dependency_graph , actions.py, etc. I was reluctant to pollute do_list since I don't know the codebase. What do you prefer?

Shall I keep the switches for selecting fields, skipping dep groups, changing the sort column, etc?

EDIT: It could also live in a plugin - happy to follow your guidance and suggestions from others.

The column selection is great, let's revisit the current options and the options to be added, if it is to be integrated in to pdm list:

Format/Option	`--fields`	`--sort`	`--resolve`	`--license`	`--reverse`	`--include/--exclude`¹
table	✅	✅	✅	✅	⛔	✅
csv	✅	✅	✅	✅	⛔	✅
json	✅	✅	✅	✅	⛔	✅
markdown	✅	✅	✅	✅	⛔	✅
freeze	⛔	⛔	⛔²	⛔	⛔	⛔
graph³	⛔	⛔	✅	⛔	✅	✅

I hope that doesn't make the feature too complex.

To replace the proposed --skip, they should only be allowed with --resolve, and include is prior to exclude ↩
should use pdm export --format requirements instead ↩
All format options are exclusive except --graph can be given with --json ↩

…nses

for more information, see https://pre-commit.ci

JohnHardy · 2022-10-15T01:10:38Z

I had another quick go at this again to match your feature matrix, let me know what you think!

Let me know if I missed anything or you would like something to be done differently.

Questions:

Currently using --include * to mean include everything. Do you want it to match pdm install -G:all style instead?
Currently using --exclude sub to mean exclude sub-dependencies, but sub seems like an unclear choice of word. Any suggestions?
At the moment there is the case where cachecontrol and cachecontrol[filecache] reference the cachecontrol package (seems right). Is there a case where they should be distinguished by name? Is there a way of mapping distributions back to requirement identifiers.
At the moment, the resolver works on ALL groups BEFORE being filtered and listed. However, we might want to send less packages to the resolver. Only I thought that could potentially resolve different versions / licenses. Probably nobody will care, but just wondering.

If you are happy, I can make some test cases.

frostming · 2022-10-15T02:25:15Z

Currently using --include * to mean include everything. Do you want it to match pdm install -G:all style instead?

The current implementation LGTM.

Currently using --exclude sub to mean exclude sub-dependencies, but sub seems like an unclear choice of word. Any suggestions?

Two options, 1) use a special identifier to avoid conflicting with existing groups, such as :sub 2) use another option flag --exclude-sub

At the moment there is the case where cachecontrol and cachecontrol[filecache] reference the cachecontrol package (seems right).

IMO it is okay to ignore the entries with extras []

for more information, see https://pre-commit.ci

JohnHardy · 2022-10-15T17:17:19Z

👍
Done, :sub is used. 👍
Sounds good. 👍 FYI: I noticed that the echo function will trim word[here] to word even if the string does not contain [/] to close it.
I am updating the tests now. Please can you advise how to mock the Command functions?

For example:

def test_list_command(project, invoke, mocker):  # previous test
    do_list = mocker.patch.object(actions, "do_list")
    invoke(["list"], obj=project)
    do_list.assert_called_once()   # <--- works OK in old codebase
    
def test_list_command(project, invoke, mocker):  # proposed new test
    handle = mocker.patch.object(Command, "handle")
    invoke(["list"], obj=project)
    handle.assert_called_once()   # <--- list.Command.handle is not called

frostming · 2022-10-17T01:40:16Z

3. FYI: I noticed that the echo function will trim word[here] to word even if the string does not contain [/] to close it.

Yeah, that is the markup syntax of rich console, [] need to be escaped by \

4. I am updating the tests now. Please can you advise how to mock the Command functions?

These tests around do_list will no longer be valid, feel free to remove or refactor them.

P.S. I've converted this PR to draft, until it is ready to review.

for more information, see https://pre-commit.ci

tests/conftest.py

tests/cli/test_list.py

JohnHardy · 2022-10-21T22:41:55Z

Added a few of comments on the code where I need some help/advice about what the correct behaviour should be. Thanks!

src/pdm/cli/commands/list.py

tests/cli/test_list.py

frostming · 2022-10-24T04:10:14Z

tests/cli/test_list.py

+# # matches the environment or hashes\nAdd '-v' to see the detailed traceback\n"
+# @pytest.mark.usefixtures("working_set", "repository")
+# def test_list_bare_sorted_version_resolve(project, invoke, repository):
+#     # project.environment.python_requires = PySpecSet(">=3.6")


You need to mock the .metadata(or .prepare_metadata()) cuz it will be accessed during the command, which will try to find a package link and fail.

pdm/src/pdm/cli/commands/list.py

Lines 144 to 146 in cd2f1a9

resolved_set = set(

c.prepare(project.environment).metadata for c in candidates.values()

)

tests/cli/test_list.py

frostming · 2022-10-28T02:11:23Z

@JohnHardy Thanks for the contribution, it may still need to be refined. I will merge it first and continue the work on another branch.

JohnHardy · 2022-10-28T08:06:22Z

Fantastic, thank you for making last refinements.

I look forward to seeing the changes - please ping me if I can help in the future👍

JohnHardy and others added 3 commits October 13, 2022 12:02

-Draft licenses CLI feature

d03604d

fix field name

0a8fc56

[pre-commit.ci] auto fixes from pre-commit.com hooks

3a51e1c

for more information, see https://pre-commit.ci

JohnHardy mentioned this pull request Oct 13, 2022

Can I exclude dev dependencies? raimon49/pip-licenses#124

Closed

frostming added RFC ⭐ enhancement Improvements for existing features labels Oct 14, 2022

JohnHardy and others added 7 commits October 15, 2022 01:20

Revised feature set as per discussion in #1433

7a11806

Merge branch 'licenses' of https://github.com/JohnHardy/pdm into lice…

01ffcef

…nses

Removed licenses preview feature

bc4090e

[pre-commit.ci] auto fixes from pre-commit.com hooks

cf05eb4

for more information, see https://pre-commit.ci

--resolve applies to graph

44440b0

merged in with black

aabc692

[pre-commit.ci] auto fixes from pre-commit.com hooks

9c5d132

for more information, see https://pre-commit.ci

JohnHardy and others added 3 commits October 15, 2022 17:23

special label

31d1f8f

removed stray comments and changed to :sub label

8fba529

[pre-commit.ci] auto fixes from pre-commit.com hooks

7c1c1bb

for more information, see https://pre-commit.ci

frostming marked this pull request as draft October 17, 2022 01:42

JohnHardy and others added 5 commits October 17, 2022 23:45

Refactored and added tests

424b4d9

[pre-commit.ci] auto fixes from pre-commit.com hooks

28f64df

for more information, see https://pre-commit.ci

a few static check fixes

3498b32

[pre-commit.ci] auto fixes from pre-commit.com hooks

f070869

for more information, see https://pre-commit.ci

Added more tests and fixed linting

28ab67b

frostming marked this pull request as ready for review October 21, 2022 00:20

JohnHardy added 11 commits October 21, 2022 21:14

Changed from do_list to do_lock but reverted exception checking

a2c338c

Added --freeze and tests

901c4c8

Reverted

98d5b72

Fixed spelling in argparse commands

89d144f

Graph output tests using ascii

7eadb43

Removed todo comment

62cfa28

Black formatting

ebc5cdc

Merge branch 'main' of https://github.com/pdm-project/pdm into licenses

cd2f1a9

test_list_csv_include_exclude updated

12a76ed

test_list_csv_include_exclude updated

cbe3478

added note for expected version

bae60cb

JohnHardy commented Oct 21, 2022

View reviewed changes

tests/conftest.py Show resolved Hide resolved

JohnHardy commented Oct 21, 2022

View reviewed changes

tests/cli/test_list.py Outdated Show resolved Hide resolved

frostming reviewed Oct 24, 2022

View reviewed changes

tests/cli/test_list.py Show resolved Hide resolved

JohnHardy added 7 commits October 26, 2022 21:52

Merge branch 'main' of https://github.com/pdm-project/pdm into licenses

f2ade39

Minor updates

ccfeb0c

Merge branch 'main' of https://github.com/pdm-project/pdm into licenses

56f6850

Fixed selection and hopefully list tests on posix

a21f19f

Typing fix on parse_comma_separated_string

6f34939

Typing fix on parse_comma_separated_string

6cf6e0a

Typing fix on parse_comma_separated_string

b1815a1

frostming changed the base branch from main to feat/licenses October 28, 2022 02:10

frostming approved these changes Oct 28, 2022

View reviewed changes

frostming merged commit 1c9286a into pdm-project:feat/licenses Oct 28, 2022

JohnHardy deleted the licenses branch October 28, 2022 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft licenses CLI feature #1433

Draft licenses CLI feature #1433

JohnHardy commented Oct 13, 2022 •

edited

Loading

codecov-commenter commented Oct 13, 2022 •

edited

Loading

frostming commented Oct 13, 2022 •

edited

Loading

JohnHardy commented Oct 13, 2022 •

edited

Loading

frostming commented Oct 14, 2022 •

edited

Loading

JohnHardy commented Oct 15, 2022

frostming commented Oct 15, 2022

JohnHardy commented Oct 15, 2022 •

edited

Loading

frostming commented Oct 17, 2022 •

edited

Loading

JohnHardy commented Oct 21, 2022

frostming Oct 24, 2022 •

edited

Loading

frostming commented Oct 28, 2022

JohnHardy commented Oct 28, 2022

	resolved_set = set(
	c.prepare(project.environment).metadata for c in candidates.values()
	)

Draft licenses CLI feature #1433

Draft licenses CLI feature #1433

Conversation

JohnHardy commented Oct 13, 2022 • edited Loading

Pull Request Check List

Describe what you have changed in this PR.

codecov-commenter commented Oct 13, 2022 • edited Loading

Codecov Report

frostming commented Oct 13, 2022 • edited Loading

JohnHardy commented Oct 13, 2022 • edited Loading

frostming commented Oct 14, 2022 • edited Loading

Footnotes

JohnHardy commented Oct 15, 2022

frostming commented Oct 15, 2022

JohnHardy commented Oct 15, 2022 • edited Loading

frostming commented Oct 17, 2022 • edited Loading

JohnHardy commented Oct 21, 2022

frostming Oct 24, 2022 • edited Loading

Choose a reason for hiding this comment

frostming commented Oct 28, 2022

JohnHardy commented Oct 28, 2022

JohnHardy commented Oct 13, 2022 •

edited

Loading

codecov-commenter commented Oct 13, 2022 •

edited

Loading

frostming commented Oct 13, 2022 •

edited

Loading

JohnHardy commented Oct 13, 2022 •

edited

Loading

frostming commented Oct 14, 2022 •

edited

Loading

JohnHardy commented Oct 15, 2022 •

edited

Loading

frostming commented Oct 17, 2022 •

edited

Loading

frostming Oct 24, 2022 •

edited

Loading