Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"ls dependencies" should have an option to maintain the tree structure #4101

Closed
sschuberth opened this issue Jun 20, 2018 · 22 comments · Fixed by #4424
Closed

"ls dependencies" should have an option to maintain the tree structure #4101

sschuberth opened this issue Jun 20, 2018 · 22 comments · Fixed by #4424

Comments

@sschuberth
Copy link

Currently, "ls dependencies" just outputs a flat list of package names and versions of all transitive dependencies of a project. However, it would be nice to maintain the tree hierarchy of dependencies to see their parent / child relationship. On the console, the tree output could look similar to Maven's mvn dependency:tree output, but ideally there would also be an option to output to a machine readable format like JSON.

@dbaynard
Copy link
Contributor

Hello @sschuberth . That sounds like a helpful enhancement. How much of what you would like to do can you achieve with stack dot --external (perhaps with --include-base, too)?

@sschuberth
Copy link
Author

sschuberth commented Jun 21, 2018

I was thinking about (mis-)using stack dot, too, but while it basically provides the parent -> child relationship, it's a bit inconvenient to parse, also because it repeats the parent for each child. The versions are missing, too, but together with the output from stack ls dependencies I could probably get what I want. I'll give it a try.

@dbaynard
Copy link
Contributor

That's all useful to know. It may be relatively straightforward to reuse the code required for the dot output. Would you be willing to contribute a PR yourself?

@sschuberth
Copy link
Author

I'm sorry, I'm not a Haskell guy (and I don't intend to become one 😉), I'm just trying to add basic Haskell support to the OSS Review Toolkit.

@dbaynard
Copy link
Contributor

Oh, that's an interesting project. Is there anything else that would be helpful for you to achieve that, from stack?

It might also be possible for you to do what you intend using the hackage or stackage APIs directly (depending on scale), though they are quite limited. If stack meets your other requirements, then great.

@sschuberth
Copy link
Author

Is there anything else that would be helpful for you to achieve that, from stack?

Thanks for asking. Besides listing dependencies as a tree (optionally in a machine readable format), it would be nice if the shown meta-data would not only include the name and version, but also the dependency's declared license (as an SPDX identifier). That would be the very minimum, IMO. More meta-data like the download URL would also be interesting and could be added, or simply a link to the respective hackage or stackage (REST?) APIs endpoints where more information can be gathered by the client (the program using the meta-data output) on demand.

In general, from a performance perspective it would be nice if all this meta-data could be gathered without internally downloading and unpacking the whole (source code of a) package, i.e. ideally meta-data for a package and the package itself should be separated, similar to Maven POM and JAR files.

using the hackage or stackage APIs directly

While I've found http://hackage.haskell.org/api, I didn't find something similar for stackage. Is there? Edit: I just found https://groups.google.com/forum/#!topic/stackage/9h_RA-jNz_g.

@dbaynard
Copy link
Contributor

dbaynard commented Jun 22, 2018

Perhaps we should broaden the issue, then. It seems like a useful addition, though it may not be a priority for the core team.

So to summarize,

  • List dependencies as tree

  • List dependencies in machine readable format

  • List license as SPDX in addition to name and version

    Most cabal files don't use these identifiers; internally Cabal (which processes these files and is different to cabal-install, the provider of the command line program cabal) uses an enumeration, though, so you may wish to take a look thee.

  • List URI too

In general, from a performance perspective it would be nice if all this meta-data could be gathered without internally downloading and unpacking the whole (source code of a) package, i.e. ideally meta-data for a package and the package itself should be separated, similar to Maven POM and JAR files.

stack downloads and unpacks the hackage tarball (~100MB compressed), which contains the cabal files where the metadata is located; this might update incrementally (I've never investigated). The cabal files themselves are reachable directly (/package/:package/:cabal.cabal). So the trade-off is a space vs latency one. If you're happy with the space use, stack would be useful for this.

I didn't find something similar for stackage

I think I might have just tried the hackage endpoints when I was looking for snapshot information… or I may have checked the api calls in the debugger. I think the hackage api is most relevant, though — I only needed stackage because of stack snapshots. That said, it may be useful to add the hackage mirrors, if you go that way.

@dbaynard
Copy link
Contributor

@borsboom — it may be worth supporting this enhancement to support the OSS review toolkit project mentioned earlier.

@borsboom
Copy link
Contributor

I'm very much in favour. I've often wished for an ls-dependencies that shows the tree structure. That said, I'm going to un-assign myself as it's very unlikely I'll actually be able to work on implementing it.

@borsboom borsboom removed their assignment Jun 25, 2018
@bitemyapp
Copy link
Contributor

bitemyapp commented Jul 3, 2018

$ cargo tree
lefortovo v0.1.0 (file:///home/callen/work/lefortovo)
├── bytes v0.4.8
│   ├── byteorder v1.0.0
│   └── iovec v0.1.2
│       └── libc v0.2.42
├── clap v2.25.0
│   ├── ansi_term v0.9.0
│   ├── atty v0.2.2
│   │   └── libc v0.2.42 (*)
│   ├── bitflags v0.9.1
│   ├── strsim v0.6.0
│   ├── term_size v0.3.0
│   │   └── libc v0.2.42 (*)
│   ├── textwrap v0.6.0
│   │   ├── term_size v0.3.0 (*)
│   │   └── unicode-width v0.1.4
│   ├── unicode-segmentation v1.1.0
│   ├── unicode-width v0.1.4 (*)
│   └── vec_map v0.8.0
├── hyper v0.12.5
│   ├── bytes v0.4.8 (*)
│   ├── futures v0.1.21
│   ├── futures-cpupool v0.1.8
│   │   ├── futures v0.1.21 (*)
│   │   └── num_cpus v1.6.2
│   │       └── libc v0.2.42 (*)
│   ├── h2 v0.1.10
│   │   ├── byteorder v1.0.0 (*)
│   │   ├── bytes v0.4.8 (*)
│   │   ├── fnv v1.0.6
│   │   ├── futures v0.1.21 (*)
│   │   ├── http v0.1.7
│   │   │   ├── bytes v0.4.8 (*)
│   │   │   ├── fnv v1.0.6 (*)
│   │   │   └── itoa v0.4.1
│   │   ├── indexmap v1.0.1
│   │   ├── log v0.4.3
│   │   │   └── cfg-if v0.1.4
│   │   ├── slab v0.4.0
│   │   ├── string v0.1.0
│   │   └── tokio-io v0.1.7
│   │       ├── bytes v0.4.8 (*)
│   │       ├── futures v0.1.21 (*)
│   │       └── log v0.4.3 (*)
│   ├── http v0.1.7 (*)
│   ├── httparse v1.2.3
│   ├── iovec v0.1.2 (*)
│   ├── itoa v0.4.1 (*)
│   ├── log v0.4.3 (*)
│   ├── net2 v0.2.33
│   │   ├── cfg-if v0.1.4 (*)
│   │   └── libc v0.2.42 (*)
│   ├── time v0.1.37
│   │   └── libc v0.2.42 (*)
│   ├── tokio v0.1.7
│   │   ├── futures v0.1.21 (*)
│   │   ├── mio v0.6.14
│   │   │   ├── iovec v0.1.2 (*)
│   │   │   ├── lazycell v0.6.0
│   │   │   ├── libc v0.2.42 (*)
│   │   │   ├── log v0.4.3 (*)
│   │   │   ├── net2 v0.2.33 (*)
│   │   │   └── slab v0.4.0 (*)
│   │   ├── tokio-executor v0.1.2
│   │   │   └── futures v0.1.21 (*)
│   │   ├── tokio-fs v0.1.1
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── tokio-io v0.1.7 (*)
│   │   │   └── tokio-threadpool v0.1.4
│   │   │       ├── crossbeam-deque v0.3.1
│   │   │       │   ├── crossbeam-epoch v0.4.3
│   │   │       │   │   ├── arrayvec v0.4.7
│   │   │       │   │   │   └── nodrop v0.1.12
│   │   │       │   │   ├── cfg-if v0.1.4 (*)
│   │   │       │   │   ├── crossbeam-utils v0.3.2
│   │   │       │   │   │   └── cfg-if v0.1.4 (*)
│   │   │       │   │   ├── lazy_static v1.0.1
│   │   │       │   │   ├── memoffset v0.2.1
│   │   │       │   │   └── scopeguard v0.3.3
│   │   │       │   └── crossbeam-utils v0.3.2 (*)
│   │   │       ├── futures v0.1.21 (*)
│   │   │       ├── log v0.4.3 (*)
│   │   │       ├── num_cpus v1.6.2 (*)
│   │   │       ├── rand v0.4.2
│   │   │       │   └── libc v0.2.42 (*)
│   │   │       └── tokio-executor v0.1.2 (*)
│   │   │   [dev-dependencies]
│   │   │   └── tokio-io v0.1.7 (*)
│   │   ├── tokio-io v0.1.7 (*)
│   │   ├── tokio-reactor v0.1.2
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── log v0.4.3 (*)
│   │   │   ├── mio v0.6.14 (*)
│   │   │   ├── slab v0.4.0 (*)
│   │   │   ├── tokio-executor v0.1.2 (*)
│   │   │   └── tokio-io v0.1.7 (*)
│   │   ├── tokio-tcp v0.1.0
│   │   │   ├── bytes v0.4.8 (*)
│   │   │   ├── futures v0.1.21 (*)
│   │   │   ├── iovec v0.1.2 (*)
│   │   │   ├── mio v0.6.14 (*)
│   │   │   ├── tokio-io v0.1.7 (*)
│   │   │   └── tokio-reactor v0.1.2 (*)
│   │   ├── tokio-threadpool v0.1.4 (*)
│   │   ├── tokio-timer v0.2.4
│   │   │   ├── futures v0.1.21 (*)
│   │   │   └── tokio-executor v0.1.2 (*)
│   │   └── tokio-udp v0.1.1
│   │       ├── bytes v0.4.8 (*)
│   │       ├── futures v0.1.21 (*)
│   │       ├── log v0.4.3 (*)
│   │       ├── mio v0.6.14 (*)
│   │       ├── tokio-codec v0.1.0
│   │       │   ├── bytes v0.4.8 (*)
│   │       │   ├── futures v0.1.21 (*)
│   │       │   └── tokio-io v0.1.7 (*)
│   │       ├── tokio-io v0.1.7 (*)
│   │       └── tokio-reactor v0.1.2 (*)
│   ├── tokio-executor v0.1.2 (*)
│   ├── tokio-io v0.1.7 (*)
│   ├── tokio-reactor v0.1.2 (*)
│   ├── tokio-tcp v0.1.0 (*)
│   ├── tokio-timer v0.2.4 (*)
│   └── want v0.0.5
│       ├── futures v0.1.21 (*)
│       ├── log v0.4.3 (*)
│       └── try-lock v0.2.2
├── hyper-tls v0.2.1
│   ├── bytes v0.4.8 (*)
│   ├── futures v0.1.21 (*)
│   ├── hyper v0.12.5 (*)
│   ├── native-tls v0.1.4
│   │   └── openssl v0.9.14
│   │       ├── bitflags v0.9.1 (*)
│   │       ├── foreign-types v0.2.0
│   │       ├── lazy_static v0.2.8
│   │       ├── libc v0.2.42 (*)
│   │       └── openssl-sys v0.9.14
│   │           └── libc v0.2.42 (*)
│   │           [build-dependencies]
│   │           ├── gcc v0.3.51
│   │           └── pkg-config v0.3.9
│   ├── tokio-io v0.1.7 (*)
│   └── tokio-tls v0.1.4
│       ├── futures v0.1.21 (*)
│       ├── native-tls v0.1.4 (*)
│       ├── tokio-core v0.1.17
│       │   ├── bytes v0.4.8 (*)
│       │   ├── futures v0.1.21 (*)
│       │   ├── iovec v0.1.2 (*)
│       │   ├── log v0.4.3 (*)
│       │   ├── mio v0.6.14 (*)
│       │   ├── scoped-tls v0.1.0
│       │   ├── tokio v0.1.7 (*)
│       │   ├── tokio-executor v0.1.2 (*)
│       │   ├── tokio-io v0.1.7 (*)
│       │   ├── tokio-reactor v0.1.2 (*)
│       │   └── tokio-timer v0.2.4 (*)
│       └── tokio-io v0.1.7 (*)
├── pretty_env_logger v0.2.3
│   ├── ansi_term v0.11.0
│   ├── env_logger v0.5.10
│   │   ├── atty v0.2.2 (*)
│   │   ├── humantime v1.1.1
│   │   │   └── quick-error v1.2.2
│   │   ├── log v0.4.3 (*)
│   │   ├── regex v1.0.1
│   │   │   ├── aho-corasick v0.6.5
│   │   │   │   └── memchr v2.0.1
│   │   │   │       └── libc v0.2.42 (*)
│   │   │   ├── memchr v2.0.1 (*)
│   │   │   ├── regex-syntax v0.6.1
│   │   │   │   └── ucd-util v0.1.1
│   │   │   ├── thread_local v0.3.5
│   │   │   │   ├── lazy_static v1.0.1 (*)
│   │   │   │   └── unreachable v1.0.0
│   │   │   │       └── void v1.0.2
│   │   │   └── utf8-ranges v1.0.0
│   │   └── termcolor v0.3.6
│   └── log v0.4.3 (*)
└── tokio-core v0.1.17 (*)

cargo tree has nice output. Cf. https://github.com/sfackler/cargo-tree

I think stack ls dependencies needs to go unchanged as other tools reuse the output for things like license checking, but a --tree flag or similar would be nice.

@sschuberth
Copy link
Author

And so do Maven (mvn dependency:tree), Gradle (gradle dependencies), and several other package managers / dependency tools / build systems / whatever you call them...

@creichert
Copy link

creichert commented Jul 9, 2018

NPM does as well:

$ npm ls | head
project@0.1.0 /home/christopher/dev/proj
├── adt@0.7.2
├─┬ babel-core@6.26.0
│ ├─┬ babel-code-frame@6.26.0
│ │ ├─┬ chalk@1.1.3
│ │ │ ├── ansi-styles@2.2.1
│ │ │ ├── escape-string-regexp@1.0.5
│ │ │ ├─┬ has-ansi@2.0.0
│ │ │ │ └── ansi-regex@2.1.1
│ │ │ ├─┬ strip-ansi@3.0.1

@sschuberth I'm sure you've seen stack ls dependencies --license? It's not SPDX format afaik, but may still be a useful interim solution.

$ stack ls dependencies --separator=- --license | head
Cabal-BSD3
HTTP-BSD3
HUnit-BSD3
HsOpenSSL-PublicDomain
JuicyPixels-BSD3
MonadRandom-OtherLicense
QuickCheck-BSD3
RSA-BSD3
SHA-BSD3
StateVar-BSD3

@borsboom Why does --license discard the version number?

@sschuberth
Copy link
Author

The problem with NPM is that it can only show the tree for installed dependencies. But ideally I'd like to see the list / tree of dependencies before I even install them. Maven and Gradle can do that just fine.

Thanks @creichert, I've seen stack ls dependencies --license, but I've already implemented a solution that uses a mix of calling stack and parsing *.cabal files directly from Hackage.

@akshaymankar
Copy link
Contributor

I was exploring the newcomer friendly tag and came across this. It looks interesting and also seem like nobody is try to implement this feature. I am going to take a dig at implementing this, please give me a shout if somebody is already doing it.

akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 4, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 21, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Oct 22, 2018
dbaynard added a commit that referenced this issue Nov 16, 2018
 Add option to print dependencies as tree

First part of #4101
@dbaynard
Copy link
Contributor

@akshaymankar For machine readable output I suggest json (unless @sschuberth has a better suggestion). It might be simple enough to implement some aeson instances and then plumb them in.

SPDX is well defined.

For the URI, I'm not sure what to go with. I don't really understand the use case, so perhaps @sschuberth can explain.

@dbaynard dbaynard reopened this Nov 19, 2018
@sschuberth
Copy link
Author

JSON is indeed fine from my perspective. But if e.g. more common in the Haskell community, YAML could be another option.

URI / URL was referring to the direct download location for the source code to the package. At the example of the memory package version 0.14.11, the URL would be https://hackage.haskell.org/package/memory-0.14.11/memory-0.14.11.tar.gz. The use-case is to perform automated license checks by scanning the source code of the package.

akshaymankar added a commit to akshaymankar/stack that referenced this issue Nov 30, 2018
akshaymankar added a commit to akshaymankar/stack that referenced this issue Dec 6, 2018
This ensures interface of ls dependencies remains clean and users can
only pass options when they make sense

[commercialhaskell#4101]
@akshaymankar
Copy link
Contributor

The URL in the JSON doesn't really work, because a package could either be from hackage, a local package, an archive specified by URL or a git repository. So, I guess a URL to a will not always be possible.

I am not sure what the best possible format for this would be. @sschuberth, do you have any suggestions/preferences?

@dbaynard I also couldn't find a place where stack figures out URL from PakcageLocation object. Can you please point me to it, so I don't invent the logic again.

@qrilka
Copy link
Contributor

qrilka commented Dec 13, 2018

@akshaymankar what's the problem to use file:// for local packages and git:// for git?

@qrilka
Copy link
Contributor

qrilka commented Dec 13, 2018

And I think you'll need a custom function for that, I don't think there's already some code doing that.

@sschuberth
Copy link
Author

I was also about to say a URL should work:

  • a package could either be from hackage -> Use a http(s) URL to hackage.haskell.org
  • a local package -> Use a file:// URL
  • an archive specified by URL -> Use a http(s) URL to a non-hackage place
  • a git repository -> Here's the problem. Git supports a variety of transports, and in particular not all Git repos can be actually accessed via the git:// protocol (as that protocol offers no authentication). So either we come up with a convention that says that http(s) URLs that refer to Git repos have to end in .git (which is the common case anyway), or we introduce some additional meta-data field to indicate which type of URL this is. If packages are never going to be hosted in some other VCS than Git, the former approach would probably work fine. But e.g. for Mercurial there is no convention that the http(s) URL to a Mercurial repo should end in .hg, and we'd need the second approach of additional meta-data.

@dbaynard
Copy link
Contributor

@dbaynard I also couldn't find a place where stack figures out URL from PakcageLocation object. Can you please point me to it, so I don't invent the logic again.

And I think you'll need a custom function for that, I don't think there's already some code doing that.

I think @qrilka was answering the question you asked of me.

@akshaymankar
Copy link
Contributor

I think file:// is a good idea for local packages.
For git and hg repositories, even if we go with any URL specification, we still won't be able to show the commit in the URL. So, I'm leaning towards the second approach of additional metadata.

Maybe it could look something like this for git (and similar for hg):

{
  "name": "foo",
  "version": "1.0.0.0",
  "license": "MIT",
  "location": {
    "type": "git",
    "url": "<URL provided by user>",
    "commit": "<sha>"
  }
}

For hackage:

{
  "name": "foo",
  "version": "1.0.0.0",
  "license": "MIT",
  "location": {
    "type": "hackage",
    "url": "<computed hackage URL>"
  }
}

snoyberg added a commit that referenced this issue Mar 19, 2019
With the changes to the dot command, the test currently fails due to not
having the dependencies of the transformers package in the expected
output. this fixes the situation.
snoyberg added a commit that referenced this issue Mar 19, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue May 3, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue May 3, 2019
This ensures interface of ls dependencies remains clean and users can
only pass options when they make sense

[commercialhaskell#4101]
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 15, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 15, 2019
This ensures interface of ls dependencies remains clean and users can
only pass options when they make sense

[commercialhaskell#4101]
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 15, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 15, 2019
And thus remove dependency on `aeson-pretty`
[commercialhaskell#4101]
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 15, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 18, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 18, 2019
This ensures interface of ls dependencies remains clean and users can
only pass options when they make sense

[commercialhaskell#4101]
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 18, 2019
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 18, 2019
And thus remove dependency on `aeson-pretty`
[commercialhaskell#4101]
akshaymankar added a commit to akshaymankar/stack that referenced this issue Jun 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants