Support non-Hackage dependencies #2189

Open
23Skidoo opened this Issue Oct 30, 2014 · 19 comments

Projects

None yet

9 participants

@23Skidoo
Member
23Skidoo commented Oct 30, 2014 edited

Summary by @ezyang:

  • Syntactic support for these URLs exists: in the packages field you can specify remote tarballs or local tarballs, and using the source-repository syntax (from Cabal file) you can point to a VCS URL (this parser should also have been reused for cabal.project)
  • Support for downloading from various places exists. There's support in the old-build path for local/remote tarballs. cabal get -s supports getting things from source repositories.
  • However, these are not used for anything: the internals need to be plumbed up to use this. Look at old stuff and determine how to port it over.

This should be doable in a day.


This came up on Twitter:

https://twitter.com/darinmorrison/status/527716059889954816

Basically, people want to publish packages that depend on stuff that is not on Hackage. One solution is to upload a forked package (example), but this has its own downsides (Hackage namespace becomes polluted). Implementing something like build-depends: https://repo.git@revision would be a nice alternative. Most other package managers (e.g. sbt) support this.

It should also be possible to prohibit downloading packages not on Hackage via a config file option.

See also #1534.

@23Skidoo 23Skidoo self-assigned this Oct 31, 2014
@mietek
Contributor
mietek commented Nov 11, 2014

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

@ttuegel ttuegel added this to the _|_ milestone Apr 24, 2015
@winterland1989

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

@ezyang
Contributor
ezyang commented Jul 11, 2016

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

@winterland1989

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

@23Skidoo
Member

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

@phadej
Collaborator
phadej commented Jul 13, 2016 edited

Looks like we only need to teach cabal-install how to fetch stuff from the internet: https://github.com/haskell/cabal/blob/5423d7750a23e25880097801f3be752fa21f9f32/cabal-install/Distribution/Client/ProjectConfig.hs#L745

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

@phadej
Collaborator
phadej commented Jul 13, 2016 edited

(For now one can use git submodule, effectively the result should be the same)

@winterland1989

FYI, every github commit can be downloaded as a zip file, e.g.

Yes, but that's github's convention, we probably shouldn't rely on that convention. In fact we can safely assume user have git binary installed if they want to use git depends, and we can use process package manage git clone/pull.

@alanz
Collaborator
alanz commented Jul 13, 2016

A use case for this is during development when another client component has not yet been released to hackage. E.g. ghc-mod for HaRe. This is a temporary state.

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

@alanz
Collaborator
alanz commented Jul 13, 2016

And there is a general-purpose variant too

https://git-scm.com/docs/git-archive

This supports a --remote=XXX flag

@phadej
Collaborator
phadej commented Jul 13, 2016

@winterland1989 I hope we can interpret @alanz comment, as so even support for remote tarballs would make him a lot happier :)

@winterland1989
winterland1989 commented Jul 13, 2016 edited

Yes, support for remote tarballs is enough for me too ; )

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

This is also my main motivation for this feature requesting, manually manage forked source is frustrating.

@23Skidoo 23Skidoo removed their assignment Jul 27, 2016
@ezyang
Contributor
ezyang commented Aug 28, 2016

If you do this for nix-local-build, a determination of whether or not such external dependencies should be local or external needs to be made. If they're treated as local (which I believe is what the current code will do if you list it in packages or similar fields), then they are not eligible to be cached and reused in the external store. That's probably not desirable.

@dcoutts
Member
dcoutts commented Sep 8, 2016

In principle if we can get a reliable source hash (and guarantee this isn't changed locally) then we can use the store approach. So for git we can use the git hash, but we'd have to avoid letting users fiddle with the sources locally in that case. If we allow fiddling then it has to be treated like a local dir.

@23Skidoo
Member
23Skidoo commented Sep 8, 2016

If we allow fiddling then it has to be treated like a local dir.

We can ask git if the source has been fiddled with and then fall back to treating it like a local dir if it is the case.

@ezyang ezyang modified the milestone: Default nix-local-build, Sep 8, 2016
@ezyang ezyang added the priority: low label Sep 8, 2016
@ezyang
Contributor
ezyang commented Sep 8, 2016

I'm inclined to just not allow specification of a hash if you don't specify a remote URL. Don't really want new-build to be checking out git branches or ignoring my hash...

@bitemyapp
Contributor

I'm looking forward to celebrating this issue's second 🎂 in two days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment