New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non-Hackage dependencies #2189

Closed
23Skidoo opened this Issue Oct 30, 2014 · 26 comments

Comments

@23Skidoo
Member

23Skidoo commented Oct 30, 2014

Summary by @ezyang:

  • Syntactic support for these URLs exists: in the packages field you can specify remote tarballs or local tarballs, and using the source-repository syntax (from Cabal file) you can point to a VCS URL (this parser should also have been reused for cabal.project)
  • Support for downloading from various places exists. There's support in the old-build path for local/remote tarballs. cabal get -s supports getting things from source repositories.
  • However, these are not used for anything: the internals need to be plumbed up to use this. Look at old stuff and determine how to port it over.

This should be doable in a day.


This came up on Twitter:

https://twitter.com/darinmorrison/status/527716059889954816

Basically, people want to publish packages that depend on stuff that is not on Hackage. One solution is to upload a forked package (example), but this has its own downsides (Hackage namespace becomes polluted). Implementing something like build-depends: https://repo.git@revision would be a nice alternative. Most other package managers (e.g. sbt) support this.

It should also be possible to prohibit downloading packages not on Hackage via a config file option.

See also #1534.

@23Skidoo 23Skidoo self-assigned this Oct 31, 2014

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek Nov 11, 2014

Contributor

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

Contributor

mietek commented Nov 11, 2014

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

@winterland1989

This comment has been minimized.

Show comment
Hide comment
@winterland1989

winterland1989 Jul 11, 2016

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

winterland1989 commented Jul 11, 2016

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Jul 11, 2016

Contributor

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

Contributor

ezyang commented Jul 11, 2016

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

@winterland1989

This comment has been minimized.

Show comment
Hide comment
@winterland1989

winterland1989 Jul 11, 2016

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

winterland1989 commented Jul 11, 2016

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Jul 11, 2016

Member

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

Member

23Skidoo commented Jul 11, 2016

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

@alanz

This comment has been minimized.

Show comment
Hide comment
@alanz
Collaborator

alanz commented Jul 11, 2016

@phadej

This comment has been minimized.

Show comment
Hide comment
@phadej

phadej Jul 13, 2016

Collaborator

Looks like we only need to teach cabal-install how to fetch stuff from the internet:

fail $ "TODO: add support for fetching and reading local tarballs, remote "

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

Collaborator

phadej commented Jul 13, 2016

Looks like we only need to teach cabal-install how to fetch stuff from the internet:

fail $ "TODO: add support for fetching and reading local tarballs, remote "

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

@phadej

This comment has been minimized.

Show comment
Hide comment
@phadej

phadej Jul 13, 2016

Collaborator

(For now one can use git submodule, effectively the result should be the same)

Collaborator

phadej commented Jul 13, 2016

(For now one can use git submodule, effectively the result should be the same)

@alanz

This comment has been minimized.

Show comment
Hide comment
@alanz
Collaborator

alanz commented Jul 13, 2016

@winterland1989

This comment has been minimized.

Show comment
Hide comment
@winterland1989

winterland1989 Jul 13, 2016

FYI, every github commit can be downloaded as a zip file, e.g.

Yes, but that's github's convention, we probably shouldn't rely on that convention. In fact we can safely assume user have git binary installed if they want to use git depends, and we can use process package manage git clone/pull.

winterland1989 commented Jul 13, 2016

FYI, every github commit can be downloaded as a zip file, e.g.

Yes, but that's github's convention, we probably shouldn't rely on that convention. In fact we can safely assume user have git binary installed if they want to use git depends, and we can use process package manage git clone/pull.

@alanz

This comment has been minimized.

Show comment
Hide comment
@alanz

alanz Jul 13, 2016

Collaborator

A use case for this is during development when another client component has not yet been released to hackage. E.g. ghc-mod for HaRe. This is a temporary state.

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

Collaborator

alanz commented Jul 13, 2016

A use case for this is during development when another client component has not yet been released to hackage. E.g. ghc-mod for HaRe. This is a temporary state.

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

@alanz

This comment has been minimized.

Show comment
Hide comment
@alanz

alanz Jul 13, 2016

Collaborator

And there is a general-purpose variant too

https://git-scm.com/docs/git-archive

This supports a --remote=XXX flag

Collaborator

alanz commented Jul 13, 2016

And there is a general-purpose variant too

https://git-scm.com/docs/git-archive

This supports a --remote=XXX flag

@phadej

This comment has been minimized.

Show comment
Hide comment
@phadej

phadej Jul 13, 2016

Collaborator

@winterland1989 I hope we can interpret @alanz comment, as so even support for remote tarballs would make him a lot happier :)

Collaborator

phadej commented Jul 13, 2016

@winterland1989 I hope we can interpret @alanz comment, as so even support for remote tarballs would make him a lot happier :)

@winterland1989

This comment has been minimized.

Show comment
Hide comment
@winterland1989

winterland1989 Jul 13, 2016

Yes, support for remote tarballs is enough for me too ; )

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

This is also my main motivation for this feature requesting, manually manage forked source is frustrating.

winterland1989 commented Jul 13, 2016

Yes, support for remote tarballs is enough for me too ; )

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

This is also my main motivation for this feature requesting, manually manage forked source is frustrating.

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Aug 28, 2016

Contributor

If you do this for nix-local-build, a determination of whether or not such external dependencies should be local or external needs to be made. If they're treated as local (which I believe is what the current code will do if you list it in packages or similar fields), then they are not eligible to be cached and reused in the external store. That's probably not desirable.

Contributor

ezyang commented Aug 28, 2016

If you do this for nix-local-build, a determination of whether or not such external dependencies should be local or external needs to be made. If they're treated as local (which I believe is what the current code will do if you list it in packages or similar fields), then they are not eligible to be cached and reused in the external store. That's probably not desirable.

@dcoutts

This comment has been minimized.

Show comment
Hide comment
@dcoutts

dcoutts Sep 8, 2016

Member

In principle if we can get a reliable source hash (and guarantee this isn't changed locally) then we can use the store approach. So for git we can use the git hash, but we'd have to avoid letting users fiddle with the sources locally in that case. If we allow fiddling then it has to be treated like a local dir.

Member

dcoutts commented Sep 8, 2016

In principle if we can get a reliable source hash (and guarantee this isn't changed locally) then we can use the store approach. So for git we can use the git hash, but we'd have to avoid letting users fiddle with the sources locally in that case. If we allow fiddling then it has to be treated like a local dir.

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Sep 8, 2016

Member

If we allow fiddling then it has to be treated like a local dir.

We can ask git if the source has been fiddled with and then fall back to treating it like a local dir if it is the case.

Member

23Skidoo commented Sep 8, 2016

If we allow fiddling then it has to be treated like a local dir.

We can ask git if the source has been fiddled with and then fall back to treating it like a local dir if it is the case.

@ezyang ezyang modified the milestones: Default nix-local-build, Sep 8, 2016

@ezyang ezyang added the priority: low label Sep 8, 2016

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Sep 8, 2016

Contributor

I'm inclined to just not allow specification of a hash if you don't specify a remote URL. Don't really want new-build to be checking out git branches or ignoring my hash...

Contributor

ezyang commented Sep 8, 2016

I'm inclined to just not allow specification of a hash if you don't specify a remote URL. Don't really want new-build to be checking out git branches or ignoring my hash...

@bitemyapp

This comment has been minimized.

Show comment
Hide comment
@bitemyapp

bitemyapp Oct 28, 2016

Contributor

I'm looking forward to celebrating this issue's second 🎂 in two days.

Contributor

bitemyapp commented Oct 28, 2016

I'm looking forward to celebrating this issue's second 🎂 in two days.

@bitemyapp

This comment has been minimized.

Show comment
Hide comment
@bitemyapp

bitemyapp Oct 30, 2017

Contributor

screenshot from 2017-10-30 11-14-13
screenshot from 2017-10-30 11-14-21

Happy 🎂!

Contributor

bitemyapp commented Oct 30, 2017

screenshot from 2017-10-30 11-14-13
screenshot from 2017-10-30 11-14-21

Happy 🎂!

@gbaz

This comment has been minimized.

Show comment
Hide comment
@gbaz

gbaz Mar 24, 2018

Collaborator

The provenance-qualified imports proposal (ghc-proposals/ghc-proposals#115) is relevant here.

Collaborator

gbaz commented Mar 24, 2018

The provenance-qualified imports proposal (ghc-proposals/ghc-proposals#115) is relevant here.

@typedrat typedrat referenced this issue May 19, 2018

Closed

Improved VCS support #5333

0 of 4 tasks complete

@typedrat typedrat moved this from ToDo to InProgress in Have new-build become build (GSOC2018) May 22, 2018

@typedrat typedrat moved this from In Progress to To-Do in Have new-build become build (GSOC2018) Jun 12, 2018

@typedrat typedrat moved this from To-Do to In Progress in Have new-build become build (GSOC2018) Jun 30, 2018

@typedrat typedrat moved this from In Progress to Done in Have new-build become build (GSOC2018) Jul 14, 2018

@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Jul 14, 2018

Member

It's a pity @bitemyapp won't get to celebrate schadenfreude this year... ;-)

Member

hvr commented Jul 14, 2018

It's a pity @bitemyapp won't get to celebrate schadenfreude this year... ;-)

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jul 15, 2018

Collaborator

Closed by #5351.

Collaborator

typedrat commented Jul 15, 2018

Closed by #5351.

@typedrat typedrat closed this Jul 15, 2018

@nomeata

This comment has been minimized.

Show comment
Hide comment
@nomeata

nomeata Jul 18, 2018

Contributor

Exciting! I want to play with this

But is this really fixed? https://github.com/haskell/cabal/blob/master/Cabal/doc/nix-local-build.rst on master still says


    3. [STRIKEOUT:They can specify an ``http``, ``https`` or ``file``
       URL, representing the path to a remote tarball to be downloaded
       and built.] (not implemented yet)

and

[STRIKEOUT:There is also a stanza source-repository-package for specifying packages from an external version control.] (Not implemented.)
Contributor

nomeata commented Jul 18, 2018

Exciting! I want to play with this

But is this really fixed? https://github.com/haskell/cabal/blob/master/Cabal/doc/nix-local-build.rst on master still says


    3. [STRIKEOUT:They can specify an ``http``, ``https`` or ``file``
       URL, representing the path to a remote tarball to be downloaded
       and built.] (not implemented yet)

and

[STRIKEOUT:There is also a stanza source-repository-package for specifying packages from an external version control.] (Not implemented.)
@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jul 18, 2018

Collaborator

That's a documentation bug, I believe.

Collaborator

typedrat commented Jul 18, 2018

That's a documentation bug, I believe.

@nomeata

This comment has been minimized.

Show comment
Hide comment
@nomeata

nomeata Jul 18, 2018

Contributor

I will know in a few minutes when cabal has built and I can try the feature :-)

Yes, seems to work. Great!

Contributor

nomeata commented Jul 18, 2018

I will know in a few minutes when cabal has built and I can try the feature :-)

Yes, seems to work. Great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment