-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create an installation package for conda package manager #120
Comments
Hi @gvyshnya ! Thank you for your feedback! Anaconda actually was our first guess when we were developing installers for dvc(you can actually see traces of it in git log), but considering that dvc is currently more of a standalone utility, we actually opted in favor of pyinstaller to create a standalone binary for dvc and distribute it in usual packages(rpm,dev,exe), and pip to distribute it as a python package. That being said, we actually were thinking of creating anaconda/miniconda package in the future, when dvc will be more fit to be used as a library. We can now see that there is a clear demand for it and will try to deliver it in the near future. |
looking forward to conda support! |
Fixed 79d7100 the issue with download_url/url fields in our package info that didn't allow me to use |
Creating a conda package for dvc requires creating packages for all dependencies, as meta.yaml doesn't support pip dependencies for conda packages, only for environments. Thus making creating conda package for dvc time-consuming and tedious. If anyone from the community feels like working on it, please feel free to do so. For now, considering that we provide (among others) a pip package, which can be specified in conda env as a dependency, I don't see a real need in creating conda package right now and might revisit this issue in releases after 0.9.7. |
Closing as stale. Please feel free to reopen if you feel like working on this. |
Conda seems to have better support for creating identical and consistent environments on different platforms. For example, my development env is OSX (my laptop) but production is Ubuntu linux. I need to make sure that there are no differences in the packages installed on the two environments and that I am able to easily spin up a new machine with the same packages... |
I agree with @yfarjoun. There are a few reasons why it would be really nice to have a recipe for dvc in one of the main conda channels:
|
Thanks @efiop. This is essentially the strategy I'm using, but it's a bit more complicated than that. What that section actually ends up looking like is more like this:
... because without pinning the versions of all the dependencies, it's hard to guarantee reproducibility. Currently this is working because where |
@efiop just curious, is anyone actively working on this issue? If not, it seems like something I wouldn't mind working on over the next week. |
@J0 That would be amazing! 🙂 No, no one is working on it right now. Thank you so much for looking into this! |
FYI, the outstanding DVC dependencies that do not have a conda build are:
For DVC to provide a conda build, I believe the above packages will also need a conda build. See contributing packages guidelines on conda-forge. The process for porting a PyPi package to conda-forge is becoming increasingly streamlined but still not a trivial task. I would like to see DVC on conda but currently do not have the time to assist on this issue. |
Started to work on this. A basic About dependencies:
I guess it is better to put them in the same PR with the DVC. Btw, @brbarkley could you please share how did you get the list of outstanding dependencies? |
that guy seems to have packaged everything that is needed. Including grandalf https://anaconda.org/derickl/grandalf . |
@efiop , those are outside conda-forge (don't know if this is like the official distribution or something) |
Hi @derickl ! We've found your conda package for dvc and we were wondering if you would be willing to contribute your scripts to create an official dvc repo, that we could help maintaining and keeping up-to-date? |
Thanks to all of you working on this. It world be awesome to have a Conda dvc package, as I mainly use conda as package manager. However, I prefer if it is possible to have the dvc package in the main or conda-force channel. |
Help is needed on this, right? Whom can I discuss that with? |
I'm happy to talk as a user. |
@GildedHonour we actually have a guy who is looking into this right now. Are you interested in helping us for this specific task or just want to be involved and help DVC in general? Would be happy to discuss and find more stuff where we need more hands :) |
@shcheklein in general too. Yes, let's discuss. |
@GildedHonour Alex, can you find me and/or Ruslan on |
@shcheklein just done |
conda-forge/staged-recipes#8963 was merged. Dvc should be available throug conda-forge now https://github.com/conda-forge/dvc-feedstock , unless I'm missing something. Big thanks to @Maxris 🎉 |
@efiop unfortunately, dvc package will be uploaded to conda-forge channel once we will have 1st successful ci build in feedstock repo's master https://github.com/conda-forge/dvc-feedstock/commits/master ( so far the build was failed because of others not yet uploaded dependencies ). Another important thing is that only Python 2.7 and 3.6 builds are enabled for dvc feedstock. To enable Python 3.7 builds it will be needed to remove restriction from there https://github.com/conda-forge/dvc-feedstock/blob/master/recipe/meta.yaml#L14 , but before we can do that it's required to bring Python 3.7 based builds for all DVC's dependencies. |
@Maxris Thanks for the clarification! Let's keep this open for now then. |
DVC 0.53.2 for Python 2.7 and 3.6 is available through conda-forge now!
|
Python 3.7 build of dvc is available now! Odd thing is that on Windows 10 I'm receiving following error when trying to run installed dvc from
Will try to investigate this more. |
Finally, dvc 0.54.1 build 1 with all extra deps is available in conda-forge |
@Maxris awesome stuff! Thanks. The only thing is the doc on how do we support/update it in the future before we close this ticket (finally). |
k, thanks, @Maxris, we have all the docs ready now - https://github.com/iterative/dvc/wiki/Maintenance-of-Anaconda-package-in-conda-forge-channel @efiop please, take a look and let's update our release check list to include a step to upgrade requirements is necessary. I think we are ready to close this issue at last 🎉 |
@shcheklein Added a quick one https://github.com/iterative/dvc/wiki/Release-checklist |
thanks @efiop 🙏 :) |
Anaconda (https://www.continuum.io/what-is-anaconda) is the leading Python distribution for data science today. It has its internal package manager - conda (https://conda.io/docs/index.html), which is a rival to a well-known pip.
Since Anaconda as well as its python-only lightweight version of Miniconda (https://conda.io/miniconda.html) are getting more and more tracking within Data Science community these days, porting DVC installer to conda may become a good step to streamline DVC usage across industrial analytical circles.
The text was updated successfully, but these errors were encountered: