-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
deb/rpm: build granular packages #2800
Comments
Hi @Abrosimov-a-a ! Great question! We package it like that for simplicity, as we don't have to deal with packaging all our dependencies into deb and rpm packages. The way we build current deb and rpm packages is fully reflected by https://github.com/iterative/dvc/blob/master/scripts/build_posix.sh . We build binaries by pyinstaller and then package them into deb/rpm using fpm tool. Then we publish them on our s3 bucket using https://github.com/iterative/dvc-s3-repo . Turning these packages into proper python-based, would require packaging all our dependencies into deb/rpm packages first, which is tiresome, but could be simplified a lot by using that fpm tool, which supports converting python packages into deb/rpm. If you feel like it, we would very appreciate you taking a look 🙂 |
Hi @efiop ! What do you think about SCons? |
@Abrosimov-a-a Yep, 5 deps doesn't sound too bad, indeed 🙂
First time I hear about it, tbh. Could you elaborate please? Also worth mentioning that we are about to have a snap package #2778 thanks to @casperdcl 🙏 , but it does seem to suffer from similar limitations, as it runs in an isolated container. |
@efiop SCons is a pythonic replacement for CMake. |
Next questions:
|
Creating missing packages for our dependencies and placing them on our s3 deb/rpm repo should be good enough.
I guess we need to package each dependency, to set this up correctly, right? |
Yes. It's more preferrable. |
@efiop @shcheklein this is one reason people create |
@Abrosimov-a-a Please let us know if you have any questions or need any help 🙂 @casperdcl Good point! |
I make the roadmap and current status for all dependencies. Please, let me know if I forgot any dependency. |
@Abrosimov-a-a Please create your own fork and use it to create a PR into the upstream, this is the preferred workflow for us. Let us know if you need any help, thank you so much for looking into that! 🙏 |
At this point, all dependencies are builded, except asciimatics. |
@Abrosimov-a-a Great news! Luckily we are getting rid of asciimatics in #2815 , so better just skip it for now :) |
@efiop Now I use Docker as building environment. I think it's a good idea for building many packages with many temporarily dependencies. It's keps the system clean. Dockerfile can be simple converted to the shell script. If you want to work in the native system. |
@Abrosimov-a-a Sure, let's indeed use docker to build them, that sounds great! |
@efiop The building system sugres several additional packages:
They are not in the |
Afaik they're already in setup.py. |
@casperdcl Yes, my fault. Already in |
Debian stable has some extra packages needed by DVC. But Debian version is older then setup.py needed. Are you sure you need a newer version of that packages?
|
Now you can test new building system: in my repo. |
@Abrosimov-a-a Are you talking about the stable deb repo? I'm pretty sure we'll be alright with those older versions. Btw, asciimatics is already removed 😉 |
Yes.
This will facilitate the work.
I know. I watched it. |
Waiting for your suggestions. |
build_deb: Debian directory appended. Issue #2800
So, what next?
|
|
Line 1 in 93280c9
Got it! https://www.debian.org/doc/manuals/maint-guide/dother.en.html |
Our deps are rapidly changing and it will be a challenge to keep up with them with corresponding deb packages. Considering that there hasn't been much activity with the deb package, I'm removing unused code for now and closing this issue. If anyone is interested in granularly packaging deb, please see https://github.com/iterative/dvc-s3-repo where we already build and deploy our standalone deb/rpm packages on our public repositories. Granular deb logic should be added there. |
What is your current way to build packages?
What about the system for automatically assembling the correct packages? I can help you with that.
Roadmap:
Determine the initial status.Build all dependencies in auto mode.Build the primary package: python3-dvcdebian/
for all packages.Current status:
Already in Debian repository:
Not in Debian:
asciimatics(Thrown out of code!)The text was updated successfully, but these errors were encountered: