Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try to fetch dependencies instead of building them locally #319

Closed
jbeich opened this issue May 31, 2015 · 21 comments
Closed

Try to fetch dependencies instead of building them locally #319

jbeich opened this issue May 31, 2015 · 21 comments
Labels
Performance
Projects
Default Tool Simplifications
Milestone
3.4.0

Comments

@jbeich
Copy link
Contributor

jbeich commented May 31, 2015

Testing ports before committing changes is expensive. So, is building overlay of a few ports with non-default options.

  • /head is a fast moving target, ports -u often renders many out-of-date
  • Tier1 support consists of 4 * 2 = 8 platform combinations
  • redports.org (CI) is still not available after it went down a year ago
  • repo layout doesn't support NO_ARCH, NO_OSREL, NO_OPSYS yet
  • some ports may require a lot of memory/time/diskactivity to build

Fortunately, pkg.freebsd.org already provides ready-to-use packages. Why not take advantage? Caveat: only useful when local ports tree and remote snapshot at the time don't diverge much.
@bdrewery
Copy link
Member

bdrewery commented Jun 6, 2015

Yes this is a wanted feature. I've called it 'package seeding'. The pkg.freebsd.org build script actually does this itself so we can build on different systems between each build. It's been on the TODOLIST forever but not yet implemented in Poudriere.

@f-andrey
Copy link

Also recently started to actively use poudriere to build packages for themselves and would like a similar function for the -f pkg-list.txt if part of a package with the changed options, then build them locally, and with default options fetch upstream repo.

@bdrewery bdrewery added this to the 3.2 milestone Jun 9, 2017
@bdrewery bdrewery mentioned this issue Jun 9, 2017
Closed
@bdrewery bdrewery modified the milestones: 3.2, 3.2.1 Jul 31, 2017
@tobiastom
Copy link

Is this already implemented in a branch, I'd like to help to test it.

@bdrewery
Copy link
Member

It's not yet implemented in Poudriere but I'm scheduled it for after the next release.

yunchih added a commit to yunchih/poudriere that referenced this issue Nov 25, 2017
@yunchih
bulk: Package seeding
5a5fbc9 
Sometimes it is not necessary to build ports that has been built
elsewhere with default options.  Just pull them over and use them
if they are just dependencies of the ports we really want to build
ourselves.  In particular, we can opportunistically pull binary
packages from pkg.freebsd.org if they have the exact version we want.

Fix freebsd#319
yunchih added a commit to yunchih/poudriere that referenced this issue Nov 25, 2017
@yunchih
bulk: Package seeding
218005a 
Sometimes it is not necessary to build ports that has been built
elsewhere with default options.  Just pull them over and use them
if they are just dependencies of the ports we really want to build
ourselves.  In particular, we can opportunistically pull binary
packages from pkg.freebsd.org if they have the exact version we want.

Fix freebsd#319
yunchih added a commit to yunchih/poudriere that referenced this issue Nov 25, 2017
@yunchih
bulk: Package seeding
fd19248 
Sometimes it is not necessary to build ports that has been built
elsewhere with default options.  Just pull them over and use them
if they are just dependencies of the ports we really want to build
ourselves.  In particular, we can opportunistically pull binary
packages from pkg.freebsd.org if they have the exact version we want.

Fix freebsd#319
@bdrewery bdrewery modified the milestones: 3.2.1, 3.2.2 Nov 27, 2017
@yunchih yunchih mentioned this issue Nov 27, 2017
Closed
yunchih added a commit to yunchih/poudriere that referenced this issue Dec 2, 2017
@yunchih
bulk: Package seeding
e4c61a8 
Sometimes it is not necessary to build ports that has been built
elsewhere with default options.  Just pull them over and use them
if they are just dependencies of the ports we really want to build
ourselves.  In particular, we can opportunistically pull binary
packages from pkg.freebsd.org if they have the exact version we want.

Fix freebsd#319
@bdrewery bdrewery modified the milestones: 3.2.3, 3.2.4 Dec 4, 2017
@bdrewery bdrewery modified the milestones: 3.2.4, 3.3.0 Dec 14, 2017
@yunchih
Copy link

yunchih commented Jan 16, 2018

Hi, @bdrewery any update on this patch? Really need your feedback on how I can refine this patch. Any hint appreciated. Thanks a lot.

@bdrewery
Copy link
Member

Sorry I'm been quite busy lately. I'll try to look in the next few weeks.

@shivno
Copy link

shivno commented Nov 28, 2018

any new info? thanks!

@bdrewery bdrewery modified the milestones: 3.3.0, 3.4.0 Feb 26, 2019
@heliosyne
Copy link

This would be very nice feature to have. I just updated to 3.3.99.20190311, but didn't see an option for pkg seeding. Is there news on this feature?

@lwhsu
Copy link
Member

lwhsu commented May 11, 2019

In the mean time, I am using some hackish way to do this: https://github.com/lwhsu/freebsd-ports-libreoffice/blob/master/porttest.sh

@wjwithagen
Copy link

Well, try to poudriere testport which needs both GCC 9.x and Clang 8.0 to build....
took about a day last time I needed to test that.

@b-a-t
Copy link
Member

b-a-t commented Nov 21, 2019

@bdrewery any hope to see this feature soon enough?

@bdrewery
Copy link
Member

Not likely as my focus is elsewhere, such as building less in incremental build and distributed support.

@ghost
Copy link

ghost commented May 28, 2020

Any update coming for this issue? Is there anything we can do to help?

@uqs
Copy link
Member

uqs commented Jun 22, 2020

Sorry for another "me too". I do like to build some ports that I'm using, as I've selected special flags on them. Things like Mesa et al. need a specific LLVM version though, and, well, I really don't want or need to build that myself. So it would be great if I could tell poudriere for some select ports to never build them, but fetch them instead.

Otherwise, my run ends up like so:

[freebsd:12:x86:64-default] [2020-06-22_09h45m06s] [parallel_build:] Queued: 1616 Built: 97   Failed: 0    Skipped: 0    Ignored: 0    Tobuild: 1519  Time: 02:21:47
        [01]: devel/llvm80              | llvm80-8.0.1_3            build           (01:50:32 / 01:51:56)
        [02]: devel/llvm90              | llvm90-9.0.1_1            build           (01:50:59 / 01:52:03)
        [03]: lang/gcc9                 | gcc9-9.3.0_1              build           (01:51:28 / 01:51:54)

I'd like to avoid that pointless churn. Thanks!

@swills
Copy link
Member

swills commented Jun 22, 2020

@uqs I hear you. I also think that there is another way to solve that particular problem, which is getting official packages built using the flags that you want. Could you share details on what flags you change?

@darksoul42
Copy link

Another me too here. Namely, I need the dns/unbound port with python, but I don't need to have all underlying dependencies built.

@Corvan
Copy link

Corvan commented Jul 14, 2020

+1

1 similar comment
@Panza0815
Copy link

+1

@pmhausen
Copy link

Either this feature or some documentation on how to use hooks to to achieve the same would be great!

@bdrewery
Copy link
Member

bdrewery commented Nov 6, 2020

#797 will cover this

@bdrewery
Copy link
Member

#797 is merged but there are a lot of pitfalls that make this not as useful as it seems. For example, llvm and all the other big stuff still builds. #822 will probably be needed to fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Performance
Projects
Default Tool Simplifications
Todo
Milestone
3.4.0
Development

Successfully merging a pull request may close this issue.

(WIP) Package seeding yunchih/poudriere