-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide support for passing build flags down the dependency chain #259
Comments
You've misunderstood something. Ports does recursive builds with chaining down deps. Poudriere does not. It gathers a list of dependencies and then builds them in 1 level separate from each other. There is no recursion or chain to pass environment variables down into. There's no relation in package building between py-foo and py-dep. There's no way to have py-foo tell py-dep it wants version 33 versus 27. Yes we could have py-foo have a DEP_FLAGS variable in its Makefile that Poudriere will ensure all deps are built with, but that would still limit you to 1 version. How would poudriere know it needs to build py-foo twice? Building py-dep twice should be fine (py-dep27 and py-dep33) as they have different PKGNAMES can be stored in the queue separately. But I see no way to have the leaf port built twice as Poudriere all origin-based for this piece. I still think the simplest solution is to either package all versions of python in py-* (as gentoo does) or create slave ports as the rest of the ports framework/tree does. Creating them on-demand would be the most interesting. I would prefer this be in the framework though and not part of Poudriere as then we get into another "Port build tool XYZ is broken" like we are with Tinderbox now and portupgrade/portmaster often. |
This is about building dependencies with the correct information. py-foo tells the world that it (as well as its dependencies) needs version 33. A dependency thus needs to receive a piece of information that it shall built itself with version 33. Or do you talk about non-first-class ports (direct dependency of a port vs. dependency of a dependency)? That also should not be a problem as long as the initial port causing the build will set everything correctly (the same happens right now within a ports tree build) Slave ports are not a viable option here, since this easily grows out of hand (it already does for the small couple of gnome-related ones, which only target two version (ranges). In the worst case, one needs a slave port for each particular version (with 4-5 python version being supported at the same time in the tree), which easily adds up to hundreds of slave ports. |
I'm taking another look at this. I noticed that even bsd.ruby.mk has the |
Caching If we get dependencies B and C from PortA and dependencies C and D from PortB, we need to lookup further dependencies given the tuples B:PortA_DEPENDS_ARGS, C:PortA_DEPENDS_ARGS, C:PortB_DEPENDS_ARGS, and D:PortB_DEPENDS_ARGS. The DEPENDS_ARGS may all match so there would only be 1 actual lookup, but if they don't then we can discover new PKGNAMES from those lookups to queue. It could end up with A B C D, or A B B-II C C-II D. I think implementing that makes sense and would solve this. The problem then is that it's not easy with the current code (especially to keep performance acceptable for 24000 port build startups) but I do consider it a bug and in need of fixing. |
And those |
I think to fix this and set us up for the future of sub-packages, everything tracking an origin needs to instead track a "package" object which has stuff like the |
dns/unbound/Makefile:DEPENDS_ARGS+= WITH_GOST=yes dns/unbound/Makefile:DEPENDS_ARGS+= WITH_ECDSA=yes These are wrong since we don't generally support multiple packages installed at once but the plan with Poudriere will allow supporting something like this in the future perhaps. |
@bapt 's comment on this is that the queue should track |
There's 2 implementations that tackle this problem. https://github.com/bbqsrc/poudriere/commit/73dfbc4a88b79ca2288c0732522e04f38eabdcd8 and #314 |
I have a PoC for this working. testport net/tiny-network-utilities Current: [00:00:08] ====>> Debug: net/tiny-network-utilities | depends on net/py-netifaces [00:00:08] ====>> Debug: net/tiny-network-utilities | depends on textproc/py-hexdump [00:00:09] ====>> Debug: net/py-netifaces depends on lang/python27 [00:00:09] ====>> Debug: net/py-netifaces depends on devel/py-setuptools27 [00:00:09] ====>> Debug: devel/py-setuptools27 depends on lang/python27 [00:00:09] ====>> Debug: textproc/py-hexdump depends on devel/py-setuptools27 [00:00:09] ====>> Debug: textproc/py-hexdump depends on lang/python27 Fixed: [00:00:08] ====>> Debug: net/tiny-network-utilities | tiny-network-utilities-0.150519 depends on net/py-netifaces [00:00:08] ====>> Debug: net/tiny-network-utilities | tiny-network-utilities-0.150519 depends on textproc/py-hexdump [00:00:10] ====>> Debug: net/py-netifaces | py34-netifaces-0.10.4 depends on lang/python34 | python34-3.4.4_2 [00:00:10] ====>> Debug: net/py-netifaces | py34-netifaces-0.10.4 depends on devel/py-setuptools34 | py34-setuptools34-20.0 [00:00:10] ====>> Debug: textproc/py-hexdump | py34-hexdump-3.3 depends on devel/py-setuptools34 | py34-setuptools34-20.0 [00:00:10] ====>> Debug: textproc/py-hexdump | py34-hexdump-3.3 depends on lang/python34 | python34-3.4.4_2 There's several challenges to fix still. One is that for Another challenge is the What I've done is make a 2-tuple of |
Actually I will just add in a |
This may also break some ports. Example at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195342 with fix at https://bz-attachments.freebsd.org/attachment.cgi?id=150142 |
There's an actual patch to resolve this and to take it a step further to support |
Support DEPENDS_ARGS/FLAVORS - Building multiple packages from one origin. FLAVORS support is not yet in the Ports tree but has a patch and some discussion at https://reviews.freebsd.org/D10327. DEPENDS_ARGS support is only for python ports and allows dependencies to properly build py3- variants where a parent port has a USES=python=3 setting and the dependency is not a py3- slaveport that does the same. #259 has more information on this. Fixes #259 Fixes #313 Fixes #314
6ce2d0b dropped this support. |
Abstract
poudriere creates packages from ports in isolated jails, which feature their own set of environment variables that are usually set on starting the jail. A dependency X of a port Y thus is unaware of any environment settings the ports framework does on Y's behalf.
In some situations, it is necessary that environment settings are passed down the build chain to allow dependencies to build correctly based on whatever the framework decides "to be the best" for port Y.
Problem Space
As of writing, the FreeBSD ports tree allows users and ports to build, install and use different versions of the Python language (
lang/python
) at the same time. Ports using the Python bits of the ports framework (namelyMk/Uses/python.mk
) are enabled to choose a specific Python version or can be marked to support a series of Python versions, giving users (and other ports) the opportunity and flexibility to pick their specific preferred and supported version of the Python language.The pre-pkg era did not allow ports to install packages for different Python versions, which used the same origin information, which was resolved with the availability of pkg 1.3. The ports and pkg framework are capable of installing e.g. py33-foo and py27-foo originating from
devel/py-foo
at the same time.A single poudriere instance however is unable to create such a set of packages. Within a single run, it can only create a package py33-foo or py27-foo, but not both. This in turn limits the possibilities of creating different version-specific packages based on the same origin and won't allow different ports to refer to a shared dependency, which would support those versions.
Previous Workarounds
The limiting situation was resolved in the pre-pkg era by duplicating ports, renaming them and referring to either one of them within the specific consumers.
A port
py-onlypy33
may be limited to Python 3.3 and may depend onpy-foo
, which supports not only Python 3.3, but also Python 2.7. Another portpy-onlypy27
may also depend onpy-foo
, but only works with Python 2.7. This lead to the typical following workaround:BUILD_DEPENDS
orRUN_DEPENDS
BUILD_DEPENDS
orRUN_DEPENDS
Current Situation within the Ports
With pkg 1.3 and newer, a workaround as shown above is not necessary anymore. The Python bits of the ports framework allow ports to pass certain flags (specifically
DEPENDS_ARGS=PYTHON_VERSION=<versionnumber>
) to their dependencies, which can (re)act according to those. If a user wants to installpy-onlypy33
, the following will happen:make install
forpy-onlypy33
python.mk
will appendPYTHON_VERSION=python3.3
to theDEPENDS_ARGS
py-foo
dependencypython.mk
forpy-foo
noticesPYTHON_VERSION
and will instruct it to be built for Python 3.3py33-foo
is installedpy-onlypy33
detects the properpy33-foo
package and will continue to build and installIf the user is going to install
py-onlypy27
, the following will happen:make install
forpy-onlypy27
python.mk
will appendPYTHON_VERSION=python2.7
to theDEPENDS_ARGS
py-foo
dependencypython.mk
forpy-foo
noticesPYTHON_VERSION
and will instruct it to be built for Python 2.7py27-foo
is installedpy-onlypy27
detects the properpy27-foo
package and will continue to build and installCurrent Situation with Poudriere
Poudriere builds ports and their dependencies in (mostly) isolated jail environments, with the ability to parallelize builds. Information, such as the
DEPENDS_ARGS
from above are not shared or passed down the build chain. This results in the following behaviour forpy-onlypy33
:py-onlypy33
py-foo
using the configured default python version (e.g. 2.7)py27-foo
py-onlypy33
due to the unsatisfied dependency forpy33-foo
Solution
Poudriere needs to be able to pass around specific flags or settings, which can be set by a port or the ports framework. The ports framework may define a specific set of global variables, which should be respected by package builders (poudriere, tinderbox, etc.).
To prevent misbehaviour and to avoid limiting the parallelization features of package builders, those global variables should be available prior to any specific task for a port (
make fetch
,make patch
, etc.), so that the internal package builder workflow may look like:A single variable
PACKAGE_BUILD_FLAGS
may be sufficient and should be able to be passed down to dependencies of dependencies of dependencies of ...The text was updated successfully, but these errors were encountered: