Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support DEPENDS_ARGS/FLAVORS - Building multiple packages from one origin #483

Merged
merged 122 commits into from
Jun 16, 2017
Merged

Support DEPENDS_ARGS/FLAVORS - Building multiple packages from one origin #483

merged 122 commits into from
Jun 16, 2017

Conversation

bdrewery
Copy link
Member

@bdrewery bdrewery commented Jun 6, 2017

This fixes #259 and #313, and obsoletes #314.

Remaining items:

  1. The next thing will be to add FLAVORS support for https://reviews.freebsd.org/D10327. This should mostly be trivial.
  2. Support ORIGIN@FLAVOR syntax for bulk -f list and bulk port, etc via listed_ports
    Ensure that ORIGIN ORIGIN@FLAVOR1 ORIGIN@FLAVOR2 works together.
  3. Support RUN_DEPENDS+= foo>0:devel/foo:FLAVOR=flave
  4. Make is_bad_flavor_slave_port requeue the main port with the proper DEPENDS_ARGS=PYTHON_VERSION for the originally request port. This will relieve the pressure to commit FLAVORS support immediately to prevent breaking all of the python ports listed.
  5. The chosen FLAVOR or DEPENDS_ARGS is not always clear in the web interface but should be computable from the log. Not sure I will modify this before merge. (Will handle in JSON/HTML tracks by origin but should be pkgname #415)
  6. All cache_get_origin need auditted to ensure they shouldn't be using originspec, like bulk showing ports to build.
  7. The reinjection of items into the queue for DEPENDS_ARGS and FLAVORS needs to be unified for simplicity. moved to FLAVORS and DEPENDS_ARGS support is insane #492
  8. Similar to (7), FLAVORS and DEPENDS_ARGS should not be supported set together.
  9. Support origin@- to only build the first default flavor.
  10. Need a test suite desperately. There's so many test cases here that keep failing when I change things. Need automated bulk tests #488
  11. Fix the CHECK_CHANGED_DEPS support for ORIGIN@FLAVOR dependency lines. It currently will bomb since the packages only record ORIGIN. Need to convert to PKGBASE or something. This code belongs in ports badly. CHECKED_CHANGED_DEPS support for originspecs #495

The python slave port issue is quite complex.
If a port like net/tiny-network-utilities requires net/py-netifaces as python3 it currently does not work due to the DEPENDS_ARGS bug. The workaround today in ports is to change the dependency to a new slave port such as net/py3-netifaces. In this specific example net/tiny-network-utilities does not depend on a py3 slaveport. Once Poudriere is merged with DEPENDS_ARGS support it, with bulk -a will then queue the virtual net/py-netifaces for python2 (default) and then also python3 from the net/tiny-network-utilities DEPENDS_ARGS dependency. It will also queue a build for net/py3-netifaces. The slave port and the dependency-generated package will both be 'py3-netifaces' for PKGNAME and thus hit the duplicated PKGNAME assertion and fail to build.
To work around this I force mapped origins that had a slave port added but would build just fine now from the main port from a DEPENDS_ARGS-generated package. The problem that comes from all of this is that if a py3 slave port was added not for a dependency solution but purely to provide a py3 package for a user. In that case without FLAVORS a py3- package will never generate from the default port. So merging this into the package builds will drop a lot of py3- packages. That is fixed by merging FLAVORS support to the python ports which then will ensure all main python ports provide a py2 and py3 package from the origin whether they are a dependency or not.

Performance of bulk -a startup is hurt by 7% but it's relatively fast still.

x before
+ after
+------------------------------------------------------------+
|                                                           +|
|   x                                                       +|
| x x      x                                                +|
||__MA___|                                                   |
|                                                           A|
+------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   4        275.69        279.24        276.52      276.9925     1.5485773
+   3           297           297           297           297             0
Difference at 95.0% confidence
        20.0075 +/- 2.35542
        7.22312% +/- 0.911779%
        (Student's t, pooled s = 1.19952)

Such as net/tiny-network-utilities which Poudriere previously
failed to build on.
Mapping an existing package with an origin back to what origin and PKGNAME
might be in the queue now is not straight forward because the origin may be in
it with any number of DEPENDS_ARGS/FLAVOR to produce multiple packages.  What
we can do is look for a matching PKGBASE at the known origin.  From here we
take the closet matching PKGBASE as the new PKGNAME.
This is ugly and needs some better handling, but we need
to display at least PKGNAME to make it more clear what
is being displayed since 1 origin may now produce
multiple packages.  It would be enough to just show
PKGNAME but it is far less convenient.
…pecs.

The default originspec is now looked up first.  Any dependency wanting
to lookup a specific originspec (DEPENDS_ARGS or FLAVOR) is added
to the flavorqueue and processed later.
This fixes the -a case and avoids writing extra files.
This will push non-default originspecs into the depqueue.
…f gather_port_vars()

For the non -a case still assume that all flavors of an origin are to be
considered as listed.
Conflicts:
	src/share/poudriere/common.sh
…orts.

- Trim them from the build queue
- Replace them as dependencies to the main port which supports
  DEPENDS_ARGS and FLAVORS
This also allows testing on virtual (non-existent) py3 ports like
textproc/py3-hexdump, but it otherwise is required to be able
to properly build and map ports like net/py3-netifaces.
Other things were refactored to use pkgname and cached lists for that.
…_ARE_FATAL.

Consider bad dependency lines on the wrong PKGNAME as fatal.
For example:
   BUILD_DEPENDS=  p5-List-MoreUtils>=0:lang/p5-List-MoreUtils
If this port's PKGNAME were really "List-MoreUtils" then it would
not be recorded into the resulting package.  The next build with
CHECK_CHANGED_DEPS enabled would consider it a "new dependency"
since it is in the the port but not in the package.  This is usually
a warning but can be made fatal instead by enabling this option.
…LAVORS.

This also reworks some of the previously added assertions that FLAVORS cannot
be combined with DEPENDS_ARGS since we don't ever lookup both now.
Conflicts:
	src/share/poudriere/common.sh
@bdrewery bdrewery merged commit 8be6e8d into freebsd:master Jun 16, 2017
@bdrewery bdrewery deleted the pkgspec branch June 16, 2017 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide support for passing build flags down the dependency chain
1 participant