How to ignore ports without skipping actual ones? #883

michael-o · 2021-06-22T10:44:26Z

Prerequisites

[X ] Have you checked for an existing issue describing the issue?

I believe that this isn't possible yet, so seeking for advice or need to create a feature request.

I have a repo with custom ports which aren't and won't be in the FreeBSD ports tree. One port is not selfcontained:

root@deblndw013x3j:~/ports/dns/register-hostnames
# cat Makefile
PORTNAME=               register-hostnames
PORTVERSION=    20210622
CATEGORIES=             dns
DISTFILES=

MAINTAINER=             michael-o
COMMENT=                Dynamically register hostnames in Active Directory

NO_ARCH=                yes
NO_BUILD=               yes
NO_TEST=                yes

RUN_DEPENDS=    kinit:security/krb5 nsupdate:dns/bind-tools

WRKSRC=                 ${FILESDIR}

do-install:
        @${MKDIR} ${STAGEDIR}/bin
        @${INSTALL_SCRIPT} ${WRKSRC}/register-hostnames ${STAGEDIR}/bin

.include <bsd.port.mk>

My jails have a base set of ports which include both dependencies from RUN_DEPENDS. That repos is build with:

poudriere bulk -j 122-release-amd64 -z ldadw-base -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-base-ports

Now I want to build this custom tree:

poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports

This of course will trigger the build of both runtime dependencies which I don't want since those are already in ldadw-base. So I have added both ports to a blacklist file and poudriere tells me:

[00:00:09] Skipped ports: dns/register-hostnames
[00:00:09] Ignored ports: security/krb5 dns/bind-tools
[122-release-amd64-default-ldadw-custom] [2021-06-22T12:25:16] [committing:] Queued: 3  Built: 0  Failed: 0  Skipped: 1  Ignored: 2  Tobuild: 0   Time: 00:00:07
[00:00:09] Logs: /var/poudriere/data/logs/bulk/122-release-amd64-default-ldadw-custom/2021-06-22T12:25:16
[00:00:09] Cleaning up
[00:00:09] Unmounting file systems

My port is skipped. My question: Is there anyway to tell poudriere to continue with the port and assume that the dependencies will be provided at runtime? This of course does not apply to build dependencies, but this could also be solved by compiling them, but not exposing them in the repo metadata when hosted for pkg.

Any help is appreciated. The only workaround I see is to remove the RUN_DEPENDS or move this port to ldadw-base, but other maybe upcoming custom ports will suffer from the same issue.

The text was updated successfully, but these errors were encountered:

bdrewery · 2021-06-22T18:23:09Z

Why are you making 2 separate repositories? Poudriere won't support the pkg idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.

michael-o · 2021-06-23T10:40:53Z

Why are you making 2 separate repositories? Poudriere won't support the pkg idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.

I will try to expand. While I could add this package into by base repository this would only solve the symptom not the problem. My idea is to have a base repo for all jails. Then, repos per jail (use case) type on top of base which may depend on packages in base or any other packages from the official ports tree. So to speak a layered design. I want to avoid recompiling ports which aren't necessary. My idea is very similar to one of @rootwyrm in #534.

Theoretically one could do the following:

Assume that repo base has been build with this id: 122-release-amd64-default-ldadw-base
I instruct poudriere to nullfs mount this local repo into the jail 122-release-amd64-default-ldadw-custom and poudriere will try to use these packages first and then build the rest from ports:

poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -M `122-release-amd64-default-ldadw-base` -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports

where -M would mount previously prepared local repos in /usr/local/etc/pkg/repos in the jail. Alternatively, there would be a directory for this run which could contain repos to include.

With that one can avoid duplication and introduce a layered design. WDYT?

michael-o · 2021-06-23T10:54:05Z

It seems like that #797 achieves a, simpler, but same goal though limited for some reason to official FreeBSD repo and the arbitrary one.

michael-o · 2021-06-23T12:26:36Z

I was able to very uglily trick poudriere with a modified version of #797:

Modified common.sh for:

    cat >> "${MASTERMNT}/etc/pkg/poudriere.conf" <<-EOF
    FreeBSD: {
            url: ${packagesite},
            mirror_type: "NONE",
            pubkey: "NONE",
            signature_type: "NONE"
    }
    EOF

Started the build:

poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports -b https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base -c

but it still wanted to build too many packages, then I did the following:

root@deblndw011x:/usr/local/etc/poudriere.d
# cp -p 122-release-amd64-default-ldadw-base-make.conf 122-release-amd64-default-ldadw-custom-make.conf
# cp -rp 122-release-amd64-default-ldadw-base-options 122-release-amd64-default-ldadw-custom-options

It is not fully clear why I need to fake this overlay and why pkg simply doesn't install both RUN_DEPENDS from the repo, but insists on the default options config. I guess it does not take it uses its own tree.

Is this a starting point we can discuss for layering? WDYT?

bdrewery · 2021-07-13T20:41:29Z

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

michael-o · 2021-07-13T20:47:58Z

Thanks, I'll look into it after my vacation.

michael-o · 2022-11-09T20:21:20Z

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.
I have a base set of ports I deploy on all servers, jails, etc. Recompiling them means also merge base options and specific options. That is rather a waste.

For one usecase I have now tried -b URL:

[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base
Bootstrapping pkg from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base, please wait...

This works, but since the repo config done in common.sh I need to apply: #883 (comment) since URL does not gurantee that it is a mirror of FreeBSD, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?

michael-o · 2022-11-09T21:25:26Z

The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.

michael-o · 2022-11-09T21:54:21Z

The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.

Found it. I still need to merge the port options from the repo to be used and the port options for this bulk. I will create an issue.

bdrewery · 2022-11-09T22:49:35Z

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.

To clarify something the dependencies are automatic but do not show in pkg autoremove because the metaports are non-automatic and still installed. So pkg autoremove shows nothing typically. pkg noauto is quite clean and only shows pkg and the metaports.

The leaf packages only show up in pkg autoremove if I trim them from the metaport and upgrade it.

Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.

samba# pkg noauto
pkg-1.18.4
local-fileserver-meta-20210811_12
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
samba# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

roundcube# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
local-webmail-meta-20220802_3
roundcube# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

prometheus# pkg noauto
pkg-1.18.4
prometheus-2.36.0_5
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
prometheus# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

matrix# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-matrix-meta-20221030_4
local-postfix-relay-meta-20210527_18,2
matrix# pkg autoremove -n
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 3 packages:

Installed packages to be REMOVED:
        matrix-conduit: 0.4.0_2
        py39-pip: 22.2.2
        py39-virtualenv: 16.7.5

Number of packages to be removed: 3

The operation will free 69 MiB.

For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or PORTREVISION change happens (and sane things like OPTIONS). It won't force rebuild everything so much anymore.

For one usecase I have now tried -b URL:

Yeah that makes sense as a workaround for what you want.

[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base
Bootstrapping pkg from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base, please wait...
This works, but since the repo config done in common.sh I need to apply: #883 (comment) since URL does not gurantee that it is a mirror of FreeBSD, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?

Definitely please open a separate issue so it can be fixed and tracked.

michael-o · 2022-11-09T23:02:27Z

@bdrewery Thanks, let me crunch on this and get back to you tomorrow. I think I have found yet another bug in Poudriere when building java/openjdk8:

====================================================
A new configuration has been successfully created in
/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/build/bsd-x86_64-normal-server-release
using configure arguments '--with-boot-jdk=/usr/local/bootstrap-openjdk8 --with-build-number=b07 --with-update-version=342 --with-vendor-name='OpenJDK BSD Porting Team' --with-vendor-url=https://github.com/battleblow/jdk8u/ --with-vendor-bug-url=mumu --with-vendor-vm-bug-url=mumu --disable-freetype-bundling --disable-zip-debug-info --with-cacerts-file=/usr/ports/java/openjdk8/files/cacerts --with-cups=/usr/local --with-debug-level=release --with-freetype=/usr/local --with-freetype-include=/usr/local/include/freetype2 --with-jobs=20 --with-jvm-variants=server --with-milestone=fcs --with-package-path=/usr/local --with-zlib=system --enable-unlimited-crypto --disable-debug-symbols --with-giflib=system --x-includes=/usr/local/include --x-libraries=/usr/local/lib --x-libraries=/usr/local/lib --x-includes=/usr/local/include --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd12.3'.

Configuration summary:
* Debug level:    release
* JDK variant:    normal
* JVM variants:   server
* OpenJDK target: OS: bsd, CPU architecture: x86, address length: 64

Tools summary:
* Boot JDK:       openjdk version "1.8.0_144" OpenJDK Runtime Environment (build 1.8.0_144-b01) OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)  (at /usr/local/bootstrap-openjdk8)
* Toolchain:      clang (clang/LLVM)
* C Compiler:     Version 10.0.1 (at /usr/bin/cc)
* C++ Compiler:   Version 10.0.1 (at /usr/bin/c++)

Build performance summary:
* Cores to use:   20
* Memory limit:   32601 MB

===========================================================================
=======================<phase: build          >============================
===== env: NO_DEPENDS=yes USER=nobody UID=65534 GID=65534
===>  Building for openjdk8-8.342.07.1_1
gmake[1]: Entering directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1'
(cd /wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make && /usr/local/bin/gmake 3 VERBOSE="3" LOG_LEVEL="warn" -R -I /wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make/common "SPEC=/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/build/bsd-x86_64-normal-server-release/spec.gmk" -j20 -f SourceRevision.gmk create-source-revision-tracker)
gmake[2]: Entering directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make'
gmake[2]: *** No rule to make target '3'.  Stop.
gmake[2]: Leaving directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make'
gmake[1]: *** [/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1//make/Main.gmk:207: create-source-revision-tracker] Error 2
gmake[1]: Leaving directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

is choking on the spaces here in the vendor arg:
https://github.com/freebsd/freebsd-ports/blob/main/java/openjdk8/Makefile#L70
patch:

root@deblndw011x:/var/poudriere/ports/default/java/openjdk8 (2022Q4-custom *)
# git diff -U0
diff --git a/java/openjdk8/Makefile b/java/openjdk8/Makefile
index 2638b33328ab..96e1e74d1156 100644
--- a/java/openjdk8/Makefile
+++ b/java/openjdk8/Makefile
@@ -70 +70 @@ CONFIGURE_ARGS+=  --with-update-version=${JDK_UPDATE_VERSION}
-CONFIGURE_ARGS+=   --with-vendor-name="OpenJDK BSD Porting Team" \
+CONFIGURE_ARGS+=   --with-vendor-name="OpenJDK_BSD_Porting_Team" \

makes it work. Will report also tomorrow as well.

bdrewery · 2022-11-09T23:04:25Z

That's a make thing Poudriere can't do much about.

michael-o · 2022-11-10T08:14:32Z

That's a make thing Poudriere can't do much about.

But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.

michael-o · 2022-11-10T08:49:16Z

That's a make thing Poudriere can't do much about.

But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.

Forget that, this is a problem in the build scripts of the OpenJDK, bad use of printf. I have created a patch and will create a bug with the OpenJDK 8 port.

michael-o · 2022-11-10T12:57:34Z

FTR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267690

michael-o · 2022-11-10T13:13:00Z

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.

To clarify something the dependencies are automatic but do not show in pkg autoremove because the metaports are non-automatic and still installed. So pkg autoremove shows nothing typically. pkg noauto is quite clean and only shows pkg and the metaports.

The leaf packages only show up in pkg autoremove if I trim them from the metaport and upgrade it.

Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.
samba# pkg noauto
pkg-1.18.4
local-fileserver-meta-20210811_12
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
samba# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

roundcube# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
local-webmail-meta-20220802_3
roundcube# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

prometheus# pkg noauto
pkg-1.18.4
prometheus-2.36.0_5
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
prometheus# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

matrix# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-matrix-meta-20221030_4
local-postfix-relay-meta-20210527_18,2
matrix# pkg autoremove -n
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 3 packages:

Installed packages to be REMOVED:
        matrix-conduit: 0.4.0_2
        py39-pip: 22.2.2
        py39-virtualenv: 16.7.5

Number of packages to be removed: 3

The operation will free 69 MiB.
For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or PORTREVISION change happens (and sane things like OPTIONS). It won't force rebuild everything so much anymore.

This adds up. I have, indeed, a very similar approach, but without the meta ports, so called "packages classes". I will consider this in the next round of improvements. The downside is that you have an extra layer which simplifies installation, but the actually ports you use aren't direct anymore and these are which I document. In anycase, both approaches work and have pros and cons. Thanks for sharing.

michael-o changed the title ~~How do ignore ports without skipping actual ones?~~ How to ignore ports without skipping actual ones? Jun 22, 2021

bdrewery added invalid Ports Issue An issue or behavior in the ports framework, not Poudriere labels Jun 22, 2021

bdrewery added the question label Aug 3, 2021

michael-o mentioned this issue Nov 10, 2022

Make bulk/testport -b URL not only support official repositories, but any repository #1025

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to ignore ports without skipping actual ones? #883

How to ignore ports without skipping actual ones? #883

michael-o commented Jun 22, 2021

bdrewery commented Jun 22, 2021

michael-o commented Jun 23, 2021 •

edited

michael-o commented Jun 23, 2021

michael-o commented Jun 23, 2021

bdrewery commented Jul 13, 2021

michael-o commented Jul 13, 2021

michael-o commented Nov 9, 2022 •

edited

michael-o commented Nov 9, 2022

michael-o commented Nov 9, 2022

bdrewery commented Nov 9, 2022

michael-o commented Nov 9, 2022

bdrewery commented Nov 9, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

How to ignore ports without skipping actual ones? #883

How to ignore ports without skipping actual ones? #883

Comments

michael-o commented Jun 22, 2021

Prerequisites

bdrewery commented Jun 22, 2021

michael-o commented Jun 23, 2021 • edited

michael-o commented Jun 23, 2021

michael-o commented Jun 23, 2021

bdrewery commented Jul 13, 2021

michael-o commented Jul 13, 2021

michael-o commented Nov 9, 2022 • edited

michael-o commented Nov 9, 2022

michael-o commented Nov 9, 2022

bdrewery commented Nov 9, 2022

michael-o commented Nov 9, 2022

bdrewery commented Nov 9, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

michael-o commented Nov 10, 2022

michael-o commented Jun 23, 2021 •

edited

michael-o commented Nov 9, 2022 •

edited