Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package list inconsistency for make dvd on raspberry pi 4 #290

Closed
JohannesNeu opened this issue May 15, 2022 · 28 comments
Closed

Package list inconsistency for make dvd on raspberry pi 4 #290

JohannesNeu opened this issue May 15, 2022 · 28 comments
Assignees
Labels
cleanup Low impact changes

Comments

@JohannesNeu
Copy link

JohannesNeu commented May 15, 2022

Hi,
after my last try (#285) I set the whole system up again and ran the compilation another time. The results were different.

I started screen and ran the following command: "make dvd DEVICE=ARM64 | & tee compile.log"
A full log is available, but 161MB big, which is why I will not attach it yet, but can provide it if necessary.

uname -a gives the following output:

root@rpicompile:/home/freebsd # uname -a
FreeBSD rpicompile 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr  9 06:06:55 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC  arm64

and the packages installed have the following versions:

root@rpicompile:/home/freebsd # pkg info
aarch64-binutils-2.37_2,1      GNU binary tools
ca_root_nss-3.76               Root certificate bundle from the Mozilla Project
curl-7.82.0                    Command line tool and library for transferring data with URLs
expat-2.4.8                    XML 1.0 parser written in C
gettext-runtime-0.21           GNU gettext runtime libraries and programs
git-2.35.2                     Distributed source code management tool
htop-3.1.2                     Better top(1) - interactive process viewer
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libnghttp2-1.46.0              HTTP/2.0 C Library
libssh2-1.10.0,3               Library implementing the SSH2 protocol
mpdecimal-2.5.1                C/C++ arbitrary precision decimal floating point libraries
nano-6.0                       Nano's ANOther editor, an enhanced free Pico clone
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.54                    Handle Common Gateway Interface requests and responses
p5-Clone-0.45                  Recursively copy Perl datatypes
p5-Digest-HMAC-1.04            Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05          Determine the locale encoding
p5-Error-0.17029               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_2               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.77            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-HTTP-Date-6.05              Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.36           Representation of HTTP style messages
p5-IO-HTML-1.004               Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.074         Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04         Guess media type for a file or a URL
p5-Mozilla-CA-20211001         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.90             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1             Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.10                    Perl5 interface to Uniform Resource Identifier (URI) references
pcre2-10.39_1                  Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.17.5_1                   Package manager
python38-3.8.13                Interpreted object-oriented programming language
readline-8.1.2                 Library for editing command lines as they are typed
screen-4.9.0_4                 Multi-screen window manager

I am seeing this error and do not know how to fix it:

The operation will free 20 MiB.
[1/9] Deinstalling urwfonts-1.0_8...
[1/9] Deleting files for urwfonts-1.0_8: .......... done
[2/9] Deinstalling fontconfig-2.13.94_2,1...
[2/9] Deleting files for fontconfig-2.13.94_2,1: .......... done
[3/9] Deinstalling mkfontscale-1.2.1...
[3/9] Deleting files for mkfontscale-1.2.1: ....... done
[4/9] Deinstalling freetype2-2.12.0...
[4/9] Deleting files for freetype2-2.12.0: .......... done
[5/9] Deinstalling expat-2.4.8...
[5/9] Deleting files for expat-2.4.8: .......... done
[6/9] Deinstalling libfontenc-1.1.4...
[6/9] Deleting files for libfontenc-1.1.4: ......... done
[7/9] Deinstalling pkg-1.17.5_1...
[7/9] Deleting files for pkg-1.17.5_1: .......... done
[8/9] Deinstalling pkgconf-1.8.0,1...
[8/9] Deleting files for pkgconf-1.8.0,1: .......... done
[9/9] Deinstalling png-1.6.37_1...
[9/9] Deleting files for png-1.6.37_1: .......... done
Creating repository in /usr/obj/usr/tools/config/22.1/OpenSSL:aarch64/.pkg-new/: .......... done
Packing files for repository: .... done
>>> Running build step: clean
>>> Passing arguments: packages
>>> Removing packages set
>>> Creating package mirror set for 22.1.7_30-OpenSSL-aarch64... done
-rw-r--r--  1 root  wheel   670M May 14 22:17 packages-22.1.7_30-OpenSSL-aarch64.tar
>>> WARNING: The build may have integrity issues!
>>> Package list inconsistency for emulators/qemu-guest-agent
>>> ERROR: The build encountered fatal issues!
>>> Aborted version 12.1 for devel/gdb
>>> Aborted version 4.6.2 for dns/powerdns-recursor
*** Error code 1

Stop.
make: stopped in /usr/tools

I tried to cross compile in a VM and receive a similar error (https://pastebin.com/1UNK0vzg), but would like to concentrate on the one I see with the raspberry pi.

The only reference I could find so far is this bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235346 but it is from some time ago and states that the issue should have been fixed.

When I try to compile without the "DEVICE=ARM64" I get the following error: https://pastebin.com/7XJBuSRm

I am a little bit stuck here and do not know how to proceed.
Any ideas?

@fichtner fichtner added the support Community support label May 15, 2022
@fichtner
Copy link
Member

Second error: only the “arm” image works on ARM, “dvd” doesn’t work. As for build errors in FreeBSD ports: it’s just the way it is sometimes. Something is fixed later, something else is broken.

@fichtner
Copy link
Member

PS: we could set power dns to ignore on aarch64, about geb I’m not sure. It should be useful, but was updated to a new version lately which likely causes this so rather wait this one out?

@JohannesNeu
Copy link
Author

@fichtner what would the command look like to create an arm image and what would the image be? A raw image I can copy onto a thumb drive/emmc?

@JohannesNeu
Copy link
Author

PS: we could set power dns to ignore on aarch64, about geb I’m not sure. It should be useful, but was updated to a new version lately which likely causes this so rather wait this one out?

I have not looked into modifying the image yet, but this may be a possibillity, on the other hand, as you can see from my cross compile efforts: a lot more packages seem to be affected. I would like to try to compile the arm image you mentioned next and see how it goes.

@fichtner
Copy link
Member

It’s in the docs really: „make arm“, DEVICE or ARCH may be required.

@JohannesNeu
Copy link
Author

I will try it, thanks.

@JohannesNeu
Copy link
Author

It looks a lot better now, when I use "make arm", this time I get stuck with a command not found error 127, when the arm.sh (https://github.com/opnsense/tools/blob/master/build/arm.sh) script is run. If I understand it right, the command "arm_install_uboot" which is mentioned in line 103 cannot be found. I checked whether there is a freebsd package containing the term uboot, but nothing turned up with "pkg search uboot".

The whole error looks like this:

=====
Message from opnsense-22.1.7_6:

--
Owl be watching you
>>> Begin extra: arm_hook
rc.loader: assembling banner
rc.loader: assembling brand
rc.loader: assembling misc
rc.loader: assembling modules
>>> End extra: arm_hook
>>> Setting up entropy in /usr/obj/tmp/opnsense/tools/config/22.1/OpenSSL:aarch64
1+0 records in
1+0 records out
4096 bytes transferred in 0.000511 secs (8014495 bytes/sec)
1+0 records in
1+0 records out
4096 bytes transferred in 0.000412 secs (9939070 bytes/sec)
>>> Building arm image... ./arm.sh: arm_install_uboot: not found
*** Error code 127

Stop.
make: stopped in /tmp/opnsense/tools

Should I open a new ticket for this?
A few lines below in line 106 I found a reference to something called "setup_efiboot" this is something I couldn't find either in the git repository. Did I miss to install something?

@fichtner
Copy link
Member

Do you have DEVICE set? If yes set UEFI= (empty value)

@fichtner
Copy link
Member

It rather looks like DEVICE is not set, which also means source build is busted anyway.

@fichtner
Copy link
Member

Forget about UEFI note, misinterpreted the code. It’s definitely DEVICE missing.

@JohannesNeu
Copy link
Author

I made sure, that DEVICE was used and the same error occurs: https://pastebin.com/cPBtvwBR

@fichtner
Copy link
Member

If you build rpi why not use the RPI(2) config? ARM64 doesn't have arm_install_uboot indeed

@fichtner
Copy link
Member

RPI2 seems to be the right one (if freebsd support is up for it now in FreeBSD 13)

@fichtner
Copy link
Member

I added the missing function in 184d300 -- from the technical side this bug can be closed?

@fichtner fichtner self-assigned this May 16, 2022
@fichtner fichtner added cleanup Low impact changes and removed support Community support labels May 16, 2022
fichtner added a commit that referenced this issue May 16, 2022
@JohannesNeu
Copy link
Author

I reinstalled freebsd and created a fresh setup with git clone, but it looks like there is a different error now.
Could this have been caused by the changes?

root@freebsd:/usr/tools # make arm DEVICE=ARM64

ABI_FILE=/usr/lib/crt1.o
BLOCKSIZE=K
CONFIGDIR=/usr/tools/config/22.1
CONFIG_XML=/usr/local/etc/config.xml
COREBRANCH=stable/22.1
COREDIR=/usr/core
COREENV=CORE_PHP=74 CORE_ABI=22.1 CORE_PYTHON=38
CPUS=4
DEVICE=ARM64
DEVICEDIR=/usr/tools/device
EDITOR=vi
ENV_FILTER=env -i USER=freebsd LOGNAME=freebsd HOME=/root SHELL=/bin/csh BLOCKSIZE=K MAIL=/var/mail/freebsd PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin TERM=xterm-256color HOSTTYPE=FreeBSD VENDOR=unknown OSTYPE=FreeBSD MACHTYPE=unknown PWD=/usr/tools/build GROUP=freebsd HOST=freebsd EDITOR=vi PAGER=less ABI_FILE=/usr/lib/crt1.o
EXTRABRANCH=
GROUP=freebsd
HOME=/root
HOST=freebsd
HOSTTYPE=FreeBSD
IMAGESDIR=/usr/local/opnsense/build/22.1/aarch64/images
LANG=C.UTF-8
LOGNAME=freebsd
LOGSDIR=/usr/local/opnsense/build/22.1/aarch64/logs
MACHTYPE=unknown
MAIL=/var/mail/freebsd
MAKEFLAGS= DEVICE=ARM64
MAKELEVEL=1
MAKE_ARGS_DEV=
META_MODE=normal
MM_CHARSET=UTF-8
OLDPWD=/usr/obj/usr/tools
OSTYPE=FreeBSD
PACKAGESDIR=/.pkg
PAGER=less
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
PLUGINSBRANCH=stable/22.1
PLUGINSDIR=/usr/plugins
PLUGINSENV=PLUGIN_PHP=74 PLUGIN_ABI=22.1 PLUGIN_PYTHON=38
PORTSBRANCH=master
PORTSDIR=/usr/ports
PORTSENV=
PORTSREFBRANCH=main
PORTSREFDIR=/usr/freebsd-ports
PORTSREFURL=https://git.FreeBSD.org/ports.git
PRINT_ENV_SKIP=1
PRODUCT_ABI=22.1
PRODUCT_ADDITIONS=
PRODUCT_ARCH=aarch64
PRODUCT_COMSPEED=115200
PRODUCT_CORE=opnsense
PRODUCT_CORES=opnsense opnsense-devel opnsense-business
PRODUCT_CRYPTO=openssl
PRODUCT_DEBUG=
PRODUCT_DEVEL=
PRODUCT_DEVICE_REAL=ARM64
PRODUCT_FLAVOUR=OpenSSL
PRODUCT_GITBASE=https://github.com/opnsense
PRODUCT_HOST=aarch64
PRODUCT_KERNEL=SMP-ARM
PRODUCT_LUA=5.3
PRODUCT_MIRROR=http://mirror.wdc1.us.leaseweb.net/opnsense
PRODUCT_NAME=OPNsense
PRODUCT_PERL=5.32
PRODUCT_PHP=74
PRODUCT_PLUGIN=os-*
PRODUCT_PLUGINS=os-*
PRODUCT_PRIVKEY=/usr/tools/config/22.1/repo.key
PRODUCT_PUBKEY=/usr/tools/config/22.1/repo.pub
PRODUCT_PYTHON=38
PRODUCT_RELEASE=OPNsense-202205161242-OpenSSL
PRODUCT_RUBY=27
PRODUCT_SERVER=user@does.not.exist
PRODUCT_SETTINGS=22.1
PRODUCT_SIGNCHK=/usr/tools/scripts/pkg_fingerprint.sh /usr/tools/config/22.1/repo.pub
PRODUCT_SIGNCMD=/usr/tools/scripts/pkg_sign.sh /usr/tools/config/22.1/repo.pub /usr/tools/config/22.1/repo.key
PRODUCT_SUFFIX=
PRODUCT_TARGET=arm64
PRODUCT_TYPE=opnsense
PRODUCT_UEFI=arm dvd serial vga vm
PRODUCT_VERSION=202205161242
PRODUCT_WANTS_CROSS=aarch64-binutils qemu-user-static
PRODUCT_ZFS=
PWD=/usr/tools/build
REMOTEHOST=pc.fritz.box
SETSDIR=/usr/local/opnsense/build/22.1/aarch64/sets
SHELL=/bin/csh
SHLVL=2
SRCABI=FreeBSD:13:aarch64
SRCBRANCH=stable/22.1
SRCDIR=/usr/src
SRCREVISION=13.0
SSH_CLIENT=192.168.178.20 56066 22
SSH_CONNECTION=192.168.178.20 56066 192.168.178.39 22
SSH_TTY=/dev/pts/0
STAGEDIR=/usr/obj/usr/tools/config/22.1/OpenSSL:aarch64
STAGEDIRPREFIX=/usr/obj
TARGETDIR=/usr/local/opnsense/build/22.1/aarch64
TARGETDIRPREFIX=/usr/local/opnsense/build
TERM=xterm-256color
TOOLSBRANCH=master
TOOLSDIR=/usr/tools
UPLOADDIR=.
USER=freebsd
VENDOR=unknown
WITHOUT_MODULES=cloudabi32
>>> Running build step: arm
>>> Passing arguments: (none)
>>> Running build step: clean
>>> Passing arguments: arm
>>> Removing arm image
>>> Setting up stage in /usr/obj/usr/tools/config/22.1/OpenSSL:aarch64
md0 created
md0s1 added
active set on md0s1
/dev/md0s1: 102264 sectors in 12783 FAT16 clusters (4096 bytes/cluster)
BytesPerSec=512 SecPerClust=8 ResSectors=1 FATs=2 RootDirEnts=512 Media=0xf0 FATsecs=50 SecPerTrack=63 Heads=255 HiddenSecs=0 HugeSectors=102400
md0s2 added
md0s2 created
md0s2a added
/dev/md0s2a: 3021.0MB (6187008 sectors) block size 32768, fragment size 4096
	using 5 cylinder groups of 625.22MB, 20007 blks, 80128 inodes.
	with soft updates
super-block backups (for fsck_ffs -b #) at:
 192, 1280640, 2561088, 3841536, 5121984
>>> Setting up base in /usr/obj/usr/tools/config/22.1/OpenSSL:aarch64
tar: Error opening archive: Failed to open '/usr/local/opnsense/build/22.1/aarch64/sets/base-*-aarch64.txz'
*** Error code 1

Stop.
make: stopped in /usr/tools

@fichtner
Copy link
Member

No, this line is executed before the line you reported earlier. 🤔

@JohannesNeu
Copy link
Author

No, this line is executed before the line you reported earlier. thinking

Should I create a new ticket for this?

@fichtner
Copy link
Member

If you know what the issue is supposed to be? tar: Error opening archive: Failed to open '/usr/local/opnsense/build/22.1/aarch64/sets/base-*-aarch64.txz' is where "tar" utility says the file isn't there or damaged. How would we fix? How would we know which it is without further information?

Cheers,
Franco

@JohannesNeu
Copy link
Author

I set it all up again and did some tests:
I ran make arm first and got the same problem: base-*-aarch64.txz missing
Then I ran "make dvd DEVICE=ARM64" and somewhere midway through the file "base-22.1.5_3-aarch64.txz" appeared in /usr/local/opnsense/build/22.1/aarch64/sets/ .
Could It be that "make arm" skips the compilation step and tries to copy the tar files into the image right away?

@fichtner
Copy link
Member

@JohannesNeu ah you are right... likely because arm was added much later than other image types, see
0a66d2c -- close then? :)

@JohannesNeu
Copy link
Author

@fichtner I did some more testing and compiled "make dvd" and "make arm" and I get the same error now:

make dvd:

Creating repository in /usr/obj/usr/tools/config/22.1/OpenSSL:aarch64/.pkg-new/: .......... done
Packing files for repository: .... done
>>> Running build step: clean
>>> Passing arguments: packages
>>> Removing packages set
>>> Creating package mirror set for 22.1.7_93-OpenSSL-aarch64... done
-rw-r--r--  1 root  wheel   653M May 17 14:46 packages-22.1.7_93-OpenSSL-aarch64.tar
>>> WARNING: The build may have integrity issues!
>>> Package list inconsistency for emulators/qemu-guest-agent
>>> ERROR: The build encountered fatal issues!
>>> Aborted version 12.1 for devel/gdb
>>> Aborted version 3.5.12_1,1 for mail/postfix35
>>> Aborted version 2.13.2 for net-mgmt/icinga2
*** Error code 1

Stop.
make: stopped in /usr/tools

make arm:

Creating repository in /usr/obj/usr/tools/config/22.1/OpenSSL:aarch64/.pkg-new/: .......... done
Packing files for repository: .... done
>>> Running build step: clean
>>> Passing arguments: packages
>>> Removing packages set
>>> Creating package mirror set for 22.1.7_93-OpenSSL-aarch64... done
-rw-r--r--  1 root  wheel   653M May 17 21:29 packages-22.1.7_93-OpenSSL-aarch64.tar
>>> WARNING: The build may have integrity issues!
>>> Package list inconsistency for emulators/qemu-guest-agent
>>> ERROR: The build encountered fatal issues!
>>> Aborted version 12.1 for devel/gdb
>>> Aborted version 3.5.12_1,1 for mail/postfix35
>>> Aborted version 2.13.2 for net-mgmt/icinga2
*** Error code 1

Stop.
make: stopped in /usr/tools

I assume the last change fixed the problems "make arm" had and now it is only left with the same problems "make dvd" had anyway :-)

icinga2 seems to be another packet that has problems being built on arm and could be blacklisted to get at least one step further? There may be more packages that will show up bit by bit though.

fichtner added a commit that referenced this issue May 18, 2022
@fichtner
Copy link
Member

Well, of course it's both problematic for "dvd" and "arm" since the "ports" stage is aborting here which is needed for any build. I disabled icinga, but again for gdb and postfix35 I'm not sure. You can throw both out locally to progress, but postfix also has a plugin that needs to be removed from plugins.conf then...

@JohannesNeu
Copy link
Author

Can I built a single package in debug mode to find out what is going wrong? Something like "make ports icinga2"?
I would then be able to create specific tickets, otherwise we might end up with a built that has all ports blacklisted.
And where would I create this issue? in opnsense/tools or in opnsense/ports?

@fichtner
Copy link
Member

Sure but to reiterate all of this is documented...

# make ports-icinga2

any argument given is treated as a package name to be removed and restarts the build which then fails for individual ports rather than doing a batch build.

@fichtner
Copy link
Member

@JohannesNeu
Copy link
Author

I created a bug ticket at Freebsd, the built process mentions devel/gdb so I assume that not only icinga2 may be affected by this, but some of the other ports as well. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264095

@fichtner
Copy link
Member

Thanks for reporting upstream. However, I think that we can't do much more here? Can we close this issue and wait and see? I'll be syncing upstream fixes each day for ports tree anyway.

@JohannesNeu
Copy link
Author

We can close it. If anything new arises, I will open a new ticket. The problem seems to be upstream and it takes time to fix the problem for each package, maybe some blacklisted ports can be removed from the blacklist in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cleanup Low impact changes
Development

No branches or pull requests

2 participants