Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2690 - Imagebuilder: "opkg_install_pkg: Package size mismatch" error #7487

Open
openwrt-bot opened this issue Dec 21, 2019 · 24 comments
Open
Labels

Comments

@openwrt-bot
Copy link

openwrt-bot commented Dec 21, 2019

bjonglez:

The Imagebuilder fails in the following situation:

  • download and unpack the imagebuilder
  • build an image: it works
  • wait a few days/weeks
  • try to build an image again with the same imagebuilder: it fails

It fails quite late during building with lots of errors like this:

 opkg_install_pkg: Package size mismatch: libubox is 19365 bytes, expecting 19361 bytes
 opkg_install_cmd: Cannot install package luci.
 opkg_install_pkg: Package size mismatch: libuci is 15454 bytes, expecting 15461 bytes
 opkg_install_cmd: Cannot install package base-files.
 opkg_install_pkg: Package size mismatch: busybox is 212951 bytes, expecting 212973 bytes
 opkg_install_cmd: Cannot install package busybox.

This happens with stable releases (18.06.5), RC releases (19.07.0-rc2), and with several targets.

It looks like package sizes are somehow kept in a local cache during the first build, and are not updated when building again a few days later. This is strange because the imagebuilder seems to fetch new versions of the ''Packages.gz'' files.

Below is the full log.

$ make image PROFILE=rb-nor-flash-16M"" BIN_DIR="${IMAGEBUILDER_DIR}" EXTRA_IMAGE_NAME="rzn" FILES="files/" PACKAGES="luci luci-mod-admin-full luci-i18n-base-en luci-i18n-base-fr luci-i18n-firewall-fr -ppp -ppp-mod-pppoe -kmod-ppp -kmod-pppoe -kmod-pppox"

Building images for ar71xx - MikroTik RouterBoard (16 MB SPI NOR)
Packages: luci luci-mod-admin-full luci-i18n-base-en luci-i18n-base-fr luci-i18n-firewall-fr base-files busybox dnsmasq dropbear firewall fstools ip6tables iptables kernel kmod-ath9k kmod-gpio-button-hotplug kmod-ledtrig-gpio libc libgcc logd mtd netifd odhcp6c odhcpd-ipv6only opkg rbcfg rssileds swconfig uboot-envtools uci uclient-fetch wpad-mini

Downloading http://downloads.openwrt.org/releases/18.06.5/targets/ar71xx/mikrotik/packages/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_core
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/base/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_base
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_luci
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/packages/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_packages
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/routing/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_routing
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/telephony/Packages.gz
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/openwrt_telephony
Downloading file:packages/Packages
Updated list of available packages in /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/build_dir/target-mips_24kc_musl/root-ar71xx/../../../../../../../../../home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/imagebuilder

Installing packages...
Installing libc (1.1.19-2) to root...
Installing libgcc (7.3.0-2) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libgcc_7.3.0-2_mips_24kc.ipk.
Configuring libgcc.
Configuring libc.
Installing kernel (4.9.198-1-3ba2e8c5db2fc4389b537731cf37120c) to root...
Configuring kernel.
Installing luci (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci_git-19.353.67991-868abc1-1_all.ipk
Installing libubox (2018-07-25-c83a84af-2) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libubox_2018-07-25-c83a84af-2_mips_24kc.ipk.
Installing luci-mod-admin-full (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-mod-admin-full_git-19.353.67991-868abc1-1_mips_24kc.ipk
Installing luci-i18n-base-en (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-i18n-base-en_git-19.353.67991-868abc1-1_all.ipk
Installing luci-base (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-base_git-19.353.67991-868abc1-1_mips_24kc.ipk
Installing luci-i18n-base-fr (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-i18n-base-fr_git-19.353.67991-868abc1-1_all.ipk
Installing luci-i18n-firewall-fr (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-i18n-firewall-fr_git-19.353.67991-868abc1-1_all.ipk
Installing luci-app-firewall (git-19.353.67991-868abc1-1) to root...
Downloading http://downloads.openwrt.org/releases/18.06.5/packages/mips_24kc/luci/luci-app-firewall_git-19.353.67991-868abc1-1_all.ipk
Installing base-files (194.2-r7897-9d401013fc) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/base-files_194.2-r7897-9d401013fc_mips_24kc.ipk.
Installing libuci (2019-05-17-f199b961-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libuci_2019-05-17-f199b961-1_mips_24kc.ipk.
Installing busybox (1.28.4-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/busybox_1.28.4-3_mips_24kc.ipk.
Installing dnsmasq (2.80-1.4) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/dnsmasq_2.80-1.4_mips_24kc.ipk.
Installing dropbear (2017.75-7.1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/dropbear_2017.75-7.1_mips_24kc.ipk.
Installing firewall (2018-08-13-1c4d5bcd-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/firewall_2018-08-13-1c4d5bcd-1_mips_24kc.ipk.
Installing fstools (2019-03-28-ff1ded63-5) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/fstools_2019-03-28-ff1ded63-5_mips_24kc.ipk.
Installing ip6tables (1.6.2-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/ip6tables_1.6.2-3_mips_24kc.ipk.
Installing kmod-nf-reject6 (4.9.198-1) to root...
Downloading file:packages/kmod-nf-reject6_4.9.198-1_mips_24kc.ipk
Installing kmod-nf-ipt (4.9.198-1) to root...
Downloading file:packages/kmod-nf-ipt_4.9.198-1_mips_24kc.ipk
Installing kmod-nf-ipt6 (4.9.198-1) to root...
Downloading file:packages/kmod-nf-ipt6_4.9.198-1_mips_24kc.ipk
Installing kmod-ipt-core (4.9.198-1) to root...
Downloading file:packages/kmod-ipt-core_4.9.198-1_mips_24kc.ipk
Installing kmod-ip6tables (4.9.198-1) to root...
Downloading file:packages/kmod-ip6tables_4.9.198-1_mips_24kc.ipk
Installing libip4tc (1.6.2-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libip4tc_1.6.2-3_mips_24kc.ipk.
Installing libip6tc (1.6.2-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libip6tc_1.6.2-3_mips_24kc.ipk.
Installing libxtables (1.6.2-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libxtables_1.6.2-3_mips_24kc.ipk.
Installing iptables (1.6.2-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/iptables_1.6.2-3_mips_24kc.ipk.
Package iptables (1.6.2-3) installed in root is up to date.
Package kernel (4.9.198-1-3ba2e8c5db2fc4389b537731cf37120c) installed in root is up to date.
Installing kmod-ath9k (4.9.198+2017-11-01-10) to root...
Downloading file:packages/kmod-ath9k_4.9.198%2b2017-11-01-10_mips_24kc.ipk
Installing libnl-tiny (0.1-5) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/libnl-tiny_0.1-5_mips_24kc.ipk.
Installing kmod-gpio-button-hotplug (4.9.198-2) to root...
Downloading file:packages/kmod-gpio-button-hotplug_4.9.198-2_mips_24kc.ipk
Installing kmod-ledtrig-gpio (4.9.198-1) to root...
Downloading file:packages/kmod-ledtrig-gpio_4.9.198-1_mips_24kc.ipk
Package libc (1.1.19-2) installed in root is up to date.
Package libgcc (7.3.0-2) installed in root is up to date.
Installing logd (2018-02-14-128bc35f-2) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/logd_2018-02-14-128bc35f-2_mips_24kc.ipk.
Installing mtd (23) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/mtd_23_mips_24kc.ipk.
Installing libubox (2018-07-25-c83a84af-2) to root...
Installing netifd (2019-01-31-a2aba5c7-2.1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/netifd_2019-01-31-a2aba5c7-2.1_mips_24kc.ipk.
Installing odhcp6c (2018-07-14-67ae6a71-15) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/odhcp6c_2018-07-14-67ae6a71-15_mips_24kc.ipk.
Installing odhcpd-ipv6only (1.15-3) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/odhcpd-ipv6only_1.15-3_mips_24kc.ipk.
Installing opkg (2019-06-14-dcbc142e-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/opkg_2019-06-14-dcbc142e-1_mips_24kc.ipk.
Installing rbcfg (2) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/rbcfg_2_mips_24kc.ipk.
Installing rssileds (0.2-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/rssileds_0.2-1_mips_24kc.ipk.
Installing swconfig (11) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/swconfig_11_mips_24kc.ipk.
Installing uboot-envtools (2018.03-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/uboot-envtools_2018.03-1_mips_24kc.ipk.
Installing uci (2019-05-17-f199b961-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/uci_2019-05-17-f199b961-1_mips_24kc.ipk.
Installing uclient-fetch (2018-11-24-3ba74ebc-1) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/uclient-fetch_2018-11-24-3ba74ebc-1_mips_24kc.ipk.
Installing wpad-mini (2018-05-21-62566bc2-8) to root...
Copying /home/foo/doc/openwrt_imagebuilder/openwrt-imagebuilder-18.06.5-ar71xx-mikrotik.Linux-x86_64/dl/wpad-mini_2018-05-21-62566bc2-8_mips_24kc.ipk.
Configuring kmod-nf-reject6.
Configuring uboot-envtools.
Configuring kmod-ledtrig-gpio.
Configuring rbcfg.
Configuring luci-app-firewall.
Configuring kmod-nf-ipt.
Configuring libxtables.
Configuring libip4tc.
Configuring kmod-nf-ipt6.
Configuring kmod-ipt-core.
Configuring kmod-ip6tables.
Configuring fstools.
Configuring libip6tc.
Configuring luci-base.
Configuring luci-mod-admin-full.
Configuring kmod-gpio-button-hotplug.
Configuring luci-i18n-firewall-fr.
Configuring iptables.
Configuring ip6tables.
Configuring luci-i18n-base-en.
Configuring luci-i18n-base-fr.
Collected errors:
 * opkg_install_pkg: Package size mismatch: libubox is 19365 bytes, expecting 19361 bytes
 * opkg_install_cmd: Cannot install package luci.
 * opkg_install_pkg: Package size mismatch: libuci is 15454 bytes, expecting 15461 bytes
 * opkg_install_cmd: Cannot install package base-files.
 * opkg_install_pkg: Package size mismatch: busybox is 212951 bytes, expecting 212973 bytes
 * opkg_install_cmd: Cannot install package busybox.
 * opkg_install_pkg: Package size mismatch: dnsmasq is 99783 bytes, expecting 99775 bytes
 * opkg_install_cmd: Cannot install package dnsmasq.
 * opkg_install_pkg: Package size mismatch: dropbear is 86613 bytes, expecting 86610 bytes
 * opkg_install_cmd: Cannot install package dropbear.
 * opkg_install_pkg: Package size mismatch: firewall is 40776 bytes, expecting 40777 bytes
 * opkg_install_cmd: Cannot install package firewall.
 * opkg_install_pkg: Package size mismatch: libnl-tiny is 12953 bytes, expecting 12948 bytes
 * opkg_install_cmd: Cannot install package kmod-ath9k.
 * opkg_install_pkg: Package size mismatch: logd is 10114 bytes, expecting 10111 bytes
 * opkg_install_cmd: Cannot install package logd.
 * opkg_install_pkg: Package size mismatch: libubox is 19365 bytes, expecting 19361 bytes
 * opkg_install_cmd: Cannot install package mtd.
 * opkg_install_pkg: Package size mismatch: netifd is 62005 bytes, expecting 62004 bytes
 * opkg_install_cmd: Cannot install package netifd.
 * opkg_install_pkg: Package size mismatch: odhcp6c is 23650 bytes, expecting 23646 bytes
 * opkg_install_cmd: Cannot install package odhcp6c.
 * opkg_install_pkg: Package size mismatch: odhcpd-ipv6only is 28830 bytes, expecting 28845 bytes
 * opkg_install_cmd: Cannot install package odhcpd-ipv6only.
 * opkg_install_pkg: Package size mismatch: opkg is 58890 bytes, expecting 58895 bytes
 * opkg_install_cmd: Cannot install package opkg.
 * opkg_install_pkg: Package size mismatch: rssileds is 4056 bytes, expecting 4059 bytes
 * opkg_install_cmd: Cannot install package rssileds.
 * opkg_install_pkg: Package size mismatch: swconfig is 9478 bytes, expecting 9484 bytes
 * opkg_install_cmd: Cannot install package swconfig.
 * opkg_install_pkg: Package size mismatch: uci is 7267 bytes, expecting 7262 bytes
 * opkg_install_cmd: Cannot install package uci.
 * opkg_install_pkg: Package size mismatch: uclient-fetch is 7489 bytes, expecting 7485 bytes
 * opkg_install_cmd: Cannot install package uclient-fetch.
 * opkg_install_pkg: Package size mismatch: wpad-mini is 258733 bytes, expecting 258725 bytes
 * opkg_install_cmd: Cannot install package wpad-mini.
make[2]: *** [Makefile:153: package_install] Error 255
make[1]: *** [Makefile:114: _call_image] Error 2
make: *** [Makefile:193: image] Error 2
@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 1, 2020

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 1, 2020

bjonglez:

Debugging this issue a bit, looking at a specific package:

opkg_install_pkg: Package size mismatch: libjson-c2 is 14704 bytes, expecting 14699 bytes

My imagebuilder is from December and I try to build again today, here is the age and size of the files:

$ ls -l openwrt-imagebuilder-19.07.0-rc2-ath79-generic.Linux-x86_64/dl/
-rw-r--r-- 1 zorun zorun  14704 Dec 11 09:39 libjson-c2_0.12.1-3_mips_24kc.ipk
-rw-r--r-- 1 zorun zorun  52200 Jan  1 05:59 openwrt_base

$ less openwrt-imagebuilder-19.07.0-rc2-ath79-generic.Linux-x86_64/dl/openwrt_base
Package: libjson-c2
Version: 0.12.1-3
Depends: libc
Provides: libjson-c
License: MIT
Section: libs
Architecture: mips_24kc
Installed-Size: 13978
Filename: libjson-c2_0.12.1-3_mips_24kc.ipk
Size: 14699
SHA256sum: bde7f9f579c561b9af3f30108d2bc0537d690b6d07b94e2a1a18f4d5b9a66081
Description:  This package contains a library for javascript object notation backends.

$ sha256sum openwrt-imagebuilder-19.07.0-rc2-ath79-generic.Linux-x86_64/dl/libjson-c2_0.12.1-3_mips_24kc.ipk
4d8487c4b7d67ff0ffe4e7be43c49813a36afdb8584a73a078e0d53cf757d8ce  dl/libjson-c2_0.12.1-3_mips_24kc.ipk

To sum it up:

  • I still have old versions of packages in my local "dl/" folder
  • the new package index is correctly downloaded
  • the new package index refers to updated versions of packages, which have different sizes and checksum
  • instead of just deleting the outdated packages and downloading them again, opkg fails

This looks like a regression: before introducing the size check, I suspect the same issue happened (checksum mismatch), but opkg handled it fine.

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 1, 2020

bjonglez:

Same issue being reported here: openwrt/packages#10847

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 22, 2020

brianjmurrell:

I am seeing the same issue on 19.07.0 final.

@openwrt-bot
Copy link
Author

openwrt-bot commented Feb 22, 2020

brianjmurrell:

Any chance in getting this resolved?

It's making building images with the imagebuilder very painful.

@openwrt-bot
Copy link
Author

openwrt-bot commented Mar 3, 2020

brianjmurrell:

This one is getting pretty annoying. Short of nuking the entire download cache, this causes an image build to expand from 10 or so minutes into an all day affair as each image build only reveals a subset of packages that mismatch, and so only removing those that mismatch on each run and then running imagebuilder again results in a completely new set that need to be removed.

Rinse.

Repeat.

Over. And. Over. Again.

Before you know it, you've spent all day running imagebuilder and removing the next set of packages before you get a final successful build.

@openwrt-bot
Copy link
Author

openwrt-bot commented Mar 3, 2020

bjonglez:

Here is the workaround I use before running the imagebuilder, it removes download files older than one day:

find dl/ -name '*.ipk' -mtime +1 -delete

It means that running the imagebuilder several times in a row can still use the download cache.

@openwrt-bot
Copy link
Author

openwrt-bot commented Mar 3, 2020

brianjmurrell:

So yeah. That will make the success rate of IB go up, but it's still potentially deleting things that don't actually need to be deleted.

On a metered connection, that could get expensive.

I do appreciate the suggestion though. Cheers.

@openwrt-bot
Copy link
Author

openwrt-bot commented Mar 5, 2020

brianjmurrell:

The versions of the packages are not even changing. They seem to just be getting rebuilt all of the time for no apparent reason.

My dl/ archive has libtirpc_1.2.5-1_mips_24kc.ipk downloaded on Mar 2 18:30 and the packages download site currently has libtirpc_1.2.5-1_mips_24kc.ipk built on Wed Mar 4 02:51:06 2020.

Same version, why the frequent rebuilding? It seems like every time I run the IB so many packages have changed.

Although frankly it wouldn't matter except for this nasty bug.

How does it get any attention given that it's such a fundamental defect in the entire imagebuilder?

@openwrt-bot
Copy link
Author

openwrt-bot commented Mar 5, 2020

brianjmurrell:

FWIW, to cut down the number of times you have to iterate running IB, removing packages, running IB, removing packages, running IB, etc., after you hit your first failure (and thus //dl/openwrt_*// files have been updated) here's a scriplet that will remove the rest of the packages in //dl/// that will fail in subsequent runs:

for f in dl/openwrt_; do
zcat $f | sed -ne '/^Filename:/s/.
//p' -e '/^SHA256sum:/s/.* //p' | while read file; do
read sum
if [ -f dl/$file ]; then
read sum1 junk < <(sha256sum dl/$file)
if [ $sum != $sum1 ]; then
rm -f dl/$file
fi
fi
done
done

It would be nice to run this even before the first IB run but I'm not quite sure how the //dl/openwrt_*// files are collected. I suspect they are the respective package repos //Packages.gz// files, but I didn't take the above scriptlet far enough to iterate through downloading those to find out.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 22, 2020

bjonglez:

I had a look at the 18.06.4 imagebuilder (before the new size check) to see why it was working.

It should have failed similarly, because the checksum from the old package in dl/ will likely be different from what is in the new package index.

It turns out that the imagebuild doesn't seem to care at all about package checksums... It will happily use the old package from dl/ even though the checksum is wrong compared to what is in the package index.

We could "fix" the issue by telling the imagebuilder to also ignore package size, but security-wise we should definitely be verifying checksums here...

Once checksums are verified properly, fixing this issue would require to delete/re-download packages that failed the checksum.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

bjonglez:

The lack of checksum verification was a bug, it was fixed in opkg in https://git.openwrt.org/c09fe2098718807d and this fix got applied in 18.06.7

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

bjonglez:

Brian, your approach looks good, this should be a Makefile step that is run just after "opkg update".

A similar approach could be implemented directly in opkg.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

brianjmurrell:

But one thing I wonder is why do so many checksums change so often? I see literally the same package name/version/release/etc. deleted and then re-downloaded using my script above.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

jow-:

Because the repos are continuously rebuilt and packages currently aren't reproducible (mtimes in tarballs change).

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

brianjmurrell:

Because the repos are continuously rebuilt

And packages that have not changed in any way are included in the rebuild?

and packages currently aren't reproducible

I'm not parsing what this means. I'm not sure what it means for a package to be "reproducible".

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

jow-:

And packages that have not changed in any way are included in the rebuild?

Yes

I'm not sure what it means for a package to be "reproducible".

A package (or a compilation result) is not reproducible if two different compilation runs of the same sources at two different times or on two different systems do no result in bit-identical files.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

brianjmurrell:

So why do packages that have not changed in any way get rebuilt? That seems like unnecessary churn.

Is it a goal to get packages reproducible? What variants are causing them not to be?

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

jow-:

So why do packages that have not changed in any way get rebuilt? That seems like unnecessary churn.

Because there is no reliable way to only rebuild single packages yet and because it would imply moving parts of the buildroot logic to repository download servers.

In short, it is complex and has not been implemented so far.

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 24, 2020

bjonglez:

I just sent a patch series to opkg that fixes the issue: https://patchwork.ozlabs.org/project/openwrt/list/?series=197459

@openwrt-bot
Copy link
Author

openwrt-bot commented Aug 25, 2020

por:

The patch would be most appreciated. My IB recipes clean out the caches before building an image and seeing everything being downloaded again and again feels lavish.
So +1.

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 4, 2021

brianjmurrell:

What happened to the patch series mentioned 3 comments ago?

Patchwork says "No patches to display" for https://patchwork.ozlabs.org/project/openwrt/list/?series=197459.

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 4, 2021

bjonglez:

They've been merged in master in November.

Nobody seems to have complained since then, I'm going to send a backport for 19.07.

@openwrt-bot
Copy link
Author

openwrt-bot commented Jan 4, 2021

brianjmurrell:

I'm going to send a backport for 19.07.

Ah, yes, this would be wonderful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant