-
-
Notifications
You must be signed in to change notification settings - Fork 10.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#2690 - Imagebuilder: "opkg_install_pkg: Package size mismatch" error #7487
Comments
bjonglez: This change was introduced about one year ago: https://git.openwrt.org/e079591b84f8cb288ae85c3e2403ee6cb2de695b The first release with this change is 19.07.0-rc1, and it got backported in 18.06.5. |
bjonglez: Debugging this issue a bit, looking at a specific package:
My imagebuilder is from December and I try to build again today, here is the age and size of the files:
To sum it up:
This looks like a regression: before introducing the size check, I suspect the same issue happened (checksum mismatch), but opkg handled it fine. |
bjonglez: Same issue being reported here: openwrt/packages#10847 |
brianjmurrell: I am seeing the same issue on 19.07.0 final. |
brianjmurrell: Any chance in getting this resolved? It's making building images with the imagebuilder very painful. |
brianjmurrell: This one is getting pretty annoying. Short of nuking the entire download cache, this causes an image build to expand from 10 or so minutes into an all day affair as each image build only reveals a subset of packages that mismatch, and so only removing those that mismatch on each run and then running imagebuilder again results in a completely new set that need to be removed. Rinse. Repeat. Over. And. Over. Again. Before you know it, you've spent all day running imagebuilder and removing the next set of packages before you get a final successful build. |
bjonglez: Here is the workaround I use before running the imagebuilder, it removes download files older than one day:
find dl/ -name '*.ipk' -mtime +1 -delete
It means that running the imagebuilder several times in a row can still use the download cache. |
brianjmurrell: So yeah. That will make the success rate of IB go up, but it's still potentially deleting things that don't actually need to be deleted. On a metered connection, that could get expensive. I do appreciate the suggestion though. Cheers. |
brianjmurrell: The versions of the packages are not even changing. They seem to just be getting rebuilt all of the time for no apparent reason. My dl/ archive has libtirpc_1.2.5-1_mips_24kc.ipk downloaded on Mar 2 18:30 and the packages download site currently has libtirpc_1.2.5-1_mips_24kc.ipk built on Wed Mar 4 02:51:06 2020. Same version, why the frequent rebuilding? It seems like every time I run the IB so many packages have changed. Although frankly it wouldn't matter except for this nasty bug. How does it get any attention given that it's such a fundamental defect in the entire imagebuilder? |
brianjmurrell: FWIW, to cut down the number of times you have to iterate running IB, removing packages, running IB, removing packages, running IB, etc., after you hit your first failure (and thus //dl/openwrt_*// files have been updated) here's a scriplet that will remove the rest of the packages in //dl/// that will fail in subsequent runs:
It would be nice to run this even before the first IB run but I'm not quite sure how the //dl/openwrt_*// files are collected. I suspect they are the respective package repos //Packages.gz// files, but I didn't take the above scriptlet far enough to iterate through downloading those to find out. |
bjonglez: I had a look at the 18.06.4 imagebuilder (before the new size check) to see why it was working. It should have failed similarly, because the checksum from the old package in dl/ will likely be different from what is in the new package index. It turns out that the imagebuild doesn't seem to care at all about package checksums... It will happily use the old package from dl/ even though the checksum is wrong compared to what is in the package index. We could "fix" the issue by telling the imagebuilder to also ignore package size, but security-wise we should definitely be verifying checksums here... Once checksums are verified properly, fixing this issue would require to delete/re-download packages that failed the checksum. |
bjonglez: The lack of checksum verification was a bug, it was fixed in opkg in https://git.openwrt.org/c09fe2098718807d and this fix got applied in 18.06.7 |
bjonglez: Brian, your approach looks good, this should be a Makefile step that is run just after "opkg update". A similar approach could be implemented directly in opkg. |
brianjmurrell: But one thing I wonder is why do so many checksums change so often? I see literally the same package name/version/release/etc. deleted and then re-downloaded using my script above. |
jow-: Because the repos are continuously rebuilt and packages currently aren't reproducible (mtimes in tarballs change). |
brianjmurrell:
And packages that have not changed in any way are included in the rebuild?
I'm not parsing what this means. I'm not sure what it means for a package to be "reproducible". |
jow-:
Yes
A package (or a compilation result) is not reproducible if two different compilation runs of the same sources at two different times or on two different systems do no result in bit-identical files. |
brianjmurrell: So why do packages that have not changed in any way get rebuilt? That seems like unnecessary churn. Is it a goal to get packages reproducible? What variants are causing them not to be? |
jow-:
Because there is no reliable way to only rebuild single packages yet and because it would imply moving parts of the buildroot logic to repository download servers. In short, it is complex and has not been implemented so far. |
bjonglez: I just sent a patch series to opkg that fixes the issue: https://patchwork.ozlabs.org/project/openwrt/list/?series=197459 |
por: The patch would be most appreciated. My IB recipes clean out the caches before building an image and seeing everything being downloaded again and again feels lavish. |
brianjmurrell: What happened to the patch series mentioned 3 comments ago? Patchwork says "No patches to display" for https://patchwork.ozlabs.org/project/openwrt/list/?series=197459. |
bjonglez: They've been merged in master in November. Nobody seems to have complained since then, I'm going to send a backport for 19.07. |
brianjmurrell:
Ah, yes, this would be wonderful! |
bjonglez:
The Imagebuilder fails in the following situation:
It fails quite late during building with lots of errors like this:
This happens with stable releases (18.06.5), RC releases (19.07.0-rc2), and with several targets.
It looks like package sizes are somehow kept in a local cache during the first build, and are not updated when building again a few days later. This is strange because the imagebuilder seems to fetch new versions of the ''Packages.gz'' files.
Below is the full log.
The text was updated successfully, but these errors were encountered: