Atomic package distribution ideas #988
Comments
|
Thinking purely from the perspective of pkgsync, I could handle either of these. For what its worth, there is a symlink to an internal setname variant that is followed by the server: |
|
Avoiding the race mattering could also be achieved by refetching the repo metadata file after fetching the files - if they don't match, then it's probably a good idea to retry the fetch. Of course, if you're on a really slow connection then it may always take longer to get the packages you want than it takes the beefy machines to build them... |
|
I think the simplest would be to get /latest to do an actual redirect to the pool directory and then get Or we store a file in `latest/.latest_redirect' and which stores '../pool/.../' and have pkg fetch that file additionally and find to it using similar logic. Poudriere uses these symlinks currently: lrwxr-xr-x 1 root wheel 16 Mar 26 03:23 .latest@ -> .real_1427358217 drwxr-xr-x 4 root wheel 9 Mar 23 11:33 .real_1427128420/ drwxr-xr-x 4 root wheel 9 Mar 23 13:29 .real_1427134759/ drwxr-xr-x 4 root wheel 9 Mar 23 13:34 .real_1427135455/ drwxr-xr-x 4 root wheel 9 Mar 25 22:07 .real_1427339272/ drwxr-xr-x 4 root wheel 9 Mar 26 03:23 .real_1427358217/ lrwxr-xr-x 1 root wheel 11 Dec 3 2013 All@ -> .latest/All lrwxr-xr-x 1 root wheel 14 Dec 3 2013 Latest@ -> .latest/Latest lrwxr-xr-x 1 root wheel 19 Dec 3 2013 digests.txz@ -> .latest/digests.txz lrwxr-xr-x 1 root wheel 16 Sep 17 2014 meta.txz@ -> .latest/meta.txz lrwxr-xr-x 1 root wheel 23 Dec 3 2013 packagesite.txz@ -> .latest/packagesite.txz In this scheme it is currently URL/All/ which redirects to URL/.latest/All/ which goes through the .real symlink. I'd like some solution that works with both poudriere and pkg.freebsd.org. |
|
Any URL binding needs to leave out the actual hostname redirected to to not bind to specific mirrors. |
|
Of course any URL binding would need to sanity check that the URL it was given still is valid before fetching packages and if not then do a full |
|
Any file solution or repo metadata usage would need to be signed, which would require running |
|
Here is a proof of concept of @davidchisnall's idea. https://people.freebsd.org/~bdrewery/pkg-refetch.patch This is only covering Before: # pkg fetch -ya Updating myrepo repository catalogue... Fetching meta.txz: 100% 588 B 0.6kB/s 00:01 Fetching packagesite.txz: 100% 311 KiB 318.6kB/s 00:01 Processing entries: 100% myrepo repository update completed. 1222 packages processed The following packages will be fetched: New packages to be FETCHED: [...] The process will require 1 GiB more space. 1 GiB to be downloaded. Fetching irssi-0.8.17_1.txz: 100% 566 KiB 580.3kB/s 00:01 pkg: cached package irssi-0.8.17_1: size mismatch, fetching from remote Fetching irssi-0.8.17_1.txz: 100% 566 KiB 580.3kB/s 00:01 pkg: cached package irssi-0.8.17_1: size mismatch, cannot continue After: # pkg fetch -ya Updating myrepo repository catalogue... Fetching meta.txz: 100% 588 B 0.6kB/s 00:01 Fetching packagesite.txz: 100% 311 KiB 318.6kB/s 00:01 Processing entries: 100% myrepo repository update completed. 1222 packages processed The following packages will be fetched: New packages to be FETCHED: [...] The process will require 1 GiB more space. 1 GiB to be downloaded. pkg: cached package irssi-0.8.17_1: size mismatch, fetching from remote Fetching irssi-0.8.17_1.txz: 100% 566 KiB 580.3kB/s 00:01 pkg: cached package irssi-0.8.17_1: size mismatch, cannot continue [I meant to add a msg here explaining the refetch.] Updating myrepo repository catalogue... Fetching meta.txz: 100% 588 B 0.6kB/s 00:01 Fetching packagesite.txz: 100% 311 KiB 318.6kB/s 00:01 Processing entries: 100% myrepo repository update completed. 1222 packages processed The following packages will be fetched: New packages to be FETCHED: [...] The process will require 1 GiB more space. 1 GiB to be downloaded. Fetching irssi-0.8.17_1.txz: 100% 566 KiB 580.3kB/s 00:01 Fetching irssi-devel-20140530_2.txz: 100% 492 KiB 504.5kB/s 00:01 Fetching irssi-fish-1.00.r5.txz: 100% 52 KiB 54.2kB/s 00:01 Fetching irssi-otr-1.0.0_3.txz: 100% 56 KiB 58.3kB/s 00:01 Fetching irssi-scripts-20131030.txz: 100% 529 KiB 541.9kB/s 00:01 [...] This succeeds in the 2nd run. |
|
The patch above updates forever, but nevermind that. It's just a PoC. |
|
I like it |
|
goto retry: looks like quite a gratuitous way of implementing a loop. Making it a simple do...while() loop would be cleaner to me. You could also add a counter and give up after n attempts printing a helpful error message. |
|
Sure. Being a PoC I did not want to add reindent noise into it. |
|
The current status is that we need a volunteer to submit a patch that is a full solution. Mine was only for |
Question: where does I think @bdrewery's PoC is a good step in fixing this trivial, annoying scenario, and if possible, it would be nice if it was in the next release. There are multiple people that run into this particular problem at least once a month (in the developers@ group), which means there are likely more folks who are running into this issue in the wild. |
Logging here so I don't forget and for discussion.
Currently there is a race for users when packages are published. If they run
pkg upgrade, get the repo files, then before they start fetching packages the repository is modified (let's assume atomically), then their job will likely fail as the packages they are trying to fetch no longer exist or have new dependencies that require new solving.This problem exists for all pkg repositories, not only FreeBSD.org.
My idea is that when we build packages we name them as their cached name (i.e. the package has a hash in its name) just as we do when fetched. Then we publishing we do not delete any packages. Only add new. After a certain period of time (hours?) we can delete old packages. This would prevent users having a mismatch as they will always find the packages their catalog expects to be there. Without hashed filenames the user may get a broken package installed due to us not having reproducible packages. It may have the same version and rev as the older one the client expected, but now be linked against a different dependency.
Actually due to us checking checksums of fetched packages then this situation is detected and pkg bails out with an error. It's still a bit scary to the user when it is just a side effect of publishing the repository.
I'd prefer to find a solution that does not require out-of-band management of packages so I am not liking this idea much...
My previous idea was to make a "smart repository" where pkg binds itself to a specific URL advertised by the repository. Then the publisher can just make a new directory and remove the old when they please. Pkg would continue using the known URL until the next
pkg update. I still like the "smart repository" idea more. I thought I had an issue for it but cannot find it.@bapt @DarkHelmet433
The text was updated successfully, but these errors were encountered: