Cannot sync every repo's DBs #92
Comments
|
Tried one more sync without changing my setup and now multilib isn't syncing. |
|
And you can notice that it is adding an extra slash after the url before $repo. It reads archlinux// , instead of archlinux / (Not that it affects the end result) |
|
No changes in config and have this now, maybe related? |
|
Okay it looks here are several issues reported.
@Focshole had this issue before. @afonsofrancof could it be some race condition between pacoloco and reflector changing the file?
it should be fixed in
This error is weird. Do you always see this issue or it just sporadic? |
|
I had that issue when pacoloco mirrorlist was unreadable or empty for some reason. Is that file readable from the container? |
|
@Focshole Yes, the file is readable since some of the repos can sync.
|
I don't have reflector running anymore, that was the name of the file because I had it running before. And it would make no sense since it works if I separate the repos.
Only sometimes, I have no idea what conditions cause it. |
|
I think I've run into the same issue. It seems to happen when the mirrorlist file is bind-mounted directly. Bind-mounting the parent directory instead ( No idea why pacoloco chokes on bind-mounted files though... |
|
I'm having this problem without using a docker bind mount. Is there a debug log option / environment variable? Also, how do I tell what version of pacoloco I am running? Besides inspecting the docker sha256 digest which isn't very friendly? Repro below. Version: docker-compose.yaml: pacoloco:
image: ghcr.io/anatol/pacoloco
container_name: prod-pacoloco
restart: unless-stopped
user: "122:1999" # pacoloco:pacoloco
environment:
TZ: "UTC"
ports:
- "127.0.0.1:9129:9129"
volumes:
- 'pacoloco:/var/cache/pacoloco'
- '/root/etc-docker/pacoloco/pacoloco.yaml:/etc/pacoloco.yaml:ro'
- '/root/etc-docker/pacoloco:/data:ro'
- '/etc/passwd:/etc/passwd:ro'
- '/etc/group:/etc/group:ro'
- '/etc/localtime:/etc/localtime:ro'
- '/etc/timezone:/etc/timezone:ro'
logging:
<<: *loggingI have nginx proxying to pacoloco. That part works when pacoloco does. pacoloco.yaml: purge_files_after: 1814400 # 21 days
download_timeout: 3600 # download will timeout after 3600 seconds
repos:
# archlinux:
# urls:
# - http://mirror.lty.me/archlinux
# - http://mirrors.kernel.org/archlinux
archlinux:
mirrorlist: /data/mirrorlist
prefetch: # optional section, add it if you want to enable prefetching
cron: 41 5 * * *
ttl_unaccessed_in_days: 30 # defaults to 30, set it to a higher value than the number of consecutive days you don't update your systems
# It deletes and stop prefetch packages(and db links) when not downloaded after ttl_unaccessed_in_days days that it had been updated.
ttl_unupdated_in_days: 300 # defaults to 300, it deletes and stop prefetch packages which hadn't been either updated upstream or requested for ttl_unupdated_in_days.
set_timestamp_to_logs: truepacoloco can read the file: ~# docker-compose exec pacoloco sh -c "id; tail -n1 /data/mirrorlist"
uid=122(pacoloco) gid=1999(pacoloco) groups=1999(pacoloco)
Server = https://mirror.rackspace.com/archlinux/$repo/os/$archLogs: Client logs: ReproductionFor me this repro reliably prints errors, but which one fails (core or extra) is switches. docker network create paco-test; \
docker stop pacoloco-test; \
docker rm pacoloco-test; \
mkdir -p /tmp/pactest && cd /tmp/pactest && \
{ cat >pacoloco.yaml <<EOF
repos:
archlinux:
mirrorlist: /data/pacoloco-mirrorlist
cache_dir: /tmp
EOF
} && \
{ cat >pacoloco-mirrorlist <<EOF
Server = https://mirrors.mit.edu/archlinux/\$repo/os/\$arch
Server = https://arch.mirror.constant.com/\$repo/os/\$arch
EOF
} && \
{ cat >arch-mirrorlist <<EOF
Server = http://pacoloco-test:9129/repo/archlinux/\$repo/os/\$arch
EOF
} && \
docker run -d --network paco-test --name pacoloco-test -v "$PWD:/data" -v "$PWD/pacoloco.yaml:/etc/pacoloco.yaml" ghcr.io/anatol/pacoloco@sha256:b93e352f8c4d34494df208158a05d5487da03d847467406ab35132197e9a2e9d && \
docker run --network paco-test --rm -it --name archlinux-test -v "$PWD/arch-mirrorlist:/etc/pacman.d/mirrorlist" archlinux:latest pacman -Syuv --noconfirm; \
echo -e "\n^^ ARCH ^^\nvv PACOLOCO vv\n" && \
docker logs pacoloco-testRepro cleanup: docker stop archlinux-test pacoloco-test ; docker rm archlinux-test pacoloco-test;
docker network rm paco-test;
cd /tmp; rm -rf /tmp/pactest |
|
Limiting pacoloco to 1 CPU seems to SOMETIMES solve the problem for me. In my docker case that's services:
pacoloco:
image: ghcr.io/anatol/pacoloco
cpuset: "1"Edited to add: It worked for a while, but now I am getting "has no urls" and a 404 on core.db, on a container that has been running with 1 cpu for a few days. |
Fix #92 by locking access to the Mirrorlist's Urls during refresh. I had the same problem on my deployed pacoloco and also successfully reproduced the issue using @chennin's script. Using this PR, both were unable to reproduce the error on my system. I'm new to go and it's concurrency model, but my guess is that the repo archlinux has no urls error happens due to a race condition in getMirrorlistURLs(): While the Mirrorlist File is read for the first time (e.g. because of a request for 'core.db') the URLs array is still empty but LastMirrorlistCheck is already set. When in this moment the function is called concurrently (e.g. because of a parallel download request for 'extra.db') it returns the still empty URLs array due to a recent LastMirrorlistCheck, instead of waiting for it to be populated. I used a mutex instead of setting the LastMirrorlistCheck after URLs is populated to block multiple goroutines from reading and parsing the same Mirrorlist simultaneously. However this doesn't fix any possible racing with reflector or similar. Closes #92
|
Sorry for bumping this old thread, but I just got into this issue too. The issue is still present and it is with this section: Pacoloco fetches an old .db file that somehow got left in cache. This has to be removed and not served anymore. I fixed it in my installation by removing .db files from cache. I guess it is something I forgot to cleanup after caching maybe! |
|
Did you change the mirrorlist between requests of the .db file, or did you see any connection errors, or how is this related to the closed issue? For .db-files pacoloco always connects to a mirror and does a "If-Modified-Since"-Check, so the file being in cache shouldn't be a problem. |
Hello. I have been getting this error for quite some time.
If I point my repos to pacoloco and then point pacoloco to my mirror list, only some of the DBs get downloaded.
On the other DBs I get:
pacoloco.go:168: repo archlinux has no urlsdocker-compose.yml
/pacoloco/pacoloco.yaml (mapped to /etc/pacoloco.yaml inside the container)
/etc/pacman.d/reflector-mirrorlist (mapped to /etc/mirrorlist inside the container)
/etc/pacman.d/mirrorlist
Server = http://localhost:9129/repo/archlinux/$repo/os/$arch/etc/pacman.conf
(the important parts)
Pacoloco logs (in this case core.db isn't being downloaded)
As you can see, it says
repo archlinux has no urlsPacman logs
It all works if I specify individual repos for each arch repo inside the pacoloco config, like this:
and then change my /etc/pacman.conf to have this instead
Thanks :)
The text was updated successfully, but these errors were encountered: