New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aptly mirror update with already existing .aptly/pool #33

Closed
erickeller opened this Issue Apr 15, 2014 · 12 comments

Comments

Projects
None yet
2 participants
@erickeller
Copy link

erickeller commented Apr 15, 2014

Hi Andrey,

aptly version: 0.4

I was experimenting with some aptly bash wrapper scripts:
mirror_create_ubu_clone.sh: cloning ubuntu with one mirror per separate component: precise-main precise-restricted, ..., precise-security-main, ...

starting with an empty .aptly/db directory.

  1 #!/bin/sh
  2 
  3 # depends on aptly (http://aptly.info)
  4 protocol="http://"
  5 url=ch.archive.ubuntu.com/ubuntu
  6 arch="i386,amd64"
  7 sections="main restricted universe multiverse"
  8 releases="precise precise-security precise-updates precise-backports"
  9 # create new precise repositories
 10 for release in ${releases}
 11 do
 12   for section in ${sections}
 13   do
 14     echo "aptly -architectures="${arch}"  mirror create ${release}-${session} ${protocol}${url} ${release} ${section}"
 15     aptly -architectures="${arch}"  mirror create ${release}-${section} ${protocol}${url} ${release} ${section}
 16   done
 17 done
 18 # update all precise repositories
 19 aptly mirror list | sed -e 's#.*\[\(.*\)\]:.*#\1#g' | grep -E 'precise.*' | xargs -n 1 aptly mirror update

I was hoping that aptly could use the already downloaded packages present in the .aptly/pool/

after some download time, the last line updating all the existing precise repositories does end.

after what I call aptly db cleanup:

aptly db cleanup 
Loading mirrors, local repos and snapshots...
Loading list of all packages...
Deleting unreferenced packages (11528)...
Building list of files referenced by packages...
Building list of files in package pool...
Deleting unreferenced files (11779)...
Disk space freed: 33.06 GiB...

using the same line 19 from the script: started to download 30 GiB again...

after the update for all mirror finished I restarted the command once again, and it still re downloaded some package since the size seems to be different!

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 15, 2014

I'm looking into that...

@smira smira added the bug label Apr 15, 2014

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 15, 2014

Could it be that some of the aptly mirror update commands have failed?

If so, mirror won't be considered updated and won't be saved. If you run aptly db cleanup immediately after that, it would remove packages and files that have been downloaded but not linked to the mirror (as updating has failed).

If you have run aptly mirror update without calling aptly db cleanup, it would have resumed download.

That's my theory, but it could explain what you have seen.

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 15, 2014

Mirror would be saved first time it finishes updating completely, so it's not a problem to run aptly db cleanup.

@erickeller

This comment has been minimized.

Copy link

erickeller commented Apr 15, 2014

I am re-updating it and send you the logs...

@erickeller

This comment has been minimized.

Copy link

erickeller commented Apr 16, 2014

Hi Andrey,

I tried with the latest 0.5~dev version of aptly, this is resulting in the same behavior...
with a lot of mismatch errors like:

ERROR: unable to update: download errors:
http://ch.archive.ubuntu.com/ubuntu/pool/main/libx/libxcb/libxcb_1.8.1-1ubuntu0.2.diff.gz: size check mismatch 183284 != 21004
http://ch.archive.ubuntu.com/ubuntu/pool/main/p/python-keyring/python-keyring_0.9.2.orig.tar.gz: size check mismatch 184320 != 31557
http://ch.archive.ubuntu.com/ubuntu/pool/main/libk/libkdcraw/libkdcraw_4.8.5-0ubuntu0.3.debian.tar.gz: size check mismatch 71680 != 12431
http://ch.archive.ubuntu.com/ubuntu/pool/main/s/sbsigntool/sbsigntool_0.6-0ubuntu1~12.04.1.debian.tar.gz: size check mismatch 20480 != 2823
http://ch.archive.ubuntu.com/ubuntu/pool/main/q/quagga/quagga_0.99.20.1-0ubuntu0.12.04.3.debian.tar.gz: size check mismatch 163840 != 37361
http://ch.archive.ubuntu.com/ubuntu/pool/main/l/language-selector/language-selector_0.79.4.tar.gz: size check mismatch 3317760 != 414803

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 16, 2014

With 0.4 (in your logs) there are the same errors. I'm debugging that to figure out where is the bug.

@erickeller

This comment has been minimized.

Copy link

erickeller commented Apr 16, 2014

0.4 and 0.5~dev produce the same mismatches size error... could you reproduce the problem? Should I keep one instance of the repository state for testing purpose?

regards

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 16, 2014

I've been able to reproduce that easily, thanks!

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 16, 2014

Actually, I've found bug. It's rather funny: when interacting with ch.archive.ubuntu.com, Go HTTP client is thinking server is sending gzip-compressed data to optimize transfer and decompressing it on the fly... That's why size mismatch error appears.

@erickeller

This comment has been minimized.

Copy link

erickeller commented Apr 16, 2014

that would make sense, I switch form de. to ch. mirror... That would explain why all of a sudden the mismatch occurs...

smira added a commit that referenced this issue Apr 16, 2014

@smira

This comment has been minimized.

Copy link
Member

smira commented Apr 16, 2014

I've pushed a fix, error seems to go away.

@smira smira added this to the v0.5 milestone Apr 16, 2014

smira added a commit to aptly-dev/aptly-dev.github.io that referenced this issue Apr 16, 2014

@erickeller

This comment has been minimized.

Copy link

erickeller commented Apr 16, 2014

thanks for the patch, it did also work for me!

@smira smira closed this Apr 17, 2014

sliverc pushed a commit to sliverc/aptly that referenced this issue May 22, 2017

Merge pull request aptly-dev#33 from sliverc/details_mirror_update
Task details should be empty for tasks which haven't stored details yet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment