use apt-ftparchive and reprepro to enable apt-pinning; #16001

jessfraz · 2015-09-02T01:28:19Z

this is the hackiest thing of my life but it works

I cloned our repo to new.dockerproject.org if you want to test before we make live ;)

jessfraz · 2015-09-02T01:51:37Z

 docker run --rm -it debian:stretch bash
root@f8c90c889e26:/# apt-get update
Get:1 http://httpredir.debian.org sid InRelease [239 kB]
Get:2 http://httpredir.debian.org sid/main amd64 Packages [10.1 MB]
Fetched 10.3 MB in 2s (3896 kB/s)   
Reading package lists... Done
root@f8c90c889e26:/# apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.XiQ6pI4Ln0 --no-auto-check-trustdb --trust-model always --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
gpg: requesting key 2C52609D from hkp server p80.pool.sks-keyservers.net

gpg: key 2C52609D: public key "Docker Release Tool (releasedocker) <docker@docker.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
root@f8c90c889e26:/# echo deb http://new.dockerproject.org/repo debian-stretch main> /etc/apt/sources.list.d/docker.list
root@f8c90c889e26:/# apt-get update
Ign http://new.dockerproject.org debian-stretch InRelease
Hit http://httpredir.debian.org sid InRelease               
Get:1 http://new.dockerproject.org debian-stretch Release.gpg [819 B]
Get:2 http://new.dockerproject.org debian-stretch Release [14.2 kB]
Get:3 http://httpredir.debian.org sid/main amd64 Packages [10.1 MB]
Get:4 http://new.dockerproject.org debian-stretch/main amd64 Packages [1587 B]
Fetched 10.1 MB in 2s (3938 kB/s)                                
Reading package lists... Done
root@f8c90c889e26:/# apt-cache madison docker-engine
docker-engine | 1.8.1-0~stretch | http://new.dockerproject.org/repo/ debian-stretch/main amd64 Packages
docker-engine | 1.8.0-0~stretch | http://new.dockerproject.org/repo/ debian-stretch/main amd64 Packages
docker-engine | 1.7.1-0~stretch | http://new.dockerproject.org/repo/ debian-stretch/main amd64 Packages
docker-engine | 1.7.0-0~stretch | http://new.dockerproject.org/repo/ debian-stretch/main amd64 Packages

yayyyy

jessfraz · 2015-09-02T04:39:31Z

also new feature:

root@3acce655f5e7:/# apt-file update
Downloading complete file http://httpredir.debian.org/debian/dists/sid/main/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 29.5M  100 29.5M    0     0  6360k      0  0:00:04  0:00:04 --:--:-- 7536k
Downloading complete file http://new.dockerproject.org/repo/dists/debian-stretch/main/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   678  100   678    0     0   3746      0 --:--:-- --:--:-- --:--:--  3766
Downloading complete file http://new.dockerproject.org/repo/dists/debian-stretch/testing/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   687  100   687    0     0   3404      0 --:--:-- --:--:-- --:--:--  3417
Downloading complete file http://new.dockerproject.org/repo/dists/debian-stretch/experimental/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   719  100   719    0     0   3594      0 --:--:-- --:--:-- --:--:--  3613
root@3acce655f5e7:/# apt-file search /usr/bin/docker
docker-compose: /usr/bin/docker-compose
docker-engine: /usr/bin/docker
docker.io: /usr/bin/docker
root@3acce655f5e7:/#

cpuguy83 · 2015-09-02T13:02:30Z

hack/make/release-deb

@@ -35,6 +35,51 @@ if [[ ! -f "$APTDIR/conf/distributions" ]]; then
 	done > "$APTDIR/conf/distributions"
 fi

+# create/update distributions file
+if [[ ! -f "$APTDIR/conf/apt-ftparchive.conf" ]]; then


Just going to nitpick to see if @tianon corrects me:)

I think this only needs a single bracket

cpuguy83 · 2015-09-02T13:04:10Z

Pretty!

jessfraz · 2015-09-02T16:16:33Z

we could probably remove reprepro all together and just use apt-ftparchive but this is good for now,, then we can have maybe followup PR

tianon · 2015-09-02T16:22:36Z

hack/make/release-deb

+		cat <<-EOF
+		Tree "dists/${suite}" {
+			Sections "main testing experimental";
+			Architectures "i386 amd64";


Do you think we should put these fields into an array at the top of the file since they're used multiple places? Then we could use for arch in "${arches[@]}"; do ... for iterating and ${arches[*]} for printing space-separated.

Maybe the sections also?

I guess we don't use the sections explicitly in nearly as many places as the arches.

tianon · 2015-09-02T16:24:28Z

hack/make/release-deb

+
+
+# run the apt-ftparchive commands so we can have pinning
+apt-ftparchive generate $APTDIR/conf/apt-ftparchive.conf


moar quotes 😄

paultag · 2015-09-02T16:26:54Z

Can't wait to check this out! I'll pull this down in a sec later today :D
On Sep 1, 2015 9:29 PM, "Jessie Frazelle" notifications@github.com wrote:

ping @tianon https://github.com/tianon pinng @paultag
https://github.com/paultag

this is the hackiest thing of my life but it works

I cloned our repo to new.dockerproject.org if you want to test before we

make live ;)

You can view, comment on, or merge this pull request online at:

#16001
Commit Summary

use apt-ftparchive and reprepro to enable apt-pinning;

File Changes

M hack/make/release-deb
https://github.com/docker/docker/pull/16001/files#diff-0 (77)

Patch Links:

https://github.com/docker/docker/pull/16001.patch

https://github.com/docker/docker/pull/16001.diff

—
Reply to this email directly or view it on GitHub
#16001.

tianon · 2015-09-02T16:27:51Z

Yeah, it definitely feels a little bit like we're abusing reprepro, but it solves the problem well enough without having to tweak dak, so I'm +1 (especially since @paultag pointed out that launchpad and sbuild both use apt-ftparchive, so it's not like it's way out there on the esoteric tools scale).

The logic seems sane enough to me, although I think @paultag will probably have more direct feedback he'll be able to give. ❤️

Signed-off-by: Jessica Frazelle <acidburn@docker.com>

jessfraz · 2015-09-02T16:35:59Z

updated! and yayyyy! yeah it feel like we are just using reprepro to put
the deb into the pool, (because we are) hehe but we can remove it completely

On Wed, Sep 2, 2015 at 9:29 AM, Tianon Gravi notifications@github.com
wrote:

Yeah, it definitely feels a little bit like we're abusing reprepro, but
it solves the problem well enough without having to tweak dak, so I'm +1
(especially since @paultag https://github.com/paultag pointed out that
launchpad and sbuild both use apt-ftparchive, so it's not like it's way
out there on the esoteric tools scale).

The logic seems sane enough to me, although I think @paultag
https://github.com/paultag will probably have more direct feedback
he'll be able to give. [image: ❤️]

—
Reply to this email directly or view it on GitHub
#16001 (comment).

jessfraz · 2015-09-02T16:37:53Z

also yeah I wanted something without having to add another tool or
something large like dak and we already had apt-ftparchive in the container
;) from apt-utils

On Wed, Sep 2, 2015 at 9:35 AM, Jessie Frazelle jess@docker.com wrote:

updated! and yayyyy! yeah it feel like we are just using reprepro to put
the deb into the pool, (because we are) hehe but we can remove it completely

On Wed, Sep 2, 2015 at 9:29 AM, Tianon Gravi notifications@github.com
wrote:

Yeah, it definitely feels a little bit like we're abusing reprepro, but
it solves the problem well enough without having to tweak dak, so I'm +1
(especially since @paultag https://github.com/paultag pointed out that
launchpad and sbuild both use apt-ftparchive, so it's not like it's way
out there on the esoteric tools scale).

The logic seems sane enough to me, although I think @paultag
https://github.com/paultag will probably have more direct feedback
he'll be able to give. [image: ❤️]

—
Reply to this email directly or view it on GitHub
#16001 (comment).

phobologic · 2015-09-02T18:57:09Z

This would make my life a lot easier - thanks for doing this @jfrazelle, can't wait to see it go live.

paultag · 2015-09-02T19:04:31Z

This seems like a totally sensible changeset -- apt-ftparchive is a totally supported and rockin' tool. The changes made look totally sensible enough, and I think @jfrazelle 's gut that y'all can remove the reprepro dep seem entirely well founded to me.

Seems like a great set of changes. Nice! 🎊 👍 here.

jessfraz · 2015-09-02T21:51:35Z

woo hoo we should get this in for 1.8.2, ping @calavera @icecrime 😇

calavera · 2015-09-03T09:18:50Z

LGTM. Merging and cherry picking into #15809

use apt-ftparchive and reprepro to enable apt-pinning;

After docker 1.7.0 we (Remind) have seen this issue pop up: moby/moby#13914 Now that this is closed: moby/moby#16001 We can use the new docker-engine packages, but with an old version. I'm moving the empire_ami back to docker 1.7.0 till 13914 above is fixed. As well, this should fix ECS stats - we just needed a bunch of volumes, per: aws/amazon-ecs-agent#174 Doing this internally @ Remind fixed this.

GordonTheTurtle added the status/0-triage label Sep 2, 2015

jessfraz force-pushed the apt-repo-pinning branch from e227533 to 82e69ae Compare September 2, 2015 04:21

cpuguy83 added status/2-code-review and removed status/0-triage labels Sep 2, 2015

cpuguy83 reviewed Sep 2, 2015
View reviewed changes

tianon reviewed Sep 2, 2015
View reviewed changes

jessfraz force-pushed the apt-repo-pinning branch from 82e69ae to 2467b65 Compare September 2, 2015 16:24

tianon reviewed Sep 2, 2015
View reviewed changes

use apt-ftparchive and reprepro to enable apt-pinning;

12a71c8

Signed-off-by: Jessica Frazelle <acidburn@docker.com>

jessfraz force-pushed the apt-repo-pinning branch from 2467b65 to 12a71c8 Compare September 2, 2015 16:32

jessfraz added this to the 1.8.2 milestone Sep 2, 2015

calavera added a commit that referenced this pull request Sep 3, 2015

Merge pull request #16001 from jfrazelle/apt-repo-pinning

d9065fc

use apt-ftparchive and reprepro to enable apt-pinning;

calavera merged commit d9065fc into moby:master Sep 3, 2015

jessfraz deleted the apt-repo-pinning branch September 3, 2015 15:16

thaJeztah mentioned this pull request Sep 3, 2015

Please don't remove old packages from apt repo #13191

Closed

phobologic mentioned this pull request Oct 11, 2015

Lock to docker 1.7.0, fix ecs tcs mounts remind101/empire_ami#6

Merged

leedm777 mentioned this pull request Jan 25, 2016

Install a specific version nodesource/distributions#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use apt-ftparchive and reprepro to enable apt-pinning; #16001

use apt-ftparchive and reprepro to enable apt-pinning; #16001

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

cpuguy83 Sep 2, 2015

cpuguy83 commented Sep 2, 2015

jessfraz commented Sep 2, 2015

tianon Sep 2, 2015

tianon Sep 2, 2015

tianon Sep 2, 2015

tianon Sep 2, 2015

paultag commented Sep 2, 2015

make live ;)

tianon commented Sep 2, 2015

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

phobologic commented Sep 2, 2015

paultag commented Sep 2, 2015

jessfraz commented Sep 2, 2015

calavera commented Sep 3, 2015



		# run the apt-ftparchive commands so we can have pinning
		apt-ftparchive generate $APTDIR/conf/apt-ftparchive.conf

use apt-ftparchive and reprepro to enable apt-pinning; #16001

use apt-ftparchive and reprepro to enable apt-pinning; #16001

Conversation

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

cpuguy83 Sep 2, 2015

Choose a reason for hiding this comment

cpuguy83 commented Sep 2, 2015

jessfraz commented Sep 2, 2015

tianon Sep 2, 2015

Choose a reason for hiding this comment

tianon Sep 2, 2015

Choose a reason for hiding this comment

tianon Sep 2, 2015

Choose a reason for hiding this comment

tianon Sep 2, 2015

Choose a reason for hiding this comment

paultag commented Sep 2, 2015

make live ;)

tianon commented Sep 2, 2015

jessfraz commented Sep 2, 2015

jessfraz commented Sep 2, 2015

phobologic commented Sep 2, 2015

paultag commented Sep 2, 2015

jessfraz commented Sep 2, 2015

calavera commented Sep 3, 2015