New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish multiple snapshots to the same prefix/distribution ; sharing the same `/pool` #196

Open
kumy opened this Issue Feb 8, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@kumy

kumy commented Feb 8, 2015

I know we're warned about such a config, but it could have advantages.

"It is not allowed to publish two repositories or snapshots to the same prefix and distribution."

(Here we used to run reprepro for our internal mirrors. And we're planning moving to aptly)

We were able to provide daily snapshots of mirrors, each of them sharing the same /pool.

$ reprepro gensnapshot trusty  20150125
$ reprepro gensnapshot trusty  20150126
$ reprepro gensnapshot trusty  20150127

$tree -d dists
dists/
`-- trusty
    |-- main
    |   `-- binary-amd64
    `-- snapshots
        |-- 20150125
        |   `-- main
        |       `-- binary-amd64
        |-- 20150126
        |   `-- main
        |       `-- binary-amd64
        `-- 20150127
            `-- main
                `-- binary-amd64

I've found that I can publish daily snapshots in different subdirs, ie:

aptly publish snapshot  -component="main" ubuntu-security-trusty_20150207 ubuntu-security/snapshots/20150207
aptly publish snapshot  -component="main" ubuntu-security-trusty_20150208 ubuntu-security/snapshots/20150208

But this has (for us) the real disadvantage of multiplying /pool directories.

What's not "cool" with that is, that we use caching proxies.
Many servers depends on different snapshots.
Caching is not efficient, as the storing hash key is the .deb full path.

If all servers download, let's say the same openssl_[...].deb from differents snapshots, then we'll not be able to serve cached content from the 1st request, we'll cache multiple times the same .deb (as of md5 speaking).

I tried to "play" with the -distribution argument, but no success

$ aptly publish snapshot -distribution="trusty/snapshots/20150207" -component="main" ubuntu-security-trusty_20150208 ubuntu-security                      
ERROR: unable to publish: invalid distribution trusty/snapshots/20150207, '/' is not allowed

Something like that seem to be related to issue #115

Is there another method for doing such thing ?
Do you think it could be implemented ?
How could I help you ?


Reprepro doc on gensnapshot:

 gensnapshot codename directoryname

 Generate a snapshot of the distribution specified by codename in the directory dists/codename/snapshots/directoryname/ and reference all needed files in the pool as needed by that.

aptly 0.9~dev+180+g7925af9

@kumy

This comment has been minimized.

Show comment
Hide comment
@kumy

kumy Feb 8, 2015

As an alternative we could also publish daily snapshots as:

$ aptly publish snapshot -distribution="trusty-snapshots-20150207" -component="main" ubuntu-security-trusty_20150207 ubuntu-security
$ aptly publish snapshot -distribution="trusty-snapshots-20150208" -component="main" ubuntu-security-trusty_20150208 ubuntu-security

tree -d dists/
dists/
|-- trusty
|   `-- main
|       |-- binary-all
|       |-- binary-amd64
|       `-- binary-i386
|-- trusty-snapshots-20150207
|   `-- main
|       |-- binary-all
|       |-- binary-amd64
|       `-- binary-i386
`-- trusty-snapshots-20150208
    `-- main
        |-- binary-all
        |-- binary-amd64
        `-- binary-i386

kumy commented Feb 8, 2015

As an alternative we could also publish daily snapshots as:

$ aptly publish snapshot -distribution="trusty-snapshots-20150207" -component="main" ubuntu-security-trusty_20150207 ubuntu-security
$ aptly publish snapshot -distribution="trusty-snapshots-20150208" -component="main" ubuntu-security-trusty_20150208 ubuntu-security

tree -d dists/
dists/
|-- trusty
|   `-- main
|       |-- binary-all
|       |-- binary-amd64
|       `-- binary-i386
|-- trusty-snapshots-20150207
|   `-- main
|       |-- binary-all
|       |-- binary-amd64
|       `-- binary-i386
`-- trusty-snapshots-20150208
    `-- main
        |-- binary-all
        |-- binary-amd64
        `-- binary-i386

@kumy kumy changed the title from Publish multiple snapshots to the same endpoint/distribution ; sharing the same `/pool` to Publish multiple snapshots to the same prefix/distribution ; sharing the same `/pool` Feb 9, 2015

@smira

This comment has been minimized.

Show comment
Hide comment
@smira

smira Feb 9, 2015

Member

@kumy in short aptly doesn't support distribution names with / in it. All published repositories under the same prefix share pool directory.

The difficulty with / in distribution names is possible complexity with directory handling for such case. Most probably it is possible to implement that, but I need to investigate this case

Member

smira commented Feb 9, 2015

@kumy in short aptly doesn't support distribution names with / in it. All published repositories under the same prefix share pool directory.

The difficulty with / in distribution names is possible complexity with directory handling for such case. Most probably it is possible to implement that, but I need to investigate this case

@kumy

This comment has been minimized.

Show comment
Hide comment
@kumy

kumy Feb 25, 2015

The alternative method I proposed doesn't work with debootstrap

$ fakeroot debootstrap trusty-snapshot-20150209 mytest http://localhost/distrib/ubuntu/
E: No such script: /usr/share/debootstrap/scripts/trusty-snapshot-20150209

kumy commented Feb 25, 2015

The alternative method I proposed doesn't work with debootstrap

$ fakeroot debootstrap trusty-snapshot-20150209 mytest http://localhost/distrib/ubuntu/
E: No such script: /usr/share/debootstrap/scripts/trusty-snapshot-20150209
@evrardjp

This comment has been minimized.

Show comment
Hide comment
@evrardjp

evrardjp Jan 26, 2017

This is also what we are looking for.

Is there a way to do this?

I investigated the use of components for this, but then if a snapshot is bad, deleting experience is horrible:

  • deleting only a component of a publication is not possible, swapping is painful because we have to move with an intermediate snapshot.
  • re-creating a new component on top of a current publication is painful (you cannot have aptly publish ... -component=new-snapshot-version ... because it will consider the distro is already published.

It seems impossible to have a versioned repo that way.

evrardjp commented Jan 26, 2017

This is also what we are looking for.

Is there a way to do this?

I investigated the use of components for this, but then if a snapshot is bad, deleting experience is horrible:

  • deleting only a component of a publication is not possible, swapping is painful because we have to move with an intermediate snapshot.
  • re-creating a new component on top of a current publication is painful (you cannot have aptly publish ... -component=new-snapshot-version ... because it will consider the distro is already published.

It seems impossible to have a versioned repo that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment