New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide rsync access to the package repo #4020

Closed
purcell opened this Issue Jul 2, 2016 · 17 comments

Comments

Projects
None yet
4 participants
@purcell
Member

purcell commented Jul 2, 2016

This would allow people to efficiently mirror our packages.

See #4012

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 17, 2016

Member

Partially done - see #4042 (comment)

Member

purcell commented Jul 17, 2016

Partially done - see #4042 (comment)

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 17, 2016

Member

Steps to enable on our server:

Create /etc/rsyncd.conf:

max connections = 1
log file = /var/log/rsync.log
timeout = 300
[packages]
comment = MELPA packages
path = /home/melpa/melpa/packages
read only = yes
list = yes
uid = melpa
gid = melpa
list = yes

Edit /etc/default/rsync and enable, then service rsync start seems to do the job. :-)

Member

purcell commented Jul 17, 2016

Steps to enable on our server:

Create /etc/rsyncd.conf:

max connections = 1
log file = /var/log/rsync.log
timeout = 300
[packages]
comment = MELPA packages
path = /home/melpa/melpa/packages
read only = yes
list = yes
uid = melpa
gid = melpa
list = yes

Edit /etc/default/rsync and enable, then service rsync start seems to do the job. :-)

@cpitclaudel

This comment has been minimized.

Show comment
Hide comment
@cpitclaudel

cpitclaudel Jul 17, 2016

Contributor

(any reason to not use chroot = yes and run under uid = nobody gid = nogroup?)

Contributor

cpitclaudel commented Jul 17, 2016

(any reason to not use chroot = yes and run under uid = nobody gid = nogroup?)

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 17, 2016

Member

(any reason to not use chroot = yes and run under uid = nobody gid = nogroup?)

Re. chroot, no - we should do that. But for the uid/gid, we're creating some files that are not world-readable, and this was the easy fix because I don't have time right now to fix the root problem.

Member

purcell commented Jul 17, 2016

(any reason to not use chroot = yes and run under uid = nobody gid = nogroup?)

Re. chroot, no - we should do that. But for the uid/gid, we're creating some files that are not world-readable, and this was the easy fix because I don't have time right now to fix the root problem.

@cpitclaudel

This comment has been minimized.

Show comment
Hide comment
@cpitclaudel

cpitclaudel Jul 17, 2016

Contributor

Ok, neat :)

Contributor

cpitclaudel commented Jul 17, 2016

Ok, neat :)

xuchunyang added a commit to emacs-china/elpa that referenced this issue Jul 17, 2016

Mirror MELPA via Rsync
MELPA now provide rsync access, so use it

see melpa/melpa#4020

[ci skip]
@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 21, 2016

Member

/cc @xuchunyang

The inconsistent permissions in our .el files are a result of inconsistent permissions on the corresponding files in upstream git repositories. e.g. In the zones package, zone.el is readable only by its owner.

To fix this, we would need to normalise the ownership of all files when we copy them into packages, which could potentially get tricky, because some packages include executable scripts.

Not sure of the best fix, to be honest.

Member

purcell commented Jul 21, 2016

/cc @xuchunyang

The inconsistent permissions in our .el files are a result of inconsistent permissions on the corresponding files in upstream git repositories. e.g. In the zones package, zone.el is readable only by its owner.

To fix this, we would need to normalise the ownership of all files when we copy them into packages, which could potentially get tricky, because some packages include executable scripts.

Not sure of the best fix, to be honest.

@cpitclaudel

This comment has been minimized.

Show comment
Hide comment
@cpitclaudel

cpitclaudel Jul 21, 2016

Contributor

Do you actually need to do anything? It's trivial to reset permissions while rsync-ing.

Contributor

cpitclaudel commented Jul 21, 2016

Do you actually need to do anything? It's trivial to reset permissions while rsync-ing.

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 21, 2016

Member
Member

purcell commented Jul 21, 2016

@xuchunyang

This comment has been minimized.

Show comment
Hide comment
@xuchunyang

xuchunyang Jul 22, 2016

Contributor

It looks like the modification timestamp of all the files served by rsync is also updated after melpa rebuilds packages

$ rsync --list-only /var/elpa-packages/melpa/0blayout-20151021.349.el 
-rw-r--r--          5,315 2016/07/22 11:48:32 0blayout-20151021.349.el

then when I sync with melpa via rsync every day, rsync re-downloads (though I guess rsync only downloads the difference but it still needs time & network resource to compute the difference firstly) all the files since rsync thinks two files are different if they have different modification timestamp.

One possible solution is use the --size-only to inhibit checking the modification timestamp, but this is Inaccurate, because it is possible that two different files have the save size.

Contributor

xuchunyang commented Jul 22, 2016

It looks like the modification timestamp of all the files served by rsync is also updated after melpa rebuilds packages

$ rsync --list-only /var/elpa-packages/melpa/0blayout-20151021.349.el 
-rw-r--r--          5,315 2016/07/22 11:48:32 0blayout-20151021.349.el

then when I sync with melpa via rsync every day, rsync re-downloads (though I guess rsync only downloads the difference but it still needs time & network resource to compute the difference firstly) all the files since rsync thinks two files are different if they have different modification timestamp.

One possible solution is use the --size-only to inhibit checking the modification timestamp, but this is Inaccurate, because it is possible that two different files have the save size.

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 22, 2016

Member

Yes, that's by design: every time we build packages we overwrite them because if we change our build process, we want the packages to be updated. It's not a perfect system, but that's how it is.

As you say, rsync should still only transfer the modified content. Using --size-only would be the wrong solution in this case.

Member

purcell commented Jul 22, 2016

Yes, that's by design: every time we build packages we overwrite them because if we change our build process, we want the packages to be updated. It's not a perfect system, but that's how it is.

As you say, rsync should still only transfer the modified content. Using --size-only would be the wrong solution in this case.

@xuchunyang

This comment has been minimized.

Show comment
Hide comment
@xuchunyang

xuchunyang Jul 22, 2016

Contributor

Keeping the modification timestamp of the identical files can save lots of resource and time in rsync. to achieve this, melpa can use a separate directory for rsync, for example,

  for file in /path/to/melpa/package/*; do
      dest=/path/to/melpa/rsync/"$(basename $file)"
      if ! [ -f "$dest" ] || ! cmp "$file" "$dest"; then
          cp -v "$file" "$dest"
          chmod 644 "$dest"
      fi
  done

As you say, rsync should still only transfer the modified content.

During rsync, the content of most files is unchanged. Figuring out which part is modified itself is not a trivial task and also can be avoided for unchanged files. If we sustain the modification timestamp for the identical files, rsync will simply skip these file.

Contributor

xuchunyang commented Jul 22, 2016

Keeping the modification timestamp of the identical files can save lots of resource and time in rsync. to achieve this, melpa can use a separate directory for rsync, for example,

  for file in /path/to/melpa/package/*; do
      dest=/path/to/melpa/rsync/"$(basename $file)"
      if ! [ -f "$dest" ] || ! cmp "$file" "$dest"; then
          cp -v "$file" "$dest"
          chmod 644 "$dest"
      fi
  done

As you say, rsync should still only transfer the modified content.

During rsync, the content of most files is unchanged. Figuring out which part is modified itself is not a trivial task and also can be avoided for unchanged files. If we sustain the modification timestamp for the identical files, rsync will simply skip these file.

@cpitclaudel

This comment has been minimized.

Show comment
Hide comment
@cpitclaudel

cpitclaudel Jul 22, 2016

Contributor

@xuchunyang Do you have evidence that this would be significantly faster? rsync is pretty efficient at comparing differences.

Contributor

cpitclaudel commented Jul 22, 2016

@xuchunyang Do you have evidence that this would be significantly faster? rsync is pretty efficient at comparing differences.

@xuchunyang

This comment has been minimized.

Show comment
Hide comment
@xuchunyang

xuchunyang Jul 22, 2016

Contributor

@cpitclaudel I don't have evidence actually, and after doing some testing, it looks you are right. That's my guess of the cause of the following issue.

The command we are using to sync melpa is:

rsync -avz --delete --progress rsync://melpa.org/packages/ /var/elpa-packages/melpa

(only rsync can change the content of /var/elpa-packages/melpa/), during every rsync per a few hours I often notice something like

kanji-mode-20150202.25.tar
      2,010,407   5%  109.68kB/s    0:04:51

and rsync does hang here for a while (though maybe less than 4:51). It won't occur if I run the rsync command again immediately. Except the timestamp of kanji-mode-20150202.25.tar, I can't figure out what is changed in this file.

Contributor

xuchunyang commented Jul 22, 2016

@cpitclaudel I don't have evidence actually, and after doing some testing, it looks you are right. That's my guess of the cause of the following issue.

The command we are using to sync melpa is:

rsync -avz --delete --progress rsync://melpa.org/packages/ /var/elpa-packages/melpa

(only rsync can change the content of /var/elpa-packages/melpa/), during every rsync per a few hours I often notice something like

kanji-mode-20150202.25.tar
      2,010,407   5%  109.68kB/s    0:04:51

and rsync does hang here for a while (though maybe less than 4:51). It won't occur if I run the rsync command again immediately. Except the timestamp of kanji-mode-20150202.25.tar, I can't figure out what is changed in this file.

@xuchunyang

This comment has been minimized.

Show comment
Hide comment
@xuchunyang

xuchunyang Jul 26, 2016

Contributor

Could you please provide rsync access to melpa-stable? We have switched to rsync for mirroring melpa since a few days ago (July 17), it is generally working well.

Contributor

xuchunyang commented Jul 26, 2016

Could you please provide rsync access to melpa-stable? We have switched to rsync for mirroring melpa since a few days ago (July 17), it is generally working well.

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell Jul 30, 2016

Member

Could you please provide rsync access to melpa-stable? We have switched to rsync for mirroring melpa since a few days ago (July 17), it is generally working well.

@xuchunyang done!

Member

purcell commented Jul 30, 2016

Could you please provide rsync access to melpa-stable? We have switched to rsync for mirroring melpa since a few days ago (July 17), it is generally working well.

@xuchunyang done!

@xuchunyang

This comment has been minimized.

Show comment
Hide comment
@xuchunyang

xuchunyang Jul 30, 2016

Contributor

Thanks. The rsync address is rsync://stable.melpa.org/packages/ and I can confirm it is working. We will switch to rsync for mirroring melpa-stable today or tomorrow.

Contributor

xuchunyang commented Jul 30, 2016

Thanks. The rsync address is rsync://stable.melpa.org/packages/ and I can confirm it is working. We will switch to rsync for mirroring melpa-stable today or tomorrow.

xuchunyang added a commit to emacs-china/elpa that referenced this issue Jul 30, 2016

Mirror MELPA-Stable via rsync
[ci skip]

Rsync access to MELPA-Stable is now available, see
melpa/melpa#4020 (comment)

@tarsius tarsius added the website label Aug 14, 2016

@tarsius tarsius added the feature label Mar 12, 2017

@purcell

This comment has been minimized.

Show comment
Hide comment
@purcell

purcell May 18, 2017

Member

Now long completed, so closing.

Member

purcell commented May 18, 2017

Now long completed, so closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment