Skip to content

Deltametadata of repo md files

Jump to bottom
Jan Šilhan edited this page Mar 24, 2016 · 9 revisions

Goal of this project is to enable delta metadata downloads for DNF/PackageKit users. I.e. to not download whole new metadata files with each change. Zsync tool was identified as a stable tool used in production (Ubuntu images). Metadata files on server will contain generated checksums of the files (.zsync files). Client will download just differences of metadata he has on the system (no matter how old they are) and the up-to-data files on the server. This change would be forward compatible for current Fedora users. Zsync could be used on any file so it could theoretically generate .zsync files for .solv files from libsolv.

Short-term plan

Server side (proof of concept)

Client side (proof of concept)

Research

  • figure out how much bandwidth will be saved on client with day, 2 days, ..., week, ... 2 weeks old metadata

Long-term plan

Create actively developed zsync upstream

  • fork zsync-curl version
  • apply patches from other forks of no longer maintained zsync forks on github
  • separate zsync to library and client interface
  • design zsync library API so it could be used from librepo and createrepo_c
  • get rid of zlib bundling (optional, we can bundle it or include version from rsync)

Server side

  • it should be supported on Fedora infrastructure
    • createrepo_c should be used for generating of metadata
    • changes in createrepo_c
      • use by default gzip --rsyncable conversion
      • it should use zsync library to generate the .zsync files when --zsync option is set

Client side

  • changes in librepo
    • it should use zsync library API to set curl request

Things to consider

  • add into repomd.xml tag with zsync files with checksums and filesize
Clone this wiki locally