Add zchunk support #92

jdieter · 2018-06-12T13:54:51Z

This request adds zchunk support to createrepo_c. It does now pass the python test cases, but there are some untested edge cases that will definitely need some work, namely using zchunk compression for anything other than primary, filelists and other.

The zchunk metadata is in addition to the standard gzip metadata and createrepo_c currently only creates zchunk versions primary, filelists and other.

Conan-Kudo · 2018-06-12T17:36:13Z

@jdieter So this means that if I added other files to the metadata using modifyrepo_c, we can't chunk via zck for now?

jdieter · 2018-06-13T06:04:45Z

That is correct. I don't think it will be too hard to fix that, especially now that zchunk has the ability to chunk using the buzhash algorithm as well as user-specified blocks, but I first wanted to get the big three (well, more the big two: primary and filelists) sorted.

Conan-Kudo · 2018-06-13T09:51:22Z

Well, as long as there is a plan for eventually supporting any arbitrary files manipulated by createrepo_c, this looks good to me.

Conan-Kudo

Overall, looks good.

A couple of things:

Can you regenerate the man pages to include your new options? The script to do so is in utils directory.
Could you add some tests specifically for zck stuff?

Conan-Kudo · 2018-06-13T09:53:24Z

src/cmd_parser.c

+    { "zck-filelists-dict", 0, 0, G_OPTION_ARG_FILENAME, &(_cmd_options.zck_filelists_dict),
+      "Compression dictionary to use for zchunk filelists file", "ZCK_FILELISTS_DICT" },
+    { "zck-other-dict", 0, 0, G_OPTION_ARG_FILENAME, &(_cmd_options.zck_other_dict),
+      "Compression dictionary to use for zchunk other file", "ZCK_OTHER_DICT" },


Are the dictionaries mandatory, or optional? My understand of zstd is that they make it better for compression, but aren't necessary.

They are optional, but make a significant difference in the size of the zchunked metadata. I've put together dictionaries for primary, filelists and other based on data from current Fedora releases, that I'd suggest packaging for Fedora/EPEL (if that's what our builders are on).

The dictionary is embedded in the zchunk file, so clients don't need a copy.

If the dictionary changes, there will be no common chunks between the pre-change and post-change files, so dictionary changes should happen a maximum of once a release (and probably aren't needed even that often)

Forgive me for being forgetful, but is there a script or something that people can use to generate dictionaries for their distributions' metadata?

I've got a small python script that splits the metadata into much smaller files and strips out any checksums (which by definition are unpredictable).

I ran the script over a couple of days worth of updates metadata, which produced thousands of files, and then, in the directory containing those files, ran:

zstd --train --dictID=0 *

The default dict size of roughly 100KB was fairly optimal for Fedora's metadata. Too small and you don't see the savings you'd expect. Too large and you're just wasting space.

The proposed Fedora dicts are available at https://www.jdieter.net/downloads/zchunk-dicts

I've gone ahead and updated the man page. I'll work on the tests but it might take a bit longer to get them done.

cgwalters · 2018-06-13T12:33:54Z

src/misc.c

@@ -1256,6 +1256,46 @@ cr_write_to_file(GError **err, gchar *filename, const char *format, ...)
    return ret;
 }

+gchar *cr_read_from_file(GError **err, gchar *filename, size_t *file_size) {


How is this different from g_file_get_contents()?

Wow, I feel stupid. It's exactly the same. I've just pushed a commit that remove cr_read_from_file and uses g_file_get_contents instead.

jdieter · 2018-06-22T11:15:10Z

@Conan-Kudo, I've added some tests (and found and fixed some problems in the process). Do you have some further suggestions as to which tests I should create?

jdieter · 2018-06-22T11:21:40Z

Just a heads up that the latest tests require zchunk-0.7.3 (available in the copr) in order to work, as it added the --stdout flag to unzck.

Conan-Kudo · 2018-06-22T15:49:25Z

No, it looks good to me.

Conan-Kudo

One small issue with the CMakeLists...

Conan-Kudo · 2018-06-22T16:48:27Z

CMakeLists.txt

@@ -34,7 +34,13 @@ find_package(LZMA REQUIRED)
 find_package(OpenSSL REQUIRED)
 find_package(Sqlite3 REQUIRED)
 find_package(ZLIB REQUIRED)
-
+find_package(PkgConfig REQUIRED)
+find_library(ZCHUNKLIB NAMES zck)


Could you write a FindZChunk.cmake similar to the FindGLIB2.cmake module and use that instead?

Ok, let me know if this works or you want something different

ignatenkobrain · 2018-07-30T15:25:48Z

CMakeLists.txt

@@ -34,6 +34,8 @@ find_package(LZMA REQUIRED)
 find_package(OpenSSL REQUIRED)
 find_package(Sqlite3 REQUIRED)
 find_package(ZLIB REQUIRED)
+find_package(PkgConfig REQUIRED)
+find_package(Zchunk REQUIRED)


why not to use pkg_check_modules() ? do not over-complicate this. please.

I didn't know about pkg_check_modules(). Thanks for the pointer!

ignatenkobrain · 2018-07-30T15:26:23Z

src/createrepo_c.c

+            cr_contentstat_free(pri_zck_stat, NULL);
+            cr_contentstat_free(fil_zck_stat, NULL);
+            cr_contentstat_free(oth_zck_stat, NULL);
+            g_free(pri_zck_filename);


use g_autofree

g_autofree() was added in glib-2.44. What's the minimum glib version that we need to support?

ignatenkobrain · 2018-07-30T15:28:00Z

src/compression_wrapper.c

+    int ret = CRE_OK;
+
+    assert(cr_file);
+    assert(!err || *err == NULL);


should be g_return_val_if_fail

Conan-Kudo

This change set looks good to me.

ignatenkobrain

generally looks good, but you miss include_directories(ZCK_INCLUDE_DIRS) or smth like that

jdieter · 2018-08-16T11:48:48Z

There are no zck-specific include directories. zck.h is in /usr/include. Should I still have include_directories(...)?

Conan-Kudo · 2018-08-16T11:59:14Z

@jdieter Yes, because it might be installed to non-standard paths by someone else.

jdieter · 2018-08-16T12:02:32Z

Ok, is this right?

Conan-Kudo · 2018-08-16T12:16:08Z

Looks good to me.

j-mracek · 2018-08-21T10:43:24Z

Please whats a status of availability of packages that fulfill requirement "pkg_check_modules(ZCK REQUIRED zck)" in Fedora/Centos/RHEL ... ?

Conan-Kudo · 2018-08-21T10:54:29Z

@j-mracek It's been available in Fedora, CentOS, RHEL, openSUSE, and Mageia for a couple of weeks now.

j-mracek · 2018-08-21T11:03:14Z

Please and whats a package name, because I was unable to find it?

Conan-Kudo · 2018-08-21T11:08:31Z

zchunk-devel is the package you need installed for this code. It's in EPEL for CentOS/RHEL 7. That package name will work across all distributions (though this will install lib64zck-devel on Mageia, and libzck-devel on openSUSE).

You can also request it by the pkgconfig name: sudo dnf install 'pkgconfig(zck)'.

jdieter · 2018-08-23T09:12:05Z

I've been thinking about how zchunk-enabled createrepo_c uses dictionaries, and the current method is a bit awkward. To use createrepo_c without dictionaries, you just run:

createrepo_c --zck ./

But to enable dictionary use, you need to run:

createrepo_c --zck --zck-primary-dict=foo-primary.dict --zck-filelists-dict=foo-filelists.dict --zck-other-dict=foo-other.dict ./

As we look at adding zchunk support to prestodelta.xml and other files generated by createrepo_c or added using modifyrepo_c, this starts to become very unwieldy. I suggest a different solution, passing a --zck-dict-dir argument that points to a directory containing foo.dict where foo is primary, filelists, other, prestodelta, etc. The command usage would look like:

createrepo_c --zck --zck-dict-dir=/usr/share/repo-dict

If there is no dictionary in the directory that matches the current file, then the file would be compressed without a dictionary.

Thoughts? Should I go ahead and implement this as part of this patch set?

Conan-Kudo · 2018-08-23T09:58:23Z

@jdieter That sounds fine to me, but I suggest that you document what it expects for that.

Conan-Kudo · 2018-08-31T13:35:30Z

Rather than having that structure exclusively, it's probably better to support both ways.

So, for now, I'd like to see this go in as-is, and we can add the --zck-dict-dir option later.

jdieter · 2018-09-01T13:16:03Z

@Conan-Kudo, if you feel strongly about it, we can keep the current options, but I feel like they're really bulky. I think --zck-dict-dir is a more elegant method, and I'd definitely prefer to drop the current options before people start depending on them.

jdieter · 2018-11-13T16:22:44Z

@dmach, now that F29's out the door, I'd love to start generating zchunk metadata in official Fedora. Is there anything I can do to help get this reviewed?

Conan-Kudo · 2018-11-19T20:03:02Z

@jdieter Can you please rebase this change? There are reported conflicts from the other merge earlier today.

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

--zck-filelists-dict and --zck-other-dict. Jonathan

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

argument checking in cmd_parser.c rather than createrepo_c.c Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

jdieter · 2018-11-19T21:29:39Z

Done! Thanks for the heads up!

Conan-Kudo · 2018-11-19T22:43:17Z

@dmach Can we please merge this now?

dmach · 2018-11-26T11:14:42Z

src/compression_wrapper.c

 #include <zlib.h>
 #include <bzlib.h>
 #include <lzma.h>
+#include <zck.h>


This needs to be wrapped into #ifdef WITH_ZCHUNK

Ok, this is done now. I did a test build on a system without zchunk, and it built and ran just fine.

Thanks so much for committing these patches!!!

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

dmach · 2018-11-26T14:58:35Z

I'll submit a patch to Fedora's dist-git, spec needs couple tweaks to handle bcond_with(out) zchunk

…ent#92

Conan-Kudo requested changes Jun 13, 2018

View reviewed changes

cgwalters reviewed Jun 13, 2018

View reviewed changes

jdieter mentioned this pull request Jun 13, 2018

Zchunk support for dnf rpm-software-management/dnf#1107

Closed

Conan-Kudo approved these changes Jun 22, 2018

View reviewed changes

Conan-Kudo requested changes Jun 22, 2018

View reviewed changes

jdieter force-pushed the zchunk branch from d96b051 to 36709e9 Compare July 8, 2018 19:16

ignatenkobrain reviewed Jul 30, 2018

View reviewed changes

src/compression_wrapper.c

int ret = CRE_OK;

assert(cr_file);

assert(!err || *err == NULL);

Copy link

ignatenkobrain Jul 30, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be g_return_val_if_fail

jdieter force-pushed the zchunk branch from 49640f1 to 41a2fa6 Compare August 8, 2018 13:44

Conan-Kudo approved these changes Aug 8, 2018

View reviewed changes

ignatenkobrain reviewed Aug 16, 2018

View reviewed changes

jdieter added 13 commits November 19, 2018 21:21

Add zchunk integration

1502428

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Use pkg_check_modules to simplify CMakeLists

19308be

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Fix C tests

d141139

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Add zchunk include directories

4133ec2

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Add --zck-dict-dir argument and remove --zck-primary-dict,

c4ede05

--zck-filelists-dict and --zck-other-dict. Jonathan

Make zchunk dependency optional

3541f50

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Pass python tests when not compiled with zchunk support

0f70585

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Set zchunk defaults, move zchunk compression options in list and do

378bdde

argument checking in cmd_parser.c rather than createrepo_c.c Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Add zchunk support to mergerepo

7747701

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Chunk packages based on srpm rather than rpm

11f94ac

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Stop double freeing zchunk filenames

451be9e

Free header checksum from open_stat

22e2175

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

Free dictionary file names when we're done with them

07e1235

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

jdieter force-pushed the zchunk branch from 9e7c830 to 07e1235 Compare November 19, 2018 21:25

dmach requested changes Nov 26, 2018

View reviewed changes

Add missing #ifdef around zck.h header

8a89384

Signed-off-by: Jonathan Dieter <jdieter@gmail.com>

dmach approved these changes Nov 26, 2018

View reviewed changes

dmach merged commit 93c9970 into rpm-software-management:master Nov 26, 2018

kontura added a commit to kontura/createrepo_c that referenced this pull request May 20, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

b1d4526

…ent#92

kontura added a commit to kontura/createrepo_c that referenced this pull request May 20, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

c4b7265

…ent#92

kontura added a commit to kontura/createrepo_c that referenced this pull request May 21, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

52d111b

…ent#92

kontura added a commit to kontura/createrepo_c that referenced this pull request May 21, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

d8a3ed1

…ent#92

kontura added a commit to kontura/createrepo_c that referenced this pull request May 21, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

6dfd377

…ent#92

kontura added a commit to kontura/createrepo_c that referenced this pull request May 21, 2019

Fix documentation inconsistency introduced in PR rpm-software-managem…

6725385

…ent#92

Conan-Kudo pushed a commit that referenced this pull request May 24, 2019

Fix documentation inconsistency introduced in PR #92

03bf000

Add zchunk support #92

Add zchunk support #92

Conversation

jdieter commented Jun 12, 2018

Conan-Kudo commented Jun 12, 2018

jdieter commented Jun 13, 2018

Conan-Kudo commented Jun 13, 2018

Conan-Kudo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdieter commented Jun 22, 2018

jdieter commented Jun 22, 2018

Conan-Kudo commented Jun 22, 2018

Conan-Kudo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Conan-Kudo left a comment

Choose a reason for hiding this comment

ignatenkobrain left a comment

Choose a reason for hiding this comment

jdieter commented Aug 16, 2018

Conan-Kudo commented Aug 16, 2018

jdieter commented Aug 16, 2018

Conan-Kudo commented Aug 16, 2018

j-mracek commented Aug 21, 2018 • edited

Conan-Kudo commented Aug 21, 2018

j-mracek commented Aug 21, 2018

Conan-Kudo commented Aug 21, 2018 • edited

jdieter commented Aug 23, 2018

Conan-Kudo commented Aug 23, 2018

Conan-Kudo commented Aug 31, 2018

jdieter commented Sep 1, 2018

jdieter commented Nov 13, 2018

Conan-Kudo commented Nov 19, 2018

jdieter commented Nov 19, 2018

Conan-Kudo commented Nov 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmach commented Nov 26, 2018

j-mracek commented Aug 21, 2018 •

edited

Conan-Kudo commented Aug 21, 2018 •

edited