repo_add_solv() doesn't load comps data correctly

[lukash]

Minimal test case:
comps.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE comps
  PUBLIC '-//Red Hat, Inc.//DTD Comps info//EN'
  'comps.dtd'>
<comps>
  <group>
    <id>core</id>
    <name>Core</name>
    <name xml:lang="cs">Úplný základ</name>
    <name xml:lang="de">Kern</name>
    <name xml:lang="ja">コア</name>
    <description>Smallest possible installation</description>
    <description xml:lang="cs">Nejmenší možná instalace</description>
    <description xml:lang="de">Kleinstmögliche Installation</description>
    <description xml:lang="ja">最小限のインストール</description>
    <default>false</default>
    <uservisible>false</uservisible>
    <display_order>1</display_order>
    <langonly>it</langonly>
    <packagelist>
      <packagereq type="mandatory">bash</packagereq>
      <packagereq type="mandatory">glibc</packagereq>
      <packagereq type="default">dnf</packagereq>
      <packagereq type="optional">dnf-plugins-core</packagereq>
      <packagereq requires="nonexistent" type="conditional">conditional</packagereq>
    </packagelist>
  </group>
</comps>

dump.c:
Loads the comps.xml into a Pool, writes a .solv file and also dumps a testcase for inspection. The testcase contains some data for the group but it seems to be completely missing at least the translations and probably other data is missing too.

#include <solv/repo.h>
#include <solv/repo_comps.h>
#include <solv/repo_solv.h>
#include <solv/repo_write.h>
#include <solv/testcase.h>
#include <solv/solv_xfopen.h>

int main() {
    Pool * pool = pool_create();
    if (!pool) printf("failed to create pool\n");
    
    Repo * repo = repo_create(pool, "test-repo");
    if (!repo) printf("failed to create repo\n");
    
    FILE * comps_file = solv_xfopen("comps.xml", "r");
    if (!comps_file) printf("failed to open comps file\n");
    
    if (repo_add_comps(repo, comps_file, 0) != 0) printf("failed to add comps\n");
    fclose(comps_file);
    
    Repodata * data = repo_id2repodata(repo, repo->nrepodata - 1);
    if (!data) printf("failed to get repodata\n");
    
    FILE * solv_file = solv_xfopen("solv.solv", "w");
    if (!solv_file) printf("failed to open solv file\n");

    if (repodata_write(data, solv_file) != 0) printf("failed to write solv\n");
    fclose(solv_file);

    Solver * solver = solver_create(pool);
    if (!testcase_write(solver, "dump-tc", 0, NULL, NULL)) printf("failed to write testcase: %s\n", pool_errstr(pool));
    solver_free(solver);

    return 0;
}

load.c:
Loads the written solv.solv and again dumps the testcase, in this case the testcase is almost empty.

#include <solv/repo.h>
#include <solv/repo_comps.h>
#include <solv/repo_solv.h>
#include <solv/repo_write.h>
#include <solv/testcase.h>
#include <solv/solv_xfopen.h>

int main() {
    Pool * pool = pool_create();
    if (!pool) printf("failed to create pool\n");
    
    Repo * repo = repo_create(pool, "test-repo");
    if (!repo) printf("failed to create repo\n");
    
    FILE * solv_file = solv_xfopen("solv.solv", "r");
    if (!solv_file) printf("failed to open solv file\n");
    
    if (repo_add_solv(repo, solv_file, 0) != 0) printf("failed to load solv\n");
    fclose(solv_file);
    
    Solver * solver = solver_create(pool);
    if (!testcase_write(solver, "load-tc", 0, NULL, NULL)) printf("failed to write testcase: %s\n", pool_errstr(pool));
    solver_free(solver);
    
    return 0;
}

All the files tarballed: comps.tar.gz

[mlschroe]

The testcase writer only dumps the data relevant for the solver, so I'm not surprised that all the comps specific data is missing.

[lukash]

Right, that explains why the data aren't in the testcase dump, and we actually know the translations and other data are loading correctly, so that was misleading from me, sorry. But it appears they are not being loaded from the solv.solv file, in which I can still see them when I dump it via solvdump:

repo_add_solv took 0 ms
repo size: 1 solvables
repo memory used: 0 K incore, 0 K idarray
pool contains 208 strings, 1 rels, string size is 3881
repo contains 1 repodata sections:

repodata 1 has 11 keys, 2 schemata
  repository:solvables (type repokey:type:flexarray size 1 storage 2)
  solvable:summary (type repokey:type:str size 0 storage 2)
  solvable:summary:cs (type repokey:type:str size 16 storage 3)
  solvable:summary:de (type repokey:type:str size 5 storage 3)
  solvable:summary:ja (type repokey:type:str size 7 storage 3)
  solvable:description (type repokey:type:str size 31 storage 3)
  solvable:description:cs (type repokey:type:str size 29 storage 3)
  solvable:description:de (type repokey:type:str size 30 storage 3)
  solvable:description:ja (type repokey:type:str size 31 storage 3)
  solvable:order (type repokey:type:str size 0 storage 2)
  solvable:langonly (type repokey:type:str size 0 storage 2)


repo 1 contains 1 solvables
repo start: 2 end: 3

solvable 1 (2):
solvable:summary: Core
solvable:summary:cs: Úplný základ
solvable:summary:de: Kern
solvable:summary:ja: コア
solvable:description: Smallest possible installation
solvable:description:cs: Nejmenší možná instalace
solvable:description:de: Kleinstmögliche Installation
solvable:description:ja: 最小限のインストール
solvable:order: 1
solvable:langonly: it

[mlschroe]

That's because you use repodata_write, not repo_write. So you also write non solving-related data, i.e. no dependencies, package name and the like.

[lukash]

Hmm. Sorry, I don't follow, I'm getting lost in the fine differences. We use repo_write() for the "main" .solv file and repodata_write() for the "extensions". FTR you can see our code for writing these here in the write_main() and write_ext() methods.

I've scoured the documentation but haven't found much on this, more details would be appreciated.

[mlschroe]

The comps data is not an "extension", the groups/environments are mapped to pseudo packages.

[lukash]

Ok, but what are the implications then? I think it's plain to see I don't understand what's going on. Could you provide more comprehensive explanation please?

Also, you say we're using the wrong _write() function, but I can see the "correct" (correct to the point "I see something there") data are being written in the .solv file, but they aren't being loaded.

[mlschroe]

I guess the data is dropped because the corresponding packages do not exist. The "comps" data is like the "primary" data, you can't load it as extension as it contains packages.

A repo consists of packages with dependencies plus one or more repodata areas that are supposed to extend existing packages (solvables in libsolv speech).

It's different for updateinfo, because here updates are not represented as packages. This could have been done for the comps data as well, but libsolv didn't do that because the idea at that time was to include comps in the package solving process. I.e. you could ask the solver to install a comps group.

[mlschroe]

OTOH you can make this work with loading the comps file as a pseudo extension. But you need to use repo_write for writing and you need to limit the solvables to the corresponding comps range.

[mlschroe]

(I.e. use the repowriter_set_solvablerange() function)

[mlschroe]

Oh wait, I confused updateinfo with deltainfo.

So it is the same as with updateinfo, where "patch:XXX" pseudo packages get created.

[mlschroe]

See the special casing of updateinfo in your SolvRepo::write_ext function. Note that it also calls repo_write, not repodata_write.

You really should switch to a repowriter here, that code is a horrible hack. And that main_end/main_nsolvables doesn't work if you have both comps and updateinfo data.

[lukash]

Thanks, yes, I'm trying to wrap my head around this and I'm looking at the updateinfo handling, it's ugly already and would get much worse with comps as another special case.

[mlschroe]

Why is the updateinfo/comps data stored as extension?

[lukash]

So how would I do this with the repowriter that would work for both updateinfo and comps. After loading each of those, I could store repo->nrepodata - 1 which will be my repodataid for the given pseudopackage set. Then I could use repowriter_set_repodatarange(writer, repodataid, repodataid + 1); to limit the range of solvables that are written? And that would be all I need, meaning the write_ext_updateinfo_filter() function we now have for the updateinfo (in the linked file) could also be dropped. Is this the right way to do it?

EDIT: That seems what repodata_write() is effectively doing so it's likely not correct. But both repo_add_comps() and repo_add_updateinfoxml() seem to be adding new repodata (via repo_add_repodata()), so I'm getting more confused. Is the difference just the REPOWRITER_NO_STORAGE_SOLVABLE flag? Is the concept such that each time I load (any) data, a new repodata are created and if the data being loaded contain solvables, they become part of the "main" data?

[lukash]

Why is the updateinfo/comps data stored as extension?

For comps I just did it in ignorance analogous to the other extensions. For updateinfo it's the way it's been done, I'm not sure anyone on the team currently knows any more details. For dnf 5 we'll certainly welcome any simplification.

Are you questioning why they have their own .solv cache file and why aren't they stored in the main cache? The only reason I can think of is if you want to load a repository without these data from the cache (we have flags for specifying the set of "extensions" to load for a given repo).

[lukash]

@mlschroe you're leaving me hanging. I've made some meager attempts in the meanwhile here: https://github.com/lukash/libdnf/commit/a8fe1af11cebc2667d6c6cfdfeae9873f5559f5e

But it's not doing what I expect it to be doing. I'd rather prefer not to spend hours or days with trial and error or analyzing libsolv sources. Given there's no documentation, please kindly provide me with enough information to solve this effectively.

[mlschroe]

That commit looks good to me, what exactly does not work? Just use dump_solv on the written solv file to check if the written data is missing something or contains excess data.

For updateinfo writing I would also limit the repodata range to the updateinfo repodata, otherwise you might get things like the deltarpm data.

And you also should set updateinfo_solvables_start/end when loading the updateinfo from the solv file.

[lukash]

On a simple test repo it seems to be including the main solvables in the solv dump:

original code solv dump

repo_add_solv took 0 ms
repo size: 4 solvables
repo memory used: 0 K incore, 0 K idarray
pool contains 237 strings, 29 rels, string size is 4026
repo contains 1 repodata sections:

repodata 1 has 30 keys, 8 schemata
  solvable:name (type repokey:type:id size 0 storage 1)
  solvable:arch (type repokey:type:id size 0 storage 1)
  solvable:evr (type repokey:type:id size 0 storage 1)
  solvable:vendor (type repokey:type:id size 0 storage 1)
  solvable:provides (type repokey:type:idarray size 8 storage 1)
  solvable:conflicts (type repokey:type:idarray size 18 storage 1)
  repository:solvables (type repokey:type:flexarray size 1 storage 2)
  solvable:patchcategory (type repokey:type:str size 0 storage 2)
  update:status (type repokey:type:id size 0 storage 2)
  solvable:summary (type repokey:type:str size 0 storage 2)
  update:severity (type repokey:type:id size 0 storage 2)
  update:reference:href (type repokey:type:str size 0 storage 2)
  update:reference:id (type repokey:type:str size 0 storage 2)
  update:reference:title (type repokey:type:str size 0 storage 2)
  update:reference:type (type repokey:type:id size 0 storage 2)
  update:reference (type repokey:type:flexarray size 0 storage 2)
  solvable:description (type repokey:type:str size 116 storage 3)
  update:module:name (type repokey:type:id size 0 storage 2)
  update:module:stream (type repokey:type:id size 0 storage 2)
  update:module:version (type repokey:type:id size 0 storage 2)
  update:module:context (type repokey:type:id size 0 storage 2)
  update:module:arch (type repokey:type:id size 0 storage 2)
  update:module (type repokey:type:flexarray size 0 storage 2)
  update:collection:name (type repokey:type:id size 0 storage 2)
  update:collection:evr (type repokey:type:id size 0 storage 2)
  update:collection:arch (type repokey:type:id size 0 storage 2)
  update:collection:filename (type repokey:type:str size 0 storage 2)
  update:collection (type repokey:type:flexarray size 0 storage 2)
  update:collectionlist (type repokey:type:flexarray size 0 storage 2)
  solvable:buildtime (type repokey:type:num size 0 storage 2)


repo 1 contains 4 solvables
repo start: 2 end: 6

solvable 1 (2):
solvable:name: patch:DNF-2019-1
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:DNF-2019-1 = 1
solvable:conflicts:
  pkg.x86_64 < 1.2-3
  pkg.noarch < 1.2-3
  filesystem.x86_64 < 3.9-2.fc29
  filesystem.noarch < 3.9-2.fc29
solvable:patchcategory: security
update:status: stable
solvable:summary: bugfix_A-1.0-1
update:severity: moderate
update:reference:
  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 1111
  update:reference:title: CVE-2999
  update:reference:type: cve

solvable:description: testing advisory 2019
update:module:
  update:module:name: perl-DBI
  update:module:stream: master
  update:module:version: 2
  update:module:context: 2a
  update:module:arch: x86_64

  update:module:name: ethereum
  update:module:stream: 2.0
  update:module:version: 2
  update:module:context: 3b
  update:module:arch: x86_64

update:collection:
  update:collection:name: pkg
  update:collection:evr: 1.2-3
  update:collection:arch: x86_64
  update:collection:filename: pkg-1.2-3.x86_64.rpm

  update:collection:name: filesystem
  update:collection:evr: 3.9-2.fc29
  update:collection:arch: x86_64
  update:collection:filename: filesystem-3.9-2.fc29.x86_64.rpm

update:collectionlist:
  update:module:
    update:module:name: perl-DBI
    update:module:stream: master
    update:module:version: 2
    update:module:context: 2a
    update:module:arch: x86_64
  
    update:module:name: ethereum
    update:module:stream: 2.0
    update:module:version: 2
    update:module:context: 3b
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: pkg
    update:collection:evr: 1.2-3
    update:collection:arch: x86_64
    update:collection:filename: pkg-1.2-3.x86_64.rpm
  
    update:collection:name: filesystem
    update:collection:evr: 3.9-2.fc29
    update:collection:arch: x86_64
    update:collection:filename: filesystem-3.9-2.fc29.x86_64.rpm
  

solvable:buildtime: 1550849401

solvable 2 (3):
solvable:name: patch:DNF-2020-1
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:DNF-2020-1 = 1
solvable:conflicts:
  wget.x86_64 < 1.19.5-5.fc29
  wget.noarch < 1.19.5-5.fc29
  yum.x86_64 < 3.4.3-0
  yum.noarch < 3.4.3-0
  bitcoin.x86_64 < 2.5-1
  bitcoin.noarch < 2.5-1
solvable:patchcategory: bugfix
update:status: stable
solvable:summary: bugfix_B-2.0-1
update:severity: critical
update:reference:
  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 2222
  update:reference:title: 222
  update:reference:type: bugzilla

  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 3333
  update:reference:title: CVE-2999
  update:reference:type: cve

solvable:description: testing advisory 2020
update:module:
  update:module:name: perl-DBI
  update:module:stream: master
  update:module:version: 2
  update:module:context: 2b
  update:module:arch: x86_64

  update:module:name: perl
  update:module:stream: master
  update:module:version: 1
  update:module:context: 1c
  update:module:arch: x86_64

update:collection:
  update:collection:name: wget
  update:collection:evr: 1.19.5-5.fc29
  update:collection:arch: x86_64
  update:collection:filename: wget-1.19.5-5.fc29.x86_64.rpm

  update:collection:name: yum
  update:collection:evr: 3.4.3-0
  update:collection:arch: x86_64
  update:collection:filename: yum-3.4.3-0.x86_64.rpm

  update:collection:name: bitcoin
  update:collection:evr: 2.5-1
  update:collection:arch: x86_64
  update:collection:filename: bitcoin-2.5-1.x86_64.rpm

update:collectionlist:
  update:module:
    update:module:name: perl-DBI
    update:module:stream: master
    update:module:version: 2
    update:module:context: 2b
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: wget
    update:collection:evr: 1.19.5-5.fc29
    update:collection:arch: x86_64
    update:collection:filename: wget-1.19.5-5.fc29.x86_64.rpm
  
    update:collection:name: yum
    update:collection:evr: 3.4.3-0
    update:collection:arch: x86_64
    update:collection:filename: yum-3.4.3-0.x86_64.rpm
  

  update:module:
    update:module:name: perl
    update:module:stream: master
    update:module:version: 1
    update:module:context: 1c
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: bitcoin
    update:collection:evr: 2.5-1
    update:collection:arch: x86_64
    update:collection:filename: bitcoin-2.5-1.x86_64.rpm
  

solvable:buildtime: 1329924602

solvable 3 (4):
solvable:name: patch:PKG-NEWER
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:PKG-NEWER = 1
solvable:conflicts:
  pkg.x86_64 < 4.0-1
  pkg.noarch < 4.0-1
solvable:patchcategory: enhancement
update:status: stable
solvable:summary: newer-version
update:severity: low
solvable:description: testing advisory with newer package
update:collection:
  update:collection:name: pkg
  update:collection:evr: 4.0-1
  update:collection:arch: x86_64
  update:collection:filename: pkg-4.0-1.x86_64.rpm

update:collectionlist:
  update:collection:
    update:collection:name: pkg
    update:collection:evr: 4.0-1
    update:collection:arch: x86_64
    update:collection:filename: pkg-4.0-1.x86_64.rpm
  

solvable:buildtime: 1329924602

solvable 4 (5):
solvable:name: patch:PKG-OLDER
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:PKG-OLDER = 1
solvable:conflicts:
  pkg.x86_64 < 0.1-1
  pkg.noarch < 0.1-1
solvable:patchcategory: bugfix
update:status: stable
solvable:summary: older-version
update:severity: low
solvable:description: testing advisory with older package
update:collection:
  update:collection:name: pkg
  update:collection:evr: 0.1-1
  update:collection:arch: x86_64
  update:collection:filename: pkg-0.1-1.x86_64.rpm

update:collectionlist:
  update:collection:
    update:collection:name: pkg
    update:collection:evr: 0.1-1
    update:collection:arch: x86_64
    update:collection:filename: pkg-0.1-1.x86_64.rpm
  

solvable:buildtime: 1329924602

my commit solv dump

repo_add_solv took 1 ms
repo size: 4 solvables
repo memory used: 1 K incore, 0 K idarray
pool contains 255 strings, 29 rels, string size is 4142
repo contains 1 repodata sections:

repodata 1 has 62 keys, 9 schemata
  solvable:name (type repokey:type:id size 0 storage 1)
  solvable:arch (type repokey:type:id size 0 storage 1)
  solvable:evr (type repokey:type:id size 0 storage 1)
  solvable:vendor (type repokey:type:id size 0 storage 1)
  solvable:provides (type repokey:type:idarray size 9 storage 1)
  solvable:obsoletes (type repokey:type:idarray size 2 storage 1)
  solvable:conflicts (type repokey:type:idarray size 14 storage 1)
  solvable:requires (type repokey:type:idarray size 4 storage 1)
  solvable:recommends (type repokey:type:idarray size 2 storage 1)
  solvable:suggests (type repokey:type:idarray size 2 storage 1)
  solvable:supplements (type repokey:type:idarray size 2 storage 1)
  solvable:enhances (type repokey:type:idarray size 2 storage 1)
  repository:solvables (type repokey:type:flexarray size 1 storage 2)
  repository:revision (type repokey:type:str size 0 storage 2)
  repository:repomd:type (type repokey:type:id size 0 storage 2)
  repository:repomd:checksum (type repokey:type:sha256 size 0 storage 2)
  repository:repomd:openchecksum (type repokey:type:sha256 size 0 storage 2)
  repository:repomd:location (type repokey:type:str size 0 storage 2)
  repository:repomd:timestamp (type repokey:type:num size 0 storage 2)
  repository:repomd:size (type repokey:type:num size 0 storage 2)
  repository:repomd (type repokey:type:flexarray size 0 storage 2)
  repository:timestamp (type repokey:type:num size 0 storage 2)
  solvable:checksum (type repokey:type:sha256 size 64 storage 3)
  solvable:summary (type repokey:type:str size 0 storage 2)
  solvable:description (type repokey:type:str size 68 storage 3)
  solvable:packager (type repokey:type:id size 0 storage 2)
  solvable:url (type repokey:type:str size 0 storage 2)
  solvable:buildtime (type repokey:type:num size 0 storage 2)
  solvable:installsize (type repokey:type:num size 0 storage 2)
  solvable:downloadsize (type repokey:type:num size 0 storage 2)
  solvable:mediafile (type repokey:type:void size 0 storage 2)
  solvable:license (type repokey:type:id size 0 storage 2)
  solvable:group (type repokey:type:id size 0 storage 2)
  solvable:buildhost (type repokey:type:str size 0 storage 2)
  solvable:sourcename (type repokey:type:void size 0 storage 2)
  solvable:sourceevr (type repokey:type:void size 0 storage 2)
  solvable:sourcearch (type repokey:type:constantid size 24 storage 2)
  solvable:headerend (type repokey:type:num size 0 storage 2)
  solvable:sourcename (type repokey:type:id size 0 storage 2)
  solvable:mediafile (type repokey:type:str size 0 storage 2)
  solvable:mediabase (type repokey:type:id size 0 storage 2)
  repository:filteredfilelist (type repokey:type:void size 0 storage 2)
  solvable:patchcategory (type repokey:type:str size 0 storage 2)
  update:status (type repokey:type:id size 0 storage 2)
  update:severity (type repokey:type:id size 0 storage 2)
  update:reference:href (type repokey:type:str size 0 storage 2)
  update:reference:id (type repokey:type:str size 0 storage 2)
  update:reference:title (type repokey:type:str size 0 storage 2)
  update:reference:type (type repokey:type:id size 0 storage 2)
  update:reference (type repokey:type:flexarray size 0 storage 2)
  update:module:name (type repokey:type:id size 0 storage 2)
  update:module:stream (type repokey:type:id size 0 storage 2)
  update:module:version (type repokey:type:id size 0 storage 2)
  update:module:context (type repokey:type:id size 0 storage 2)
  update:module:arch (type repokey:type:id size 0 storage 2)
  update:module (type repokey:type:flexarray size 0 storage 2)
  update:collection:name (type repokey:type:id size 0 storage 2)
  update:collection:evr (type repokey:type:id size 0 storage 2)
  update:collection:arch (type repokey:type:id size 0 storage 2)
  update:collection:filename (type repokey:type:str size 0 storage 2)
  update:collection (type repokey:type:flexarray size 0 storage 2)
  update:collectionlist (type repokey:type:flexarray size 0 storage 2)

repository:revision: 1550000000
repository:repomd:
  repository:repomd:type: primary
  repository:repomd:checksum: 5689e808317cf753b400c5a203b6cd45ffd340989f725c768721c560ee205dba (sha256)
  repository:repomd:openchecksum: 5689e808317cf753b400c5a203b6cd45ffd340989f725c768721c560ee205dba (sha256)
  repository:repomd:location: repodata/primary.xml
  repository:repomd:timestamp: 1597222003
  repository:repomd:size: 81503

  repository:repomd:type: filelists
  repository:repomd:checksum: 35476536de2e46856ee2f0272bac44ef3c366effa37c29322c344bef56168ce6 (sha256)
  repository:repomd:openchecksum: 35476536de2e46856ee2f0272bac44ef3c366effa37c29322c344bef56168ce6 (sha256)
  repository:repomd:location: repodata/filelists.xml
  repository:repomd:timestamp: 1597222003
  repository:repomd:size: 14137

  repository:repomd:type: other
  repository:repomd:checksum: 6b1629e202cf424f893dda1ddec703165234c22f0ad5a1657dc77b357db2d189 (sha256)
  repository:repomd:openchecksum: 6b1629e202cf424f893dda1ddec703165234c22f0ad5a1657dc77b357db2d189 (sha256)
  repository:repomd:location: repodata/other.xml
  repository:repomd:timestamp: 1597222003
  repository:repomd:size: 13799

  repository:repomd:type: updateinfo
  repository:repomd:checksum: 0bad6ed78db8e058e59eefddd20744d71219b736045b10821811180972f6d1fa (sha256)
  repository:repomd:openchecksum: 0bad6ed78db8e058e59eefddd20744d71219b736045b10821811180972f6d1fa (sha256)
  repository:repomd:location: repodata/updateinfo.xml
  repository:repomd:timestamp: 1597222003
  repository:repomd:size: 646

repository:timestamp: 1597222003
repository:filteredfilelist: (void)

repo 1 contains 4 solvables
repo start: 2 end: 6

solvable 1 (2):
solvable:name: pkg-libs
solvable:arch: x86_64
solvable:evr: 1:1.3-4
solvable:vendor: Vendor
solvable:provides:
  pkg-libs = 1:1.3-4
solvable:checksum: caa857c48130b4fdea3f7fa498da4324ae2ac00c8900d71c0eef0a90457636bd (sha256)
solvable:summary: Summary
solvable:description: Description
solvable:packager: Packager
solvable:url: http://example.com/
solvable:buildtime: 456
solvable:installsize: 222
solvable:downloadsize: 111
solvable:mediafile: (void)
solvable:license: License
solvable:group: Group
solvable:buildhost: Buildhost
solvable:sourcename: pkg
solvable:sourceevr: (void)
solvable:sourcearch: src
solvable:headerend: 22

solvable 2 (3):
solvable:name: unresolvable
solvable:arch: noarch
solvable:evr: 1:2-3
solvable:vendor: Vendor
solvable:provides:
  prv = 1:2-3
  unresolvable = 1:2-3
solvable:obsoletes:
  obs < 1:2
solvable:conflicts:
  con < 1:2
solvable:requires:
  req = 1:2-3
  solvable:prereqmarker
  prereq
solvable:recommends:
  rec
solvable:suggests:
  sug
solvable:supplements:
  sup
solvable:enhances:
  enh
solvable:checksum: acedd07199eae0111f1306b8b5d794a3358568feb0ccf0fd39bd8d815fd78ece (sha256)
solvable:summary: Summary
solvable:description: Description
solvable:packager: Packager
solvable:url: http://example.com/
solvable:buildtime: 456
solvable:installsize: 222
solvable:downloadsize: 111
solvable:mediafile: unresolvable-2-3.x86_64.rpm
solvable:mediabase: http://example.com/
solvable:license: License
solvable:group: Group
solvable:buildhost: Buildhost
solvable:sourcename: (void)
solvable:sourceevr: (void)
solvable:sourcearch: src
solvable:headerend: 22

solvable 3 (4):
solvable:name: patch:DNF-2019-1
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:DNF-2019-1 = 1
solvable:conflicts:
  pkg.x86_64 < 1.2-3
  pkg.noarch < 1.2-3
  filesystem.x86_64 < 3.9-2.fc29
  filesystem.noarch < 3.9-2.fc29
solvable:patchcategory: security
update:status: stable
solvable:summary: bugfix_A-1.0-1
update:severity: moderate
update:reference:
  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 1111
  update:reference:title: CVE-2999
  update:reference:type: cve

solvable:description: testing advisory 2019
update:module:
  update:module:name: perl-DBI
  update:module:stream: master
  update:module:version: 2
  update:module:context: 2a
  update:module:arch: x86_64

  update:module:name: ethereum
  update:module:stream: 2.0
  update:module:version: 2
  update:module:context: 3b
  update:module:arch: x86_64

update:collection:
  update:collection:name: pkg
  update:collection:evr: 1.2-3
  update:collection:arch: x86_64
  update:collection:filename: pkg-1.2-3.x86_64.rpm

  update:collection:name: filesystem
  update:collection:evr: 3.9-2.fc29
  update:collection:arch: x86_64
  update:collection:filename: filesystem-3.9-2.fc29.x86_64.rpm

update:collectionlist:
  update:module:
    update:module:name: perl-DBI
    update:module:stream: master
    update:module:version: 2
    update:module:context: 2a
    update:module:arch: x86_64
  
    update:module:name: ethereum
    update:module:stream: 2.0
    update:module:version: 2
    update:module:context: 3b
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: pkg
    update:collection:evr: 1.2-3
    update:collection:arch: x86_64
    update:collection:filename: pkg-1.2-3.x86_64.rpm
  
    update:collection:name: filesystem
    update:collection:evr: 3.9-2.fc29
    update:collection:arch: x86_64
    update:collection:filename: filesystem-3.9-2.fc29.x86_64.rpm
  

solvable:buildtime: 1550849401

solvable 4 (5):
solvable:name: patch:DNF-2020-1
solvable:arch: noarch
solvable:evr: 1
solvable:vendor: dnf-testing@redhat.com
solvable:provides:
  patch:DNF-2020-1 = 1
solvable:conflicts:
  wget.x86_64 < 1.19.5-5.fc29
  wget.noarch < 1.19.5-5.fc29
  yum.x86_64 < 3.4.3-0
  yum.noarch < 3.4.3-0
  bitcoin.x86_64 < 2.5-1
  bitcoin.noarch < 2.5-1
solvable:patchcategory: bugfix
update:status: stable
solvable:summary: bugfix_B-2.0-1
update:severity: critical
update:reference:
  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 2222
  update:reference:title: 222
  update:reference:type: bugzilla

  update:reference:href: https://foobar/foobarupdate_2
  update:reference:id: 3333
  update:reference:title: CVE-2999
  update:reference:type: cve

solvable:description: testing advisory 2020
update:module:
  update:module:name: perl-DBI
  update:module:stream: master
  update:module:version: 2
  update:module:context: 2b
  update:module:arch: x86_64

  update:module:name: perl
  update:module:stream: master
  update:module:version: 1
  update:module:context: 1c
  update:module:arch: x86_64

update:collection:
  update:collection:name: wget
  update:collection:evr: 1.19.5-5.fc29
  update:collection:arch: x86_64
  update:collection:filename: wget-1.19.5-5.fc29.x86_64.rpm

  update:collection:name: yum
  update:collection:evr: 3.4.3-0
  update:collection:arch: x86_64
  update:collection:filename: yum-3.4.3-0.x86_64.rpm

  update:collection:name: bitcoin
  update:collection:evr: 2.5-1
  update:collection:arch: x86_64
  update:collection:filename: bitcoin-2.5-1.x86_64.rpm

update:collectionlist:
  update:module:
    update:module:name: perl-DBI
    update:module:stream: master
    update:module:version: 2
    update:module:context: 2b
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: wget
    update:collection:evr: 1.19.5-5.fc29
    update:collection:arch: x86_64
    update:collection:filename: wget-1.19.5-5.fc29.x86_64.rpm
  
    update:collection:name: yum
    update:collection:evr: 3.4.3-0
    update:collection:arch: x86_64
    update:collection:filename: yum-3.4.3-0.x86_64.rpm
  

  update:module:
    update:module:name: perl
    update:module:stream: master
    update:module:version: 1
    update:module:context: 1c
    update:module:arch: x86_64
  
  update:collection:
    update:collection:name: bitcoin
    update:collection:evr: 2.5-1
    update:collection:arch: x86_64
    update:collection:filename: bitcoin-2.5-1.x86_64.rpm
  

solvable:buildtime: 1329924602

However when I run it on fedora-updates, the resulting solv file is actually a bit smaller with my commit (11MB vs 13MB), so there seems to be more to it. The dumps of these files have over a million lines, I haven't been able to make any sense of it yet.

[mlschroe]

Seems like updateinfo_solvables_start and updateinfo_solvables_end are incorrect or not set when you call repowriter_set_solvablerange. Can you please add a debug statement to verify this?

[lukash]

I've already checked that, I have that log message there (in the commit it's at the time of loading the data, but I did add one right before the call as well), it says:

loaded updateinfo solvables start 3 end 7

Which seems correct. Should I come up with a standalone reproducer?

[mlschroe]

That doesn't seem correct. Package 3 is "pkg-libs" (the second package in the primary.xml), not the first package from the updateinfo. The correct range would be 5-9.

[mlschroe]

Oh, it's because you use repo->nsolvables instead of pool->nsolvables. The code should look like this instead:

int solvables_start = pool->nsolvables;
...
updateinfo_solvables_start = solvables_start;
updateinfo_solvables_end = pool->nsolvables;

[lukash]

Well that was a stupid mistake. Sorry about that and thanks for noticing. I also had to limit the repodata to just the updateinfo as you've mentioned, otherwise I was getting some more data in the solv file, but now the results are identical for both the test repo and fedora-updates. Again, thanks!

[lukash]

@mlschroe I think I managed to completely remove the hackish way in which we were limiting the solvables for writing the cache files, I've posted a PR: rpm-software-management/libdnf#1471

The comps are now also loading properly.

I'd be grateful if you skimmed through to see if there's still anything out of place. Thank you!

[mlschroe]

Looks good to me!

[lukash]

Great, thanks for the help! Closing.