Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add APIs to disable fetching filelists #416

Conversation

cgwalters
Copy link
Collaborator

As part of rpm-ostree jigdo ♲📦
I'd like to make it cheaper to fetch metadata. For Fedora currently
the filelists metadata is enormous. In jigdo mode, we don't need it,
so let's add APIs to avoid fetching it.

As part of [rpm-ostree jigdo ♲📦](coreos/rpm-ostree#1081)
I'd like to make it cheaper to fetch metadata.  For Fedora currently
the filelists metadata is enormous.  In jigdo mode, we don't need it,
so let's add APIs to avoid fetching it.
@cgwalters
Copy link
Collaborator Author

Also related: coreos/rpm-ostree#1127

@cgwalters
Copy link
Collaborator Author

For example, today I have jigdo mode deployed in our CI:
https://ci.centos.org/artifacts/sig-atomic/fahc/jigdo/repodata/
The primary XML is 1.1MB, the filelists is 30MB. The difference is quite noticeable with this patch in "pure jigdo" mode.

@Conan-Kudo
Copy link
Member

@cgwalters So what happens when someone requests something that uses a dep not in primary.xml?

cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Feb 6, 2018
All depsolving etc. was done on the server side; we just need the `primary.xml`
in order to enumerate available jigdoRPMs.

Requires: rpm-software-management/libdnf#416
@cgwalters
Copy link
Collaborator Author

cgwalters commented Feb 6, 2018

Downstream usage: coreos/rpm-ostree#1234

@cgwalters So what happens when someone requests something that uses a dep not in primary.xml?

This patch only applies in pure jigdo mode, i.e. "image/ostree-only" mode. As soon as you e.g. rpm-ostree install we start downloading the filelists. In practice, I suspect a lot of Workstation users will end up with layered packages, but it's less true for Atomic Host.

And this way things are also (more) efficient for people who do their own custom rpm-ostree composes.

@Conan-Kudo
Copy link
Member

@cgwalters I like the idea of being able to just dynamically turn off and on parts of the metadata downloaded based on the situation.

Do you think we can have this happen during normal libdnf/DNF operations so that we only download primary until we need to resolve a file dep not there and then get filelists? And thinking a bit further, if someone wants to retrieving changelogs from the repository, it'd be nice to fetch that on-demand too...

@cgwalters
Copy link
Collaborator Author

Do you think we can have this happen during normal libdnf/DNF operations so that we only download primary until we need to resolve a file dep not there and then get filelists?

That sounds hard, and we're very likely to end up downloading the filelists anyways at least for Fedora due to how prevalent they are. In the short term I'm likely to work on my "server side depsolving/filtering" idea instead.

* @context: a #DnfContext instance.
* @enable_filelists: %TRUE to download and parse filelist metadata
*
* Enables or disables download and parsing of filelists.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since?

**/
void
dnf_context_set_enable_filelists (DnfContext *context,
gboolean enable_filelists)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, actually, maybe we should have a new setup_sack_with_flags which defaults to NONE? E.g. for auto-updates, I'd like to be able to specify UPDATEINFO. And also to add a new flag to not fall back to updating an out-of-date repo.

jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Feb 22, 2018
All depsolving etc. was done on the server side; we just need the `primary.xml`
in order to enumerate available jigdoRPMs.

Requires: rpm-software-management/libdnf#416
@jlebon
Copy link
Contributor

jlebon commented Feb 22, 2018

This patch is included in #420. There's also a fixup! there to make sure we don't even download filelists at all. (IIUC, the patch added to check_internal here just makes sure that we don't invalidate our cache based on filelists, but we would still download it in dnf_repo_update).

jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Feb 28, 2018
All depsolving etc. was done on the server side; we just need the `primary.xml`
in order to enumerate available jigdoRPMs.

Requires: rpm-software-management/libdnf#416
jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Feb 28, 2018
All depsolving etc. was done on the server side; we just need the `primary.xml`
in order to enumerate available jigdoRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416
jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Mar 1, 2018
All depsolving etc. was done on the server side; we just need the `primary.xml`
in order to enumerate available jigdoRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416
jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Mar 5, 2018
All depsolving etc. was done on the server side; we just need the
`primary.xml` in order to enumerate available rojigRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416
jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Mar 5, 2018
All depsolving etc. was done on the server side; we just need the
`primary.xml` in order to enumerate available rojigRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416
jlebon pushed a commit to jlebon/rpm-ostree that referenced this pull request Mar 5, 2018
All depsolving etc. was done on the server side; we just need the
`primary.xml` in order to enumerate available rojigRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416
rh-atomic-bot pushed a commit to coreos/rpm-ostree that referenced this pull request Mar 5, 2018
All depsolving etc. was done on the server side; we just need the
`primary.xml` in order to enumerate available rojigRPMs.

Update submodule: libdnf

Requires: rpm-software-management/libdnf#416

Closes: #1268
Approved by: cgwalters
@j-mracek j-mracek self-assigned this Mar 26, 2018
@j-mracek
Copy link
Member

Besides the patch breaks API and PackageKit, I am fine with that. @kalev and @hughsie are you fine with that?

@jlebon
Copy link
Contributor

jlebon commented Mar 26, 2018

@j-mracek Let's close this one in favour of #420? It includes that same commit as well as fixups for it.

The patch shouldn't break any APIs. Filelists are fetched by default, like before.

@kalev
Copy link
Collaborator

kalev commented Mar 26, 2018

+1 from me. Looks like a nice improvement that just adds new API without changing anything for existing consumers.

@j-mracek
Copy link
Member

Closing in favor of #420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants