Fixes #28049 - Index yum repo w/o errata refactoring w/ Pulp 3

[ianballou]

This PR enables yum content indexing with Pulp 3. Limitations:

1. Kickstart repos might not work, but I am testing this currently.
2. Cannot index package groups, SRPMs, or yum metadata files.
3. Manifest refresh is not supported yet.

TESTING:
Try to sync various repositories and view their errata, distribution information, packages, and appstream modules. Note the limitations above. Don't try to sync the RHEL 7 Server RPMs repo unless you have more than 12 GB ram.

For now, remove any empty string checks in this file:

~/foreman/.vendor/ruby/2.5.0/gems/pulp_rpm_client-<insert version>/lib/pulp_rpm_client/models/rpm_update_record.rb

The empty string checks will look like this:

if issued_date.to_s.length < 1
    fail ArgumentError, 'invalid value for "issued_date", the character length must be great than or equal to 1.'
end

We're hoping to merge this sooner rather than later. Since it's Pulp 3 and doesn't affect normal Katello users, we may merge with some of the following TODOs unchecked.

TODO:

• Write more tests
• Test Kickstart repo (this works but we don't index package groups yet)
• Get eval out of my code thanks to https://pulp.plan.io/issues/5650
• Ensure that Kickstart repos can be synced and booted from
• Try syncing RHEL/CentOS 8 AppStream repo
• Limit how much is fetched from Pulp 3 during indexing (:PULP_INDEXED_FIELDS)
• Properly test CVEs and BZs in erratum_test.rb (Pulp 3 having issues with file:///var/www/test_repos/zoo/ repo)
• Add Package Group support (in this PR or another?)
  • We'll add this in another PR.

[theforeman-bot]

Issues: #28049

[sjha4]

@ianballou : Did you hit this: http://pastebin.test.redhat.com/811594 ?

[ianballou]

@sjha4 yeah, looks like I'll need to update the bindings again.

[sjha4]

Tested sync and indexing after modifying binding gem code a little. Able to index packages,errata modules etc.. 👍
One thing I noticed is you can not go the packages> package one > details page as it tries to go to pulp v2 api..Are you seeing that behavior?

[ianballou]

oh yeah @sjha4 I see that too, I'll have to check why

[sjha4]

So using this https://fedorapeople.org/groups/katello/fakerepos/zoo repo, when syncing errata, the associated packages are not indexed correctly. I believe only the first is getting indexed.
For example RHEA-2010:9984 should have penguin and walrus and RHEA-2010:9983 should have cheetah, elephant and girraffe. (https://fedorapeople.org/groups/katello/fakerepos/zoo/repodata/updateinfo.xml)

[ianballou]

@sjha4 yeah I see that too. Looks like it doesn't happen for all repos, since more than one package per errata was indexed for this larger zoo repo https://jlsherrill.fedorapeople.org/fake-repos/needed-errata/

[ianballou]

@sjha4 I fixed the missing packages issue.

[ianballou]

I've made a work-around for the issue where individual package pages were failing to load. There's still an issue where the package 'provides' and 'requires' data comes back in a weird format, so I'm stubbing those for now.

[jlsherrill]

I'm not seeing any kickstart distribution information being indexed, is that known/tracked?

[ianballou]

The distribution data worked for me with the CentOS 7 repo, what repo did you sync? Also I have a distribution test, is that the information you're talking about?

…

[jlsherrill]

I was testing the RHEL 7.7 kickstart repo:

[ianballou]

@jlsherrill I'll give that a test, I expected that data to populate

[jlsherrill]

as we discussed on irc, it seemed to be a pulp bug, so i've filed that

[ianballou]

fixed the indexing of file and build information for packages.

[ianballou]

Now package dependencies should be listed ok. The approach I took to generate the strings should be revisited. Hopefully at some point the data will be returned from Pulp as a hash with the correct keys rather than an array where we need to guess from the indices.

[ianballou]

I've refactored some of the pulp3 tests to give them a pulp3 module. There were issues with class naming for RpmTest since there wasn't a pulp3 module but it shared a name with the pulp2 tests.

[sjha4]

@ianballou : What version of the pulp_rpm_client are you on? I believe I am on pulp_rpm_client-3.0.0b71571331815 which doesn't have the file lib/pulp_rpm_client/models/rpm_update_record.rb file. The validations are in the update_record.rb file which makes me think I am on a newer version.

[ianballou]

@sjha4 I'm on the same version as master at the moment. pulp_rpm_client-3.0.0b8.dev01573228480 has rpm_update_record.rb

[sjha4]

Findings based on testing with 3.0.0b8.dev01573228480:
Pulp3:

1. Can sync and index yum repos after some known modifications to the binding gem.
2. Tabs on the Content>Package page don't load: http://pastebin.test.redhat.com/813392

Pulp2:
All the flows I tested work as expected.

[sjha4]

@ianballou : Seeing a routing error now:

nvm if you don't see this cause my confidence in my pulp3 box is not too high..Could be unrelated..

[ianballou]

@sjha4 yeah I think it has to do with our boxes having slightly different versions of the pulp code. It'll be easier to confirm when we can bring up a new clean pulp box again

[jlsherrill]

seeing strange behavior when looking at the files list for an rpm, notice some files don't have paths:

[jlsherrill]

APJ!