Improve sync performance

[hao-yu]

Prevent unneeded looping the metadata files to speed up the
syncing process.

re: #5865
https://pulp.plan.io/issues/5865

[pulpbot]

Can one of the admins verify this patch?

[pulpbot]

Can one of the admins verify this patch?

[pulpbot]

Can one of the admins verify this patch?

[goosemania]

ok test

[hao-yu]

Any chance this patch and be reviewed and tested? Thanks.

[hao-yu]

Any chance to review this please?

[daviddavis]

@hao-yu thank you for this work. Apologies for not responding sooner.

Since the release of Pulp 3.0 GA, we're only merging critical bug fixes to 2-master. What sort of performance problems are you seeing? How much does this change speeds up things?

[hao-yu]

@daviddavis thanks for the reply. Yes. I know that Pulp-2 is already in the maintenance mode but I can still see this patch is likely to benefit the Satellite 6 at least for a couple of years more before it move to Pulp 3.

This patch is basically to reduce the number of times that Pulp need to loop from the yum metadata to fetch a list of packages. Please look at the below link. The test indicates that the patch can improve the performance of syncing a RHEL 7 server repository up to 10minutes. The "Removing missing units" usually took 5 minutes+ even there is no packages to delete now it will only take a couple seconds .

https://pulp.plan.io/issues/5865#note-2

[daviddavis]

@hao-yu I talked to others on the RPM team and they've confirmed that we're only accepting critical bug fixes right now for Pulp 2. I feel bad because this is a solid improvement and I know you worked hard on it. Also, apologies for not responding sooner. I hope you will continue to help us improve Pulp in the future though.