bld2repo should download subpackages needed for build

[kdudka]

python2-six-1.11.0-6.module+el8.4.0+9287+299307c7 needs python2-libs-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm for build but the following command does not download it:

$ bld2repo --build-id 1444241 \
    --koji-host "https://brewhub.engineering.redhat.com/brewhub" \
    --koji-storage-host "http://download.devel.redhat.com/brewroot"

... which prevents the package from being successfully built in mock:

$ mock -r rhel-8-x86_64 --quiet --installdeps" "/tmp/csmockiodxf3_u/python2-six-1.11.0-6.module+el8.4.0+9287+299307c7.src.rpm" "--addrepo" "file:///tmp/csmockiodxf3_u/local-build-repo"
Error: 
 Problem: package python2-tkinter-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64 requires python2 = 2.7.18-2.module+el8.4.0+9193+f3daf6ef, but none of the providers can be installed
  - conflicting requests
  - nothing provides python2-libs(x86-64) = 2.7.18-2.module+el8.4.0+9193+f3daf6ef needed by python2-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64
(try to add '--skip-broken' to skip uninstallable packages

[kdudka]

We might consider using the brew download-build facility instead of fetching Brew download URLs directly. It would probably resolve this issue as a side effect.

[FrostyX]

@kdudka, @mcurlej are you guys planning to work on this issue?

[kdudka]

I will definitely not be able to look closer at it this week and would prefer if @mcurlej reviewed it first anyway.

[mcurlej]

@kdudka Will plan this for our next sprint next week, to fix this.

[kdudka]

It seems that modular builds are not going away from RHEL. Could we please prioritize this ticket?

It would help to make modular builds reproducible for tools and services that need it.

[fivaldi]

@kdudka Thanks for reporting the issue. I tried reproducing it with the inputs from you. With EL 8.5 I didn't went into the issue. It seems like a transitional error for 8.4. Unfortunately, I couldn't back track the root cause. :-/

As for brew download-build, this would be possible, as bld2repo tool has koji dependency for accessing the Koji session. Anyway, it doesn't download all the RPMs within the tag, so the first step has to be getting build-required packages from the build. After this, individual RPMs are downloaded, which would have to be done the same way with brew download-build.

Or, did I misunderstand your idea?

[kdudka]

I tried reproducing it with the inputs from you. With EL 8.5 I didn't went into the issue. It seems like a transitional error for 8.4. Unfortunately, I couldn't back track the root cause. :-/

Could you please tell us what exactly you tried and what exactly the outcome was?

The commands mentioned in #42 (comment) still work on my Fedora 34 VM.

Anyway, it doesn't download all the RPMs within the tag, so the first step has to be getting build-required packages from the build.

Yes. I believe that bld2repo already does it.

After this, individual RPMs are downloaded, which would have to be done the same way with brew download-build.

Nope. brew download-build -a x86_64 -a noarch python2-2.7.18-2.module+el8.4.0+9193+f3daf6ef downloads also python2-libs-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm, which the mentioned bld2repo command does not download. python2-libs-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm needs to be downloaded to prevent the subsequent mock command from failing.

[fivaldi]

@kdudka There's a PR #51 which should fix the issue. As of brew download-build, I didn't found an API for it in koji. So I think it would be unsafe to use the CLI and implement it via a shell cmd call. Wdyt?

[kdudka]

Thank you for fixing it! So this appeared to be an implementation bug of bld2repo in the end. I get the expected results with #51. The command stated in #42 (comment) now downloads additional 24 RPMs, as expected:

PyYAML-debugsource-3.12-16.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
babel-2.5.1-9.module+el8.1.0+3111+de3f2d8e.noarch.rpm
python2-Cython-debuginfo-0.28.1-7.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python2-bson-3.6.1-11.module+el8.1.0+3446+c3d52da3.x86_64.rpm
python2-coverage-4.5.1-4.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python2-debuginfo-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm
python2-docs-2.7.16-2.module+el8.1.0+3111+de3f2d8e.noarch.rpm
python2-libs-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm
python2-lxml-debuginfo-4.2.3-3.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python2-numpy-debuginfo-1.14.2-15.module+el8.4.0+9287+299307c7.x86_64.rpm
python2-numpy-f2py-1.14.2-15.module+el8.4.0+9287+299307c7.x86_64.rpm
python2-pip-wheel-9.0.3-18.module+el8.3.0+7707+eb4bba01.noarch.rpm
python2-psycopg2-debug-debuginfo-2.7.5-7.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python2-pymongo-debuginfo-3.6.1-11.module+el8.1.0+3446+c3d52da3.x86_64.rpm
python2-scipy-1.0.0-20.module+el8.1.0+3323+7ac3e00f.x86_64.rpm
python2-setuptools-39.0.1-12.module+el8.3.0+7075+8484f0d0.noarch.rpm
python2-test-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm
python2-tools-2.7.18-2.module+el8.4.0+9193+f3daf6ef.x86_64.rpm
python2-wheel-wheel-0.31.1-2.module+el8.1.0+3725+aac5cd17.noarch.rpm
python-nose-docs-1.3.7-30.module+el8.1.0+3111+de3f2d8e.noarch.rpm
python-psycopg2-debuginfo-2.7.5-7.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python-psycopg2-debugsource-2.7.5-7.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python-psycopg2-doc-2.7.5-7.module+el8.1.0+3111+de3f2d8e.x86_64.rpm
python-pymongo-debuginfo-3.6.1-11.module+el8.1.0+3446+c3d52da3.x86_64.rpm

I believe this issue can be closed with the merge of #51.