try to fix package_id with package_revision_mode and mixed modes

[memsharded]

Changelog: Bugfix: Fix crash while computing the package_id of a package when different package_id_mode are mixed and include package_revision_mode.
Docs: Omit

Continuation of #6947
Fix #6942

#tags: slow

cc/ @fulara

[jgsogo]

I'm starting to find all this implementation around the package-id computation really complex, and also the package ID itself when mixing modes, I think I need a place in the docs explaining at least how the propagation works.

[jgsogo]

Is there a reason to do this only when full_transitive_package_id is activated?

[jgsogo]

reevaluate_node will run _compute_package_id that calls this same function. Why is it needed to update this information before doing those calls?

[memsharded]

Because reevaluate_node only runs for those nodes that has PACKAGE_ID_UNKNOWN, but the information that needs to be re-computed after being built are the one of the dependencies of the node with PACKAGE_ID_UNKNOWN.

Only full_transitive_package_id is causing the issue in this case, because if not defined, a different computation is done, based on conanfile.requires (which is updated).

I think the root cause of this is the immutability of the reference objects, that produces that there are several copies of ConanFileReference and PackageReference. I think for the future we should aim for a central definition of these for each node, in a way that an update to them automatically is seen by everyone else, because they contain a reference to this object.

[jgsogo]

Because reevaluate_node only runs for those nodes that has PACKAGE_ID_UNKNOWN, but the information that needs to be re-computed after being built are the one of the dependencies of the node with PACKAGE_ID_UNKNOWN.

Here you mean, the CONSUMERS of the node with PACKAGE_ID_UNKNOWN, right?

---

Only full_transitive_package_id is causing the issue in this case, because if not defined, a different computation is done, based on conanfile.requires (which is updated).

Having a quick look to conanfile.requires, it is built using the direct and the indirect prefs too, so maybe we can simplify all of this a lot moving logic into ConanInfo (not this PR).

[memsharded]

PkgA->PkgB->PkgC

PkgA has PACKAGE_ID_UNKNOWN, because we still need to build PkgC or PkgB and awe are in package_revision_mode

reevaluate_node will only be executed for PkgA. But the computation of transitive updated PackageReferences need to be computed for both PkgB and PkgC.

Does this clarify the issue a bit?

[jgsogo]

Ok, we are running them to populate Node::id_direct_prefs and Node::id_indirect_prefs with the new values (are we doing this twice for pkgC?)

[memsharded]

Yes, it is being called twice. That is the key of the issue, we need to run this 2 times:

• One to compute the first package_id, while analyzing the graph binaries
• A second one, while installing the binaries, to update for possible changes upstream that will be used in reevaluate_node to compute the package_id of consumers with PACKAGE_ID_UNKNOWN.

It seems it would be very challenging to skip running this for leaf nodes that wouldn't need to run this.

[jgsogo]

Probably if we are moving only one branch this function doesn't belong to node, but to self, or we should move both branches.

[memsharded]

I put it in the node because now it needs to be called from 2 different locations: graph_binaries.py and installer.py. I could leave it here in the graph_binaries.py as a public static method that receives a single node as argument, no problem.

[jgsogo]

Because reevaluate_node only runs for those nodes that has PACKAGE_ID_UNKNOWN, but the information that needs to be re-computed after being built are the one of the dependencies of the node with PACKAGE_ID_UNKNOWN.

Here you mean, the CONSUMERS of the node with PACKAGE_ID_UNKNOWN, right?

---

Only full_transitive_package_id is causing the issue in this case, because if not defined, a different computation is done, based on conanfile.requires (which is updated).

Having a quick look to conanfile.requires, it is built using the direct and the indirect prefs too, so maybe we can simplify all of this a lot moving logic into ConanInfo (not this PR).

[fulara]

Hello @memsharded just tested it - but it doesnt seem to resolve the issue I have mentioned.

With this branch the:
self.info.requires.pkg_names
still contains the name of the currently processed conanfile - which was not the case on 1.24.0.

[memsharded]

Hi @fulara I have introduced a check in 40be838, seems to work fine. Can you please tell how to reproduce what you are seeing there?

[fulara]

@memsharded
Problem i've encountered and described happens when depending upon something, not when conan create'ing it: see:

As i've written in my original problem description:

conan create of bla will not print out any errors.
conan create of bla2 will print 2 errors bla and bla2.

described in depth here:
#6942 (comment)

I got this by looking at the logs - if you are seeing different behavior i can try to look in depth.

[memsharded]

I got this by looking at the logs - if you are seeing different behavior i can try to look in depth.

Yes, please, check again.

I have had a look, and added an extra check (75ebf2d) to the intermediate dependency, seems to be working fine. I have also had a look to the source code, but couldn't figure out what could be the reason of what you are seeing there.

[fulara]

I've checked this again - I think I came to wrong conclusions about the problem,
my build still fails - but I am not sure why yet,

I will get back to you.

Possibly my conclusion about the dependency being on the pkg_names was totally wrong, and I apologise for giving you the false hints.

I'll try to analyze the issue here and tell you the rootcause why the build fails for me.

[memsharded]

Possibly my conclusion about the dependency being on the pkg_names was totally wrong, and I apologise for giving you the false hints.

No problem! Thanks for the continuous feedback and support. Let me know if you have any further insight about this issue that I could check.

[fulara]

Hello @memsharded
Yes, my build tree now builds.

The issue seems to be that sometimes the build_requires are being added into the self.info.requires.
THey are being selectively added - only when the dependencies are actually being build,
I am not sure how exactly - I havent analyzed,

In my case with your changes the gtest was sometimes added to self.info.requires.pkg_names - which was not the case before.

What I observed was:
Change the default_package_mode
conan create . --build missing
This fails on some package. repeat:
conan create . --build missing

The building process goes over ( the build_requires are no longer on the self.info.requires.pkg_names list then. )

[memsharded]

Yes, finally understood the root cause:

• package_revision_mode causes PACKAGE_ID_UNKNOWN
• When rebuilding packages, the PACKAGE_ID_UNKNOWN needs to be re-evaluated, to compute a updated package-id
• The package-id requires information of its dependencies, down to the package_revision, including transitive dependencies.
• The accumulated information about transitive dependencies need to be recomputed for each node, while the node is finally installed in disk
• The re-evaluation of transitive dependencies was pulling the build-requires too, that they didn't exist the first time the package_id was computed, because at that time, the build-requires were not expanded at all.

It has been fixed and a new red-green test added.

Thanks @fulara for all the work on this. Lets try to release this soon in 1.26.