[bug] ERROR: Build-require 'xxx' cannot be found in lockfile (but it's there)

[SSE4]

Environment Details (include every applicable attribute)

• Operating System+version: Linux, docker image conanio/gcc11
• Compiler+version: gcc (Ubuntu 11.1.0-1ubuntu1~20.04) 11.1.0
• Conan version: Conan version 1.44.0
• Python version: Python 3.7.5

Steps to reproduce (Include if Applicable)

run conanio/gcc11 image. execute the following command to create a lockfile:

$ conan lock create --reference gst-plugins-good/1.19.1@ --build -pr:b default -pr:h default

(two profiles are optional, it reproduces in single profile mode as well)

run the command to install:

$ conan install gst-plugins-good/1.19.1@ --lockfile=conan.lock --build missing

it will say:

ERROR: Build-require 'automake' cannot be found in lockfile

but it's clearly there:

$ cat conan.lock  | grep automake
    "ref": "automake/1.16.4",
    "ref": "automake/1.16.4",

(also the full lockfile is attached below)

Logs (Executed commands with output) (Include/Attach if Applicable)

conan install gst-plugins-good/1.19.1@ --lockfile=conan.lock --build missing
Using lockfile: '/home/conan/conan.lock'
Configuration (profile_host):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

ERROR: Build-require 'automake' cannot be found in lockfile

conan.lock.txt

[SSE4]

@memsharded any ideas what could be wrong here?

[memsharded]

I have had a look.
The problem seems to be when a recipe requires the same package both as a requires and a build_requires. In this case, why is libtool having a regular requires to automake? Is automake some kind of library that that is being linked into libtool?

I have checked that if:

diff --git a/recipes/libtool/all/conanfile.py b/recipes/libtool/all/conanfile.py
index b20875b66..7e254a242 100644
--- a/recipes/libtool/all/conanfile.py
+++ b/recipes/libtool/all/conanfile.py
@@ -43,15 +43,15 @@ class LibtoolConan(ConanFile):
         del self.settings.compiler.libcxx
         del self.settings.compiler.cppstd

-    def requirements(self):
-        self.requires("automake/1.16.4")
-
     @property
     def _settings_build(self):
         return getattr(self, "settings_build", self.settings)

It seems to work past that error (and still fail in a similar one with m4, that I suspect it has the same origin).

I will try to keep investigating why this happen (seems complex to solve, as it is internal to the graph resolution algorithm), I suggest trying to simplify that part of the recipes if possible.

[SSE4]

thanks for taking a look! AFAIK, automake is just a collection of scripts, and has no C++ source code.
there was a related issue: conan-io/conan-center-index#8157
/cc @madebr , maybe he knows if requirement of automake in libtool is redundant

[madebr]

libtool can be used in 2 forms:

• as a requirement providing the libltdl library
• as a build requirement providing scripts for autotools

automake is needed as a build requirement when building automake ==> added to build_requirements

When libtool is used as a build_requirement, it also needs automake (and autoconf) added as a build requirement. ==> added to requirements
We added the automake version to libtool so as to define the automake version in one place only.
This also avoid potential problems of building libtool with automake version X.Y.Z and using libtool with automake version A.B.X.

[Hopobcn]

This issue looks similar to #10345

[SSE4]

to conclude, it's fine if libtool is used in both requires and build_requires simultaneously. and we have many more cases like that, most famous is probably protobuf. so it's not something we can apply workaround for, it deserves a proper fix in the client.

[anton-danielsson]

We are having the same issue. This means we can't use lock files. It's protobuf that is our culprit.

[bukulin]

Similar issue here with cmake. It is not linked, just a tool_requirement. To reproduce the error message I've created a fork of the conan examples and tweaked it a little:
conan-io/examples@master...bukulin:conan_examples:master

libb and app1 both depend on cmake as a tool_requirement. That was enough to have the same error message as we can see in the title of this issue, but I think the reason was slightly different. Shall I open a new issue for that?

[harendt]

I'm observing the same issue. In my case it's protobuf and grpc that are needed both, as requires and tool_requires.

Is there any known workaround? Removing the tool_requires won't work, because the tool_requires package must be x86_64 and the requires package must be aarch64.

[memsharded]

When a tool-requires is missing in a lockfile, the general approach is to make sure that --build=xxx is provided in the conan lock create .... --build=xxxx to make sure that the graph expands and include the tool_requires that will be necessary later for building. Please try that.

[bukulin]

Actually the tool is listed in the lockfile even if I do not add --build=xxx.

Let's suppose that tool requirement is cmake for brevity. If I do conan lock create ... --build=cmake then the only change is that conan lock build-order lists cmake in the first place to build, but further rebuilds still missing cmake from the lockfile.

[technic]

The only workaround I found is installing some random tools like cmake, autoconf etc, and at some point ther problem was gone. But this I can not do in CI

[dornbirndevelops]

hi all,

the only pattern I see is, that a dependency of mine needs the same package as a build requirement,
causing the build requirement to appear twice in the lockfile instead of being merged into one.

with a package graph setup similar to this:

A:
  requires: B
  tool_requires: C
B:
  tool_requires: C
C: ...

you get a lock file like this:

1:
  ref: A
  context: host
  requires: 2
  tool_requires: 3
2:
  ref: B
  context: host
  tool_requires: 4
3:
  ref: C
  context: build
4: # duplicate of 3
  ref: C
  context: build

building this constellation without a locked dependency graph works fine by the way.

[connorsmacd]

Hey @memsharded, from what I've read here, Conan V2 will have entirely new lockfiles. Do you expect this problem will go away once we make the switch?

[technic]

Another workaround which seem to be working: first create lock file without -b, then based on it run conan install with -b missing or whatever you need.

[memsharded]

Hey @memsharded, from what I've read here, Conan V2 will have entirely new lockfiles. Do you expect this problem will go away once we make the switch?

Yes, absolutely, the new 2.0 lockfiles will not have these issues. I'd strongly recommend giving 2.0 lockfiles a try, with some toy recipes (it is easy to do graphs with conan new cmake_lib -d name=pkg -d version=1.0 -d requires="dep/1.0" new 2.0 command).

[harendt]

I'm not sure, but I think I found at least one cause for this issue: The problems seems to be that the equality/inequality/hash operators of conans.client.graph.graph.Edge only consider the src and the dst of the Edge, but not whether it's a build requirement or a private requirement.

As a consequence, edges for build requirements are not added if there is already an edge that represents a regular (non-build) requirement for the same src and dst.

I think this could be easily fixed by including Edge.build_require and Edge.private in the equality/inequality/hash operators. The current implementation looks as follows:

conan/conans/client/graph/graph.py

Lines 226 to 247 in a82ae09

	class Edge(object):
	def __init__(self, src, dst, require):
	self.src = src
	self.dst = dst
	self.require = require
	@property
	def private(self):
	return self.require.private
	@property
	def build_require(self):
	return self.require.build_require
	def __eq__(self, other):
	return self.src == other.src and self.dst == other.dst
	def __ne__(self, other):
	return not self.__eq__(other)
	def __hash__(self):
	return hash((self.src, self.dst))

Does this make sense? Then I could create a patch for this.

[memsharded]

Hi @harendt

Thanks for your investigation and the offer.

I am afraid that this could be a high risk change. The problem is that private do not automatically mean they should not conflict (they conflict when they are equal), they mean they do not aim to be propagated down. But having a transitive dependency to zlib and adding a private dependency to zlib shouldn't necessarily result in 2 different nodes in the graph and 2 different dependencies in all cases. So changing the equality might easily break existing usage, we are not at a stage when we could introduce this instability in Conan 1.X. If you still want to do the patch we could further discuss and have a look, but at first look, it is unlikely that we would be able to move it forward.

[harendt]

If including the private attribute would be fragile, I could just try to include the build_require attribute. Maybe that's less problematic. The build_require is the cause for this issue.

I just mentioned private as well because I recognized that it's also not included in the operators. But that may be less of a problem.

[memsharded]

Actually both can be problematic, because in Conan 1.X without using the 2 profiles approach there is little difference between build-requires and regular requires, they both live in the same context and are treated as equal. But feel free to try it and we will think about it.

[memsharded]

This referred to old Conan 1.X lockfiles. Lockfiles have been fully revamped in Conan 2.0, I am closing this ticket as outdated, please create new tickets referring to the new lockfiles for any further question or issue. Thanks!