[question] Numeric node IDs in lockfiles

[sztomi]

When our CI builder script runs, it separates the export, build, test, upload steps for each recipe from our monorepo. Exporting is performed only once for all recipes. Building (via conan create) is performed according to the build-order determined by conan. Finally, testing and uploading is done via conan test and conan upload.

The reason for separating each step is to be able to report on what exactly fails.

When using lockfiles, I lock a metapackage ("variant") with the profile. Later, I supply this lockfile to each conan create invocation.

Then, in the test step I do the same, with the expectation that test will utilize all the package revisions that were just built. However, I get an error along the lines of:

ERROR: The node ID 68 was not found in the lock 

If I look inside the lockfile, there is unsurprisingly, no node with that ID. But why is conan looking for it? My expectation would be that it would work similarly to when I pass the lockfile to conan create. And the effect should be "use the revisions that are locked in this lockfile to run the test". Am I misunderstanding something?

• I've read the CONTRIBUTING guide.

[jgsogo]

Hi, I think that your issue is a duplicated of this one (#6067), but with the new version of lockfiles (it uses numeric ids instead of the reference).

Are you using requires in the test_package of the recipe that is failing?

[sztomi]

I'm not exactly sure if it's the same, or that it's a new version of lockfiles. In fact, I'm encountering both errors.

ERROR: The node ID 68 was not found in the lock

from conans import tools
from plexconantool import PlexConanFile, plextesttool


class PNGTest(PlexConanFile):
  settings = "os", "compiler", "build_type", "arch"
  generators = "cmake"

  def build(self):
    # .conanfile_directory
    plextesttool.build_test_program(self)

  def test(self):
    plextesttool.test_program(self, "test_png")


  def imports(self):
    self.copy("*.dll", src="bin", dst="bin")

Here's another one with a non-numeric error:

ERROR: Couldn't find 'minizip/1.2.11-5' in graph-lock

from plexconantool import PlexConanFile, plextesttool
from conans import ConanFile, CMake, tools
import os

class TestPackage(PlexConanFile):
  settings = "os", "compiler", "build_type", "arch"
  generators = "cmake"

  def build(self):
    # .conanfile_directory
    plextesttool.build_test_program(self)

  def test(self):
    plextesttool.test_program(self, "test")

  def imports(self):
    # copy dll files for windows
    self.copy("*.dll", "bin", "bin")

These are structured pretty much identically (just like the vast majority of our test_packages).

[sztomi]

Are you using requires in the test_package of the recipe that is failing?

Mostly no, but in rare cases we do, for example python modules need libpython to link a test executable.

from plexconantool import PlexConanFile, plextesttool
from conans import ConanFile, CMake, tools
import os, platform


class TestPackage(PlexConanFile):
  settings = "os", "compiler", "build_type", "arch"
  plex_requires = "python/2.7.17-26a8073720-1"
  generators = "cmake"

  def build(self):
    # .conanfile_directory
    plextesttool.build_test_program(self)

  def test(self):
    env = {"PYTHONHOME": os.environ.get("PLEX_PYTHONHOME", "")}
    plextesttool.test_program(self, "test", env=env)

[jgsogo]

Hi! It's quite difficult for me to know what is happening under the hood of all those plexXXXX wrappers on top of Conan elements, so I won't be able to reproduce it. Can you reproduce the issue with a minimum example? or, is there a way we can run your recipes?
Thanks

[memsharded]

In any case, the issue #6067 is not related only to build_requires. That error will be raised for any test_package/conanfile.py that contains a requires or build_requires (that is not the tested package).

This is because the test_package is not part of the initial lockfile that was generated downstream. When a graph is being constructed and those new dependencies are not found in the lockfile, it raises, as lockfiles so far assume that they cannot have new dependencies (and it is quite difficult and risky to allow the opposite, because any failure will produce changes in the graph, instead of actually locking things). There is ongoing work to allow these "dynamic" lockfiles that can grow and add new dependencies that were not previously locked.

[sztomi]

@jgsogo I'll try that. For the purposes of this issue, those special wrappers should act the same as the upstream equivalents. I think what @memsharded says explains it pretty well.

[theodelrieu]

Hi @memsharded,

I've added cmake_installer/3.15.3@conan/stable as a build requirement, and everything works great.

I do get the following warning: WARN: Package can't be locked, not found in the lockfile

Also, the lockfile content looks a bit weird:

   "6": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830",
    "options": ""
   },
   "7": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830",
    "options": ""
   },
   "8": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830#0",
    "options": ""
   },
   "9": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830",
    "options": ""
   },
   "10": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830#0",
    "options": ""
   },
   "11": {
    "pref": "cmake_installer/3.15.3@conan/stable#0:44fcf6b9a7fb86b2586303e3db40189d3b511830#0",
    "options": ""
   }

Previous nodes each have a build_requires field, referencing a different number (i.e. from 6 to 11).

[memsharded]

Yes, the I do get the following warning: WARN: Package can't be locked, not found in the lockfile, is normal, it is exactly what is happening, the cmake_installer node was not locked, as the test_package/conanfile.py is NOT in the locked graph. Build requires are also private for each node, so it is normal that there are multiple nodes for build-requires.

I keep investigating @sztomi issues.

[theodelrieu]

@memsharded I wanted to post that comment in the PR... Oops!

[sztomi]

@memsharded still trying and failing to create a minimal reproducible example. What I found so far, is that from the list of failing/successful packages, it seems like every failing package appears as a conditional requirement somewhere (i.e. in a requirements() function), while the succeeding packages are only listed unconditionally. Maybe something is not resolving every time like they do at the time of locking? The conditions are simple, such as testing settings and options.

[memsharded]

We are releasing a general.relax_lockfile configuration in 1.23 to implement this behavior, because of concerns of breaking something. This config is highly experimental, most likely will be removed in future versions, but we wanted to at least provide an opt-in for this behavior to be able to move forward.