[question] How should replacement for scm work if recipe not in root Git dir

[gabyx]

I am working on replacing the scm attribute in https://docs.conan.io/en/latest/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature
I stumbled upon the fact that when the conanfile.py is in a subfolder /workspace/repo/tests/project (could also be a submodule, but ignore that for now) of the main repository /workspace/repo , then the example does not work, or I don't know how to work around this:

  def export(self):
    git = self.mymodule.GitRepo(self, self.recipe_folder)
    url, commit = git.get_url_and_commit()
    # We store the current url and commit in conandata.yml
    update_conandata(self, {"sources": {"commit": commit, "url": url}})

  def source(self):
    print("SRC::",os.getcwd()) # Bug: THIS is the self.folder.sources, also it should not be??

    sources = self.conan_data["sources"]
    
    self.mymodule.clone_and_checkout(self,
        url=sources["url"], dest="../../repo", ref=sources["commit"])

  def layout(self):
    self.folders.source = "repo/tests/project"

My problem is that: sources["url"] in sources points to a local folder on conan create which is good, but its the folder where the recipe lies /workspace/repo/tests/project, and not the root Git dir /workspace/repo.

How can we fix this? Do I need to check if self.conan_data["sources"]["url"] is a absolute path instead of an URL?

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @gabyx

Sorry for the delay in responding.

I am putting this test, to validate that the recipe can be inside a subfolder and not the root:

class TestConanFileSubfolder:
    conanfile = textwrap.dedent("""
        import os
        from conan import ConanFile
        from conan.tools.scm import Git
        from conan.tools.files import update_conandata, load

        class Pkg(ConanFile):
            name = "pkg"
            version = "0.1"

            def export(self):
                git = Git(self, os.path.dirname(self.recipe_folder))
                url, commit = git.get_url_and_commit()
                # We store the current url and commit in conandata.yml
                update_conandata(self, {"sources": {"commit": commit, "url": url}})
                self.output.info("URL: {}".format(url))
                self.output.info("COMMIT: {}".format(commit))

            def layout(self):
                pass # self.folders.source = "source"

            def source(self):
                git = Git(self)
                sources = self.conan_data["sources"]
                url = sources["url"]
                commit = sources["commit"]
                git.clone(url=url, target=".")
                git.checkout(commit=commit)
                self.output.info("MYCMAKE: {}".format(load(self, "CMakeLists.txt")))
                self.output.info("MYFILE: {}".format(load(self, "src/myfile.h")))
        """)

    def test_conanfile_subfolder(self):
        """
        A local repo, without remote, will have commit, but no URL
        """
        c = TestClient()
        c.save({"conan/conanfile.py": self.conanfile,
                "CMakeLists.txt": "mycmakelists",
                "src/myfile.h": "myheader"})
        commit = c.init_git_repo()
        c.run("export conan")
        assert "pkg/0.1: COMMIT: {}".format(commit) in c.out
        assert "pkg/0.1: URL: {}".format(c.current_folder.replace("\\", "/")) in c.out

        c.run("create conan")
        assert "pkg/0.1: MYCMAKE: mycmakelists" in c.out
        assert "pkg/0.1: MYFILE: myheader" in c.out

As long as the git = Git(self, os.path.dirname(self.recipe_folder)) in the export() points to the root folder, it seems to work fine.

Please have a look and let me know.

[memsharded]

Submitting a PR with the test above that will close this issue: #11533

[gabyx]

How can I test this?

[gabyx]

OK here is a test visualizing the behavior:

import unittest
from conans.test.utils.tools import TestClient
import textwrap

class TestConanFileSubfolder(unittest.TestCase):
    conanfile = textwrap.dedent("""
        import os
        from conan import ConanFile
        from conan.tools.scm import Git
        from conan.tools.files import update_conandata, load

        class Pkg(ConanFile):
            name = "pkg"
            version = "0.1"

            def export(self):
                print("E: Recipe folder", self.recipe_folder)
                print("E: Current folder:", os.getcwd())

                git = Git(self, self.recipe_folder)
                url, commit = git.get_url_and_commit()
                # We store the current url and commit in conandata.yml
                update_conandata(self, {"sources": {"commit": commit, "url": url}})
                self.output.info("URL: {}".format(url))
                self.output.info("COMMIT: {}".format(commit))

            def layout(self):
                self.folders.source = "project"

            def source(self):
                print("S: Current folder:", os.getcwd())

                git = Git(self)
                sources = self.conan_data["sources"]
                url = sources["url"]
                commit = sources["commit"]
                git.clone(url=url, target=".")
                print("CLONED:", os.getcwd(), os.listdir(os.getcwd()))

                git.checkout(commit=commit)
                self.output.info("MYCMAKE: {}".format(load(self, "project/src/CMakeLists.txt")))
                self.output.info("MYFILE: {}".format(load(self, "project/src/myfile.h")))
        """)

    def test_conanfile_subfolder(self):
        """
        A local repo, without remote, will have commit, but no URL
        """
        c = TestClient()
        c.save({"project/conanfile.py": self.conanfile,
                "project/src/CMakeLists.txt": "mycmakelists",
                "project/src/myfile.h": "myheader"})
        commit = c.init_git_repo()
        c.run("export project")

        print(c.out)
        assert "pkg/0.1: COMMIT: {}".format(commit) in c.out
        assert "pkg/0.1: URL: {}".format(c.current_folder.replace("\\", "/")) in c.out

        print("==============")
        c.run("create project")

        print(c.out)
        assert "pkg/0.1: MYCMAKE: mycmakelists" in c.out
        assert "pkg/0.1: MYFILE: myheader" in c.out

The output here of print("CLONED:", os.getcwd(), os.listdir(os.getcwd())) is:

CLONED: /private/var/folders/fl/nc7nppjn5b34rh5wj6tggw0w0000gp/T/tmptvb687ypconans/path with spaces/data/pkg/0.1/_/_/source/project ['project', '.git']

so inside .../project (which is the self.folders.source) the repo is checked out ../project/{.git,project}, which is weird, and IMO wrong.

Is my finding correct?

[gabyx]

So long story short:

repo/
  - .git
  - project
     - conanfile.py
     - src/allstufftobuild...
     - CMakeLists.txt

the git checkout should checkout inside conancache/../source and self.folders.layout then references the relative source folder relative to the checkout in conancache/../source, so project.

Also the doc clearly states that too: https://docs.conan.io/en/1.50/reference/conanfile/methods.html#self-folders

But the current working directory in the source(self) method will not include this subfolder, because it is intended to describe where the sources are after downloading (zip, git…) them, not to force where the sources should be.

[gabyx]

I ve seen there is a new, maybe that helps?

self.folders.root: To specify the relative path from the conanfile.py to the root of the project, in case the conanfile.py is in a subfolder and not in the project root. If defined, all the other paths will be relative to the project root, not to the location of the conanfile.py.

[gabyx]

@memsharded Could you reopen this.

[gabyx]

Feedback:

I still think something is wrong with the documentation:

Here https://docs.conan.io/en/latest/developing_packages/package_layout.html#example-export-sources-folder
it states that cwd in sources() is same as self.folders.source = 'src', and that is also what I experience in sources() when debugging, and that is not
what is stated here : https://docs.conan.io/en/latest/reference/conanfile/methods.html#self-folders

But the current working directory in the source(self) method will not include this subfolder, because it is intended to describe where the sources are after downloading (zip, git…) them, not to force where the sources should be. As well, the