Mixing of source code and build results during the build process #350
Comments
|
What is you concern about safety? That sources could be packaged to a binary package, and you don't want to distribute the source code?. I see the issue, and it could be a good idea if possible. |
|
I meant 'safe' as in without worrying about side effects, not in the context of security. Most CMake projects will not be designed for in-source builds, many CMake projects will depend on the build directory being empty or at least not have the entire source there, and some CMake projects even flat out refuse to build when a trying to perform an in-source build (e.g. set(CMAKE_DISABLE_IN_SOURCE_BUILD ON)). If you want to package such projects with a conanfile.py you have to perform quite a bit of trickery to make it work. I have tried it by making a _build subdirectory within the "build/" directory that contains the source code. The problem in this case is that Conan will place the conanbuildinfo.cmake file in the Having a flag available in conanfile.py that allows you to specify whether the source code has to be copied in the build directory would solve this problem and still allow for in-source builds. Having an extra variable available to specify the location of the source directory (e.g. So for instance with the Hello example from the documentation: If |
|
I agree about preparing a "build" subfolder can be tricky, and also in the include of conanbuildinfo.cmake in the CMakeLists.txt, this is a real example: It seems we could do it better. Just an idea to discuss: |
|
I suggest giving the responsibility to the |
CMake is a generator system, producing Make, Visual Studio, Ninja, etc build files. So these all support out of source builds if CMake is able to produce them. Admittedly, doing out of source builds with Make is not particularly common. I think the solution by @lasote would probably works. Although personally I believe it would be nicer if copying the source code to the build folder is an optional step - something that can be prevented if not required. Building a library like LLVM/Clang would now result in several hundred MB being copied, unnecessarily, for every different different build type. Being able to say "don't bother copying, I'm building out of source" would really help. |
|
I think I misunderstood the issue, I was not thinking about taking the sources from the source folder, just about separating the copied sources in the build folder from the actual build folder.
I understand that the copy takes time and space, but it is usually just a fraction compared with the build times or the disk space the builds use. Such space is temporary, can be removed easily ( |
|
I've recently started using conan for cmake builds and had a look at this. It turns out to be very straightforward to do, only a few lines in conans/client/{installer.py,packager.py}. I added a conanfile flag "no_copy_source" to suppress the copy and an additional copy_source instance of FileCopier to facilitate packaging. My use case is probably reasonably popular - I use cmake for all builds out of source and the install target to push the files to the package. I also don't use anything in the conanbuildinfo.cmake script except the module path as I use cmake exported targets in the package, hence I don't bother setting package_info(). An example conanfile.py: import os
from conans import ConanFile, CMake, tools
class LibreSSLConan(ConanFile):
name = "libressl"
version = "2.5.2"
url = "csl-conan-recipes.git"
license = "MIT"
generators = "cmake"
build_policy = "missing"
zip_folder = name + "-" + version
zip_file = zip_folder + ".tar.gz"
description = "LibreSSL, OpenSSL compatible library."
settings = "os", "arch", "compiler" # , "build_type"
exports_sources = ["CMakeLists.txt", zip_file, "cmake/*"]
exports = ["FindOpenSSL.cmake", "FindLibreSSL.cmake"]
no_copy_source = True
def source(self):
tools.unzip(self.zip_file)
os.unlink(self.zip_file)
for (dp, dns, fns) in os.walk("cmake"):
op = os.path.join(self.zip_folder, os.path.relpath(dp, "cmake"))
for fn in fns:
os.replace(os.path.join(dp,fn), os.path.join(op,fn))
def build(self):
# use Ninja for speed
cmake = CMake(self, generator="Ninja")
# absolute path to actual source folder
cmake_source_dir = os.path.join(self.source_folder, self.zip_folder)
# cmake defines as a dict
cmake_defs = {"CMAKE_INSTALL_PREFIX" : self.package_folder}
# build both variants
if cmake.is_multi_configuration:
cmake.configure(source_dir=cmake_source_dir, build_dir="Debug_Release", defs=cmake_defs)
for config in ("Debug", "Release"):
cmake.build(args=["--config",config], target="install")
else:
for config in ("Debug", "Release"):
cmake_defs["CMAKE_BUILD_TYPE"] = config
cmake.configure(source_dir=cmake_source_dir, build_dir=config, defs=cmake_defs)
cmake.build(target="install")
def package(self):
# copy find package
self.copy_source("FindOpenSSL.cmake")
self.copy_source("FindLibreSSL.cmake")
def package_info(self):
pass
# Sneaky self-registration script!
if __name__ == "__main__":
from conans.conan import main
main("export capitis/stable".split(" "))Anyway, here are the diffs. I have never submitted a pull request but if there is any interest please let me know and I'll try to figure it out: |
|
Thanks very much for contributing your implementation. The main issue is that it is adding a bit of complexity, and could have special corner-cases and unexpected behavior (imagine someone copy & paste a recipe with no_copy_sources, then using the typical in-source .configure & make). But we will review it and consider it for next 0.23. If it is accepted, then yes, I would encourage to submit a Pull request, so you get credit for it. Cheers! |
|
Hi @bjmacdonald, I am starting to review, and liking the idea. It results that defining a |
|
I had to make a small change to check if the build folder already exists in installer.py, please see below for diff. Otherwise it seems to work fine with the latest develop branch from the repository. I also added the members to conan_file.py but still get a pylint warning about copy_source not being callable... I will have to look closer at the local edition workflows - one thing that has been nagging me is the capability to easily configure a conan local build if I need to step into 3rd party libraries with the debugger. |
|
Cool thank you! Will take it into account, and keep updated here. |
|
I have done some initial work on this in #1259. I needed to go with a different approach, to account for an homogeneous model:
In this way, the local edition workflows are fully supported. Note than when a package is under development in user space, it is necessary to support having up to 3 folders coming from 3 repos: the package recipe, the package source code and the temporary build folder. As another advantage, it might allow users to adopt it, just setting Still need much more testing and completing the joint proposal. |
|
Hi @WimK, the |
While creating source code packages I ran into a problem that I believe makes it unnecessarily difficult to create packages in a safe way. This mainly has to do with the way Conan currently sets up the build directory during a build (for instance after doing
conan install .. --build)What Conan currently does is to copy the contents of the package's "source" directory (in the .conan/data location) into a configuration specific build directory. After the copy has been performed the "build" method of the conanfile.py is invoked, with the build directory as the default working location for the "build" method. In other words the build directory is the same as the source directory.
In many cases, especially when packaging from a source project, this means that the build directory will have both the full source as well as the build results mixed in the same directory. This is also the case, for instance, in the "hello" library example from the "Packaging from the source project"-chapter of the manual. The build results (i.e. directories lib, bin and all the files from the CMake process) will be mixed with hello.cpp and hello.h, all in the same directory.
For the hello example this does not cause any problems. However, in other cases this mixing can have undesired consequences. For instance when certain files are generated during the build process and overwrite original source or project files, or if the build process runs certain code depending on the presence or absence of files in the build directory. For this reason the hello example is not safe for general usage.
Of course, having conanfile.py allows you to make any modification to the build process to prevent problems (e.g. move the build directory source code to another location before the build starts). This requires a thorough understand of Conan and the way packages are being build. While at the same time preventing, or at least make it much more risky, one of the most common use cases (i.e. having a simply conanfile.py in the root of your library project and running only basic cmake commands in build()).
My suggestion would be to copy the source code during the build step to another directory that is separate from the actual build location and make the location of this source directory available in the build() method so it can be used in the build step.
Edit: An alternative suggestion would be to make the copy of the source to build step optional by setting a flag in conanfile.py (e.g. copy_source_for_build=False) and to make the location of the base source directory available in the build() method.
The text was updated successfully, but these errors were encountered: