question: how to develop based on qe-compiler

[Zhaoyilunnn]

Hi Team,

Currently I still cannot figure out how to make this project dependent on qe-compiler. qe-compiler is an excellent project, I believe many users would like to develop their own full compilers based on this framework. Thus it would be helpful if you could provide some instructions on how to achieve this. I am also very willing to contribute if it requires more effort to make this project a conan package that other projects can depend on.

Here are two ways I have tried but failed. Note that I can seamlessly build qe-compiler locally, but I just cannot build a conan package of it and then make my own project dependent on qe-compiler.

01 Manually

This is done by copying conandata.yml and conanfile.py to the path ./conan/qec and create a conan_deps.sh, which essentially calls conan export

Direclly use the original conanfile.py does not work because it sets exports_sources = "*", resulting in a non-empty directory ~/.conan/data/qss-compiler/0.1.0/qss-compiler/stable/export_source/. Then conan will copy conandata.yml, conanfile.py, conanmanifest.txt into ~/.conan/data/qss-compiler/0.1.0/qss-compiler/stable/source/. But conan cannot delete conandata.yml, resulting in a non-empty source dir, preventing cloning source file of qe-compiler.

Steps

Manually create a conan package and export it to local cache and then build it

./conan_deps.sh

Then install qe-compiler through conan

mkdir build && cd build
cd build
conan install .. --build=outdated -pr:h default -pr:b default

This method finally results in

CMake Error at /root/miniconda3/envs/quos-dev/lib/python3.10/site-packages/cmake/data/share/cmake-3.23/Modules/CMakeDetermineSystem.cmake:130 (message):
  Could not find toolchain file: conan_toolchain.cmake
Call Stack (most recent call first):
  CMakeLists.txt:46 (enable_language)


CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_ASM_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

02 Conan

Then I tried using conan create to create a local conan package.

In the root directory of qe-compiler

cd build
conan create .. zhaoyilun/qe-compiler --build=outdated -pr:h default -pr:b default -e QSSC_VERSION=0.1.0

Initially, I met following error

fatal: not a git repository (or any of the parent directories): .git
-- Root CMake project Git directory (used for version):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/root/miniconda3/envs/quos-dev/lib/python3.10/site-packages/setuptools_scm/__init__.py", line 148, in get_version
    _version_missing(config)
  File "/root/miniconda3/envs/quos-dev/lib/python3.10/site-packages/setuptools_scm/__init__.py", line 108, in _version_missing
    raise LookupError(
LookupError: setuptools-scm was unable to detect version for /root/.conan/data/qss-compiler/0.1.0/zhaoyilun/qe-compiler/build/d28c692a69e3ccd7dc8cd629cfb34b8fd367905b.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

The root cause is that conan will not copy .git directory into the local cache, resulting in the error of setuptools_scm. After checking the CMakeLists.txt, I found that I can explicitly set VERSION_STRING to avoid using setuptools_scm.

After fixing the above error, it results in a cyclic dependency error

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "QSSPythonSources.Dialects.quir" of type INTERFACE_LIBRARY
    depends on "QSSPythonModules.sources.QSSPythonSources.Dialects.quir" (strong)
  "QSSPythonModules.sources.QSSPythonSources.Dialects.quir" of type UTILITY
    depends on "QSSPythonSources.Dialects.quir" (strong)
At least one of these targets is not a STATIC_LIBRARY.  Cyclic dependencies are allowed only among static libraries.
CMake Generate step failed.  Build files cannot be regenerated correctly.

System info

PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[Zhaoyilunnn]

I made some changes to existing config files main...Zhaoyilunnn:qe-compiler:zhaoyilun-local-conan-package

Then a local conan package can be built through conan create . zhaoyilun/qe-compiler --build=outdated -pr:h default -pr:b default -o qss-compiler:pythonlib=False

[taalexander]

Hi @Zhaoyilunnn, thank you for writing up this issue. The qe-compiler is still in its infancy of being an opensource project and we certainly have a lot of work to do in migrating documentation that needs to be resumed.

Note that I can seamlessly build qe-compiler locally, but I just cannot build a conan package of it and then make my own project dependent on qe-compiler.

In particular, we have not attempted this path to development yet, but are very interested in this. Internally, we are using the project as a submodule with integration through the Cmake build system.

I made some changes to existing config files main...Zhaoyilunnn:qe-compiler:zhaoyilun-local-conan-package
Then a local conan package can be built through conan create . zhaoyilun/qe-compiler --build=outdated -pr:h default -pr:b default -o qss-compiler:pythonlib=False

We would be very interested if you could make a draft PR demonstrating this work so that we could discuss how it might be incorporated in-tree.

[taalexander]

An obfuscated version of our top-level CMakeLists to integrate the project

cmake_minimum_required(VERSION 3.25)
project(target-compiler)
set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/build_tools/cmake ${CMAKE_MODULE_PATH})

set(QSSC_SYSTEMS_DIR
        "${CMAKE_CURRENT_SOURCE_DIR}/targets/systems"
        CACHE STRING
        "The directory containing the common systems"
        )

option(QSSC_WITH_MOCK_TARGET "Available targets include the built-in mock target" ON)
option(QSSC_WITH_SYSTEM_CONFIG "Use system_config to retrieve configuration for targets" OFF)
option(QSSC_BUILD_DOCS "Build the documentation for the compiler" ON)

# List of directories containing compiler targets. Subtargets should be
# listed first so that they are built before top-level targets
set(QSSC_TARGET_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget;
        CACHE PATH
        "List of directories containing compiler targets."
        )

set(QSSC_TARGET_TEST_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget/test;
        CACHE PATH
        "List of directories containing compiler LIT test suites."
        )

set(QSSC_TARGET_PYTHON_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget/python_lib/
        CACHE PATH
        "List of directories containing compiler target python additions."
        )

# Build type
if (NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE Debug)
endif()
set(CMAKE_COMPILE_WARNING_AS_ERROR ON)

# Here is the submodule which we integrate through
# the Cmake build system with
add_subdirectory(qe-compiler)

In this way our internal project contains our targets in a very similar way to https://github.com/openqasm/qe-compiler/tree/main/targets/systems/mock with a top-level cmake file which we use to generate internal binaries. The rough project structure is

• target-compiler/
  • qe-compiler/ (submodule)
  • targets/systems/YourTarget/
  • CMakeLists.txt

We could be very interested in enabling the pathway to remove the qe-compiler submodule and replace it with a Conan dependency.

[Zhaoyilunnn]

An obfuscated version of our top-level CMakeLists to integrate the project

cmake_minimum_required(VERSION 3.25)
project(target-compiler)
set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/build_tools/cmake ${CMAKE_MODULE_PATH})

set(QSSC_SYSTEMS_DIR
        "${CMAKE_CURRENT_SOURCE_DIR}/targets/systems"
        CACHE STRING
        "The directory containing the common systems"
        )

option(QSSC_WITH_MOCK_TARGET "Available targets include the built-in mock target" ON)
option(QSSC_WITH_SYSTEM_CONFIG "Use system_config to retrieve configuration for targets" OFF)
option(QSSC_BUILD_DOCS "Build the documentation for the compiler" ON)

# List of directories containing compiler targets. Subtargets should be
# listed first so that they are built before top-level targets
set(QSSC_TARGET_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget;
        CACHE PATH
        "List of directories containing compiler targets."
        )

set(QSSC_TARGET_TEST_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget/test;
        CACHE PATH
        "List of directories containing compiler LIT test suites."
        )

set(QSSC_TARGET_PYTHON_DIRS
        ${CMAKE_CURRENT_SOURCE_DIR}/targets/systems/YourTarget/python_lib/
        CACHE PATH
        "List of directories containing compiler target python additions."
        )

# Build type
if (NOT CMAKE_BUILD_TYPE)
    set(CMAKE_BUILD_TYPE Debug)
endif()
set(CMAKE_COMPILE_WARNING_AS_ERROR ON)

# Here is the submodule which we integrate through
# the Cmake build system with
add_subdirectory(qe-compiler)

In this way our internal project contains our targets in a very similar way to https://github.com/openqasm/qe-compiler/tree/main/targets/systems/mock with a top-level cmake file which we use to generate internal binaries. The rough project structure is

• target-compiler/

  • qe-compiler/ (submodule)
  • targets/systems/YourTarget/
  • CMakeLists.txt

We could be very interested in enabling the pathway to remove the qe-compiler submodule and replace it with a Conan dependency.

@taalexander Thanks a lot for your example CMakeLists.txt, actually I gave up making qe-compiler a conan package and adopted a similar top-level CMakeLists.txt ^_^

I am still in the progress of finding the correct way to make it a Conan package. I will make a PR once I figure it out

[taalexander]

Thank you @Zhaoyilunnn, happy to help however you might like. In general, as you're making progress I'd love to chat to understand better what you are trying to do and how to enable it.