libcxx: fails in configure when create-spdx is enabled #721

kwavnet · 2023-01-11T11:59:07Z

Describe the bug
when create-spdx class is inherited globally, libcxx fails to configure.
This happens after an upgrade from c728c3f to bc8388c in kirkstone branch.

As to be seen in the attached log configure can’t find certain paths in the llvm-project-source shared folder.
I just checked they really do not exist in this case.

If the create-spdx tasks are deleted (the workaround I chose) or create-spdx isn’t part of the configuration, everything works fine even with the upgrade to kirkstone head.

Interestingly, even though many use the shared sources, only libcxx fails

DISTRO: poky
MACHINE: qemux86-64
LAYERS: core, poky, yocto-bsp, meta-clang

To Reproduce
Steps to reproduce the behavior:

clone poky kirkstone 4.0.6
clone meta-clang kirkstone
INHERIT += "create-spdx" in local.conf
llvm-project-source -c cleanall (in case this isn’t a new workspace)
build libcxx

Expected behavior
configure doesn’t fail when create-spdx tasks are enabled

Desktop (please complete the following information):

OS: Ubuntu 20.04 based container

Additional context

ERROR: libcxx-14.0.3-r0 do_configure: ExecutionError('/path/tmp/work/distro/libcxx/14.0.3-r0/temp/run.do_configure.139389', 1, None, None)
ERROR: Logfile of failure stored in: /path/tmp/work/distro/libcxx/14.0.3-r0/temp/log.do_configure.139389
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['/sources/meta-clang/recipes-devtools/clang/clang-cross_git.bb:do_populate_sysroot', '/sources/meta-clang/recipes-devtools/clang/compiler-rt_git.bb:do_populate_sysroot', '/sources/poky/meta/recipes-core/glibc/glibc_2.35.bb:do_populate_sysroot', '/sources/poky/meta/recipes-devtools/cmake/cmake-native_3.22.3.bb:do_populate_sysroot', '/sources/poky/meta/recipes-devtools/gcc/gcc-runtime_11.3.bb:do_populate_sysroot', '/sources/poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_populate_sysroot', 'virtual:native:/sources/poky/meta/recipes-devtools/ninja/ninja_1.10.2.bb:do_populate_sysroot', 'virtual:native:/sources/poky/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', 'virtual:native:/sources/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/sources/poky/meta/recipes-devtools/python/python3_3.10.8.bb:do_populate_sysroot']
| NOTE: Installed into sysroot: ['quilt-native', 'patch-native', 'attr-native']
| NOTE: Skipping as already exists in sysroot: ['clang-cross-x86_64', 'compiler-rt', 'glibc', 'cmake-native', 'gcc-runtime', 'ninja-native', 'pseudo-native', 'python3-native', 'linux-libc-headers', 'clang-native', 'binutils-cross-x86_64', 'libtool-native', 'libgcc', 're2c-native', 'bzip2-native', 'zstd-native', 'zlib-native', 'xz-native', 'curl-native', 'ncurses-native', 'libnsl2-native', 'sqlite3-native', 'openssl-native', 'gdbm-native', 'libffi-native', 'readline-native', 'util-linux-libuuid-native', 'libtirpc-native', 'binutils-native', 'libxml2-native', 'libedit-native', 'swig-native', 'gnu-config-native', 'texinfo-dummy-native', 'flex-native', 'gettext-minimal-native', 'perl-native', 'elfutils-native', 'libpcre-native', 'm4-native', 'make-native', 'libmicrohttpd-native', 'libarchive-native', 'libgcrypt-native', 'gnutls-native', 'lzo-native', 'libgpg-error-native', 'libcap-native', 'gmp-native', 'nettle-native', 'libunistring-native', 'libidn2-native']
| DEBUG: sed -e 's:^[^/]*/:/path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot-native/:g' /path/tmp/sysroots-components/x86_64/quilt-native/fixmepath | xargs sed -i -e 's:FIXMESTAGINGDIRTARGET:/path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot:g; s:FIXMESTAGINGDIRHOST:/path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot-native:g' -e 's:FIXME_PSEUDO_SYSROOT:/path/tmp/sysroots-components/x86_64/pseudo-native:g' -e 's:FIXME_HOSTTOOLS_DIR:/path/tmp/hosttools:g' -e 's:FIXME_PKGDATA_DIR:/path/tmp/pkgdata/qemux86-64:g' -e 's:FIXME_PSEUDO_LOCALSTATEDIR:/path/tmp/work/distro/libcxx/14.0.3-r0/pseudo/:g' -e 's:FIXME_LOGFIFO:/path/tmp/work/distro/libcxx/14.0.3-r0/temp/fifo.139389:g'
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_configure
| -- The C compiler identification is Clang 14.0.3
| -- The CXX compiler identification is Clang 14.0.3
| -- The ASM compiler identification is Clang with GNU-like command-line
| -- Found assembler: /path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot-native/usr/bin/x86_64-simplecoredistro-linux/x86_64-simplecoredistro-linux-clang
| -- Detecting C compiler ABI info
| -- Detecting C compiler ABI info - done
| -- Check for working C compiler: /path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot-native/usr/bin/x86_64-simplecoredistro-linux/x86_64-simplecoredistro-linux-clang - skipped
| -- Detecting C compile features
| -- Detecting C compile features - done
| -- Detecting CXX compiler ABI info
| -- Detecting CXX compiler ABI info - done
| -- Check for working CXX compiler: /path/tmp/work/distro/libcxx/14.0.3-r0/recipe-sysroot-native/usr/bin/x86_64-simplecoredistro-linux/x86_64-simplecoredistro-linux-clang++ - skipped
| -- Detecting CXX compile features
| -- Detecting CXX compile features - done
| -- No build type selected, default to Debug
| CMake Warning at CMakeLists.txt:89 (message):
|   Using LLVM_ENABLE_PROJECTS=libcxx is deprecated now, please use
|   -DLLVM_ENABLE_RUNTIMES=libcxx or see the instructions at
|   https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
| 
| 
| CMake Warning at CMakeLists.txt:89 (message):
|   Using LLVM_ENABLE_PROJECTS=libcxxabi is deprecated now, please use
|   -DLLVM_ENABLE_RUNTIMES=libcxxabi or see the instructions at
|   https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
| 
| 
| CMake Warning at CMakeLists.txt:89 (message):
|   Using LLVM_ENABLE_PROJECTS=libunwind is deprecated now, please use
|   -DLLVM_ENABLE_RUNTIMES=libunwind or see the instructions at
|   https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
| 
| 
| -- bolt project is disabled
| -- clang project is disabled
| -- clang-tools-extra project is disabled
| -- compiler-rt project is disabled
| -- cross-project-tests project is disabled
| -- libc project is disabled
| -- libclc project is disabled
| -- libcxx project is enabled
| -- libcxxabi project is enabled
| CMake Error at CMakeLists.txt:136 (message):
|   LLVM_ENABLE_PROJECTS requests libcxxabi but directory not found:
|   /path/tmp/work-shared/llvm-project-source-14.0.3-r0/git/llvm/../libcxxabi
| 
| 
| -- Configuring incomplete, errors occurred!
| See also "/path/tmp/work/distro/libcxx/14.0.3-r0/build/CMakeFiles/CMakeOutput.log".
| See also "/path/tmp/work/distro/libcxx/14.0.3-r0/build/CMakeFiles/CMakeError.log".
| WARNING: exit code 1 from a shell command.
NOTE: recipe libcxx-14.0.3-r0: task do_configure: Failed
ERROR: Task (/sources/meta-clang/recipes-devtools/clang/libcxx_git.bb:do_configure) failed with exit code '1'
NOTE: recipe compiler-rt-14.0.3-r0: task do_package_write_rpm: Succeeded
NOTE: recipe compiler-rt-14.0.3-r0: task do_create_spdx: Succeeded

The text was updated successfully, but these errors were encountered:

kwavnet · 2023-01-11T12:42:09Z

this just happened in another build

Summary: 3 tasks failed:
  virtual:native:/sources/meta-clang/recipes-devtools/clang/clang_git.bb:do_create_spdx
  /sources/meta-clang/recipes-devtools/clang/clang-cross_git.bb:do_create_spdx
  /sources/meta-clang/recipes-devtools/clang/llvm-project-source.bb:do_unpack

seems like a race to me

kwavnet · 2023-01-11T12:54:24Z

this just happened in another build

Summary: 3 tasks failed:
  virtual:native:/sources/meta-clang/recipes-devtools/clang/clang_git.bb:do_create_spdx
  /sources/meta-clang/recipes-devtools/clang/clang-cross_git.bb:do_create_spdx
  /sources/meta-clang/recipes-devtools/clang/llvm-project-source.bb:do_unpack

seems like a race to me

this might be unrelated to the original issue, but I think all recipes that include common-source.inc need to add do_create_spdx[depends] += "${PN}:do_configure" as the task’s do_configure depends on the unpack of the llvm shared source tree, but create_spdx runs somewhere between do_unpack and do_package

kraj · 2023-01-11T18:32:43Z

@witekio-kwe thanks for the report. I wonder how it works ok for gcc recipes, there we use similar method of shared workdir

kwavnet · 2023-01-12T05:44:12Z

def process_sources(d):
    pn = d.getVar('PN')
    assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
    if pn in assume_provided:
        for p in d.getVar("PROVIDES").split():
            if p != pn:
                pn = p
                break

    # glibc-locale: do_fetch, do_unpack and do_patch tasks have been deleted,
    # so avoid archiving source here.
    if pn.startswith('glibc-locale'):
        return False
    if d.getVar('PN') == "libtool-cross":
        return False
    if d.getVar('PN') == "libgcc-initial":
        return False
    if d.getVar('PN') == "shadow-sysroot":
        return False

    # We just archive gcc-source for all the gcc related recipes
    if d.getVar('BPN') in ['gcc', 'libgcc']:
        bb.debug(1, 'spdx: There is bug in scan of %s is, do nothing' % pn)
        return False

    return True

@kraj that’s the code - gcc exceptions are hardcoded :-(

kraj · 2023-01-12T06:22:48Z

@witekio-kwe thats explains thanks for digging. Perhaps you can propose such an exception for clang related recipes too ?
otherwise submit a patch to meta-clang to express the dependency as plan B

kwavnet · 2023-01-12T06:24:59Z

@witekio-kwe thats explains thanks for digging. Perhaps you can propose such an exception for clang related recipes too ? otherwise submit a patch to meta-clang to express the dependency as plan B

Let me thoroughly test my local patches and then I will draft a PR

kwavnet · 2023-01-12T11:51:26Z

@kraj I found why the configure task fails (had the same kind of issue now with clang-native and compiler-rt).

Normally upstream code does unpack the sources into a temporary dir under WORDKIR, unless the sources are marked as shared - for this the following snippet code is used

def is_work_shared_spdx(d):
    return bb.data.inherits_class('kernel', d) or ('work-shared' in d.getVar('WORKDIR'))

in all the recipes affected only S and B are altered to a folder under work-shared but WORKDIR remains intact.
Thus this check fails us here and every of the create_spdx tasks fires unpack task. This one is somehow empty in this scope but it fires in before the cleandirs flag operations of do_unpack, thus deleting everything under S and B - leading to exactly the error seen in the log above.

The assumption that WORKDIR needs to have work-shared might be error prone but it’s not wrong - at the same time just setting S and B to other paths seems to be reasonable as well.

I helped myself with the following additions to common-source.inc

do_create_spdx[depends] += "llvm-project-source-${PV}:do_patch"

# shared workdir detection fails as not WORKDIR is altered but S and B
# override for common-source.inc based recipes
def is_work_shared_spdx(d):
    return True

but this feels like a work around, so I’m hesitant to put that into a PR.

That brings me to my question, can we set WORKDIR to to a path under work-shared? or is there a reason to keep it untouched?

A second option would be remove S and B from the cleandirs of do_unpack forcefully, but this sounds even more hackish to me.

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Closes kraj#721

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Relates to kraj#721

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Relates to kraj#721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Closes kraj#721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Relates to #721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Closes kraj#721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Relates to kraj#721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com> (cherry picked from commit b7c0dcc)

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Relates to #721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com> (cherry picked from commit b7c0dcc)

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Closes kraj#721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

- do_create_spdx does have to wait till the shared source tree is properly populated - override the spdx class code detection to determine if sources are under work-shared. the core code does only look at WORKDIR, which isn’t overridden in every case, more likely only S and B are touched, which make the spdx class code assume that these are per-workdir sources and starts to remove the shared sources via bitbake’s cleandirs settings. Avoid that by let the code always handle recipes including common-source as shared sources. Closes #721 Signed-off-by: Konrad Weihmann <kweihmann@witekio.com>

kwavnet mentioned this issue Jan 16, 2023

Fix/spdx for clang #726

Merged

3 tasks

kraj closed this as completed in 36d2b65 Jan 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libcxx: fails in configure when create-spdx is enabled #721

libcxx: fails in configure when create-spdx is enabled #721

kwavnet commented Jan 11, 2023

kwavnet commented Jan 11, 2023

kwavnet commented Jan 11, 2023

kraj commented Jan 11, 2023

kwavnet commented Jan 12, 2023

kraj commented Jan 12, 2023

kwavnet commented Jan 12, 2023

kwavnet commented Jan 12, 2023

libcxx: fails in configure when create-spdx is enabled #721

libcxx: fails in configure when create-spdx is enabled #721

Comments

kwavnet commented Jan 11, 2023

kwavnet commented Jan 11, 2023

kwavnet commented Jan 11, 2023

kraj commented Jan 11, 2023

kwavnet commented Jan 12, 2023

kraj commented Jan 12, 2023

kwavnet commented Jan 12, 2023

kwavnet commented Jan 12, 2023