building single packages stopped working on develop

[foersleo]

Platform I'm building on:

AWS c5.metal instance (x86_64)

What I expected to happen:

Build of a single package for a given variant, for example kernel-5.10 for variant aws-k8s-1.23

What actually happened:

Getting an error where thread main of cargo (?) is panicking:

$ cargo make -e BUILDSYS_ARCH=x86_64 -e BUILDSYS_VARIANT=aws-k8s-1.23 -e PACKAGE=kernel-5_10 build-package
[cargo-make] INFO - cargo make 0.35.13
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: build-package
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: install-twoliter
[cargo-make] INFO - Execute Command: "/home/fedora/src/bottlerocket/tools/bin/twoliter" "--log-level=info" "make" "build-package" "--project-path=/home/fedora/src/bottlerocket/Twoliter.toml" "--cargo-home=/home/fedora/src/bottlerocket/.cargo"
[cargo-make][1] INFO - Build File: /tmp/.tmpidTdh2/Makefile.toml
[cargo-make][1] INFO - Task: build-package
[cargo-make][1] INFO - Profile: development
[cargo-make][1] INFO - Running Task: check-cargo-version
[cargo-make][1] INFO - Running Task: setup
[cargo-make][1] INFO - Running Task: setup-build
[cargo-make][1] INFO - Running Task: fetch-sdk
[cargo-make][1] INFO - Running Task: fetch-sources
[cargo-make][1] INFO - Running Task: build-tools
[cargo-make][1] INFO - Running Task: tuftool
[cargo-make][1] INFO - Running Task: publish-setup-tools
[cargo-make][1] INFO - Running Task: publish-setup
13:21:42 [INFO] No infra config at '/home/fedora/src/bottlerocket/Infra.toml' - using local roles/keys
[cargo-make][1] INFO - Running Task: fetch-toolchain
[cargo-make][1] INFO - Running Task: fetch-vendored
[cargo-make][1] INFO - Running Task: fetch-licenses
Skipping fetching licenses
[cargo-make][1] INFO - Running Task: build-package
thread 'main' panicked at 'activated_features for invalid package: features did not find PackageId { name: "kernel-5_10", version: "0.1.0", source: "/home/fedora/src/bottlerocket/packages/kernel-5.10" } NormalOrDev', src/tools/cargo/src/cargo/core/resolver/features.rs:321:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[cargo-make][1] ERROR - Error while executing command, exit code: 101
[cargo-make][1] WARN - Build Failed.
Error: Command 'Command { std: "cargo" "make" "--disable-check-for-updates" "--makefile" "/tmp/.tmpidTdh2/Makefile.toml" "--cwd" "/home/fedora/src/bottlerocket" "-e" "BUILDSYS_ARCH=x86_64" "-e" "BUILDSYS_VARIANT=aws-k8s-1.23" "-e" "BUILDSYS_ROOT_DIR=/home/fedora/src/bottlerocket" "-e" "BUILDSYS_BUILD_DIR=/home/fedora/src/bottlerocket/build" "-e" "BUILDSYS_PACKAGES_DIR=/home/fedora/src/bottlerocket/build/rpms" "-e" "BUILDSYS_STATE_DIR=/home/fedora/src/bottlerocket/build/state" "-e" "BUILDSYS_IMAGES_DIR=/home/fedora/src/bottlerocket/build/images" "-e" "BUILDSYS_TOOLS_DIR=/home/fedora/src/bottlerocket/tools" "-e" "BUILDSYS_SOURCES_DIR=/home/fedora/src/bottlerocket/sources" "-e" "BUILDSYS_SBKEYS_DIR=/home/fedora/src/bottlerocket/sbkeys" "-e" "BUILDSYS_SBKEYS_PROFILE=local" "-e" "BUILDSYS_TIMESTAMP=1693315292" "-e" "BUILDSYS_VERSION_BUILD=013a6c84" "-e" "BUILDSYS_RELEASE_CONFIG_PATH=/home/fedora/src/bottlerocket/Release.toml" "-e" "BUILDSYS_VERSION_IMAGE=1.15.0" "-e" "BUILDSYS_NAME=bottlerocket" "-e" "BUILDSYS_PRETTY_NAME=Bottlerocket OS" "-e" "BUILDSYS_SDK_NAME=bottlerocket" "-e" "BUILDSYS_SDK_VERSION=v0.34.0" "-e" "BUILDSYS_REGISTRY=public.ecr.aws/bottlerocket" "-e" "PUBLISH_EXPIRATION_POLICY_PATH=/home/fedora/src/bottlerocket/tools/pubsys/policies/repo-expiration/2w-2w-1w.toml" "-e" "PUBLISH_WAVE_POLICY_PATH=/home/fedora/src/bottlerocket/sources/updater/waves/default-waves.toml" "-e" "PUBLISH_INFRA_CONFIG_PATH=/home/fedora/src/bottlerocket/Infra.toml" "-e" "PUBLISH_REPO=default" "-e" "PUBLISH_TUFTOOL_VERSION=0.10.0" "-e" "PUBLISH_SSM_TEMPLATES_PATH=/home/fedora/src/bottlerocket/tools/pubsys/policies/ssm/defaults.toml" "-e" "BUILDSYS_LICENSES_CONFIG_PATH=/home/fedora/src/bottlerocket/Licenses.toml" "-e" "REPO_VALIDATE_TARGETS=true" "-e" "REPO_METADATA_EXPIRING_WITHIN=3 days" "-e" "VMWARE_IMPORT_SPEC_PATH=/home/fedora/src/bottlerocket/tools/pubsys/support/vmware/import_spec.template" "-e" "BUILDSYS_UPSTREAM_SOURCE_FALLBACK=false" "-e" "BUILDSYS_ALLOW_FAILED_LICENSE_CHECK=false" "-e" "BUILDSYS_UPSTREAM_LICENSE_FETCH=false" "-e" "BUILDSYS_JOBS=8" "-e" "AMI_DATA_FILE_SUFFIX=amis.json" "-e" "SSM_DATA_FILE_SUFFIX=ssm-params.json" "-e" "TESTSYS_TEST=quick" "-e" "CARGO_MAKE_DEFAULT_TESTSYS_KUBECONFIG_PATH=/home/fedora/src/bottlerocket/testsys.kubeconfig" "-e" "TESTSYS_STARTING_VERSION=v1.14.3" "-e" "TESTSYS_STARTING_COMMIT=764e37e4" "-e" "TESTSYS_TESTS_DIR=/home/fedora/src/bottlerocket/tests" "-e" "TESTSYS_TEST_CONFIG_PATH=/home/fedora/src/bottlerocket/Test.toml" "-e" "BUILDSYS_SDK_IMAGE=public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.34.0" "-e" "BUILDSYS_TOOLCHAIN=public.ecr.aws/bottlerocket/bottlerocket-toolchain-x86_64:v0.34.0" "-e" "CARGO_MAKE_CARGO_LIMIT_JOBS=--jobs 8" "-e" "CARGO_MAKE_CARGO_ARGS=--offline --locked" "-e" "BUILDSYS_OUTPUT_DIR=/home/fedora/src/bottlerocket/build/images/x86_64-aws-k8s-1.23" "-e" "BUILDSYS_VERSION_FULL=1.15.0-013a6c84" "-e" "BUILDSYS_NAME_VARIANT=bottlerocket-aws-k8s-1.23-x86_64" "-e" "BUILDSYS_NAME_VERSION=bottlerocket-1.15.0-013a6c84" "-e" "BUILDSYS_NAME_FULL=bottlerocket-aws-k8s-1.23-x86_64-1.15.0-013a6c84" "-e" "BUILDSYS_NAME_FRIENDLY=bottlerocket-aws-k8s-1.23-x86_64-v1.15.0" "-e" "BUILDSYS_VARIANT_DIR=/home/fedora/src/bottlerocket/build/images/x86_64-aws-k8s-1.23/1.15.0-013a6c84" "-e" "BUILDSYS_SBKEYS_PROFILE_DIR=/home/fedora/src/bottlerocket/sbkeys/local" "-e" "PUBLISH_REPO_ROOT_JSON=/home/fedora/src/bottlerocket/roles/default.root.json" "-e" "PUBLISH_REPO_KEY=/home/fedora/src/bottlerocket/keys/default.pem" "-e" "PUBLISH_REPO_BASE_DIR=/home/fedora/src/bottlerocket/build/repos" "-e" "PUBLISH_REPO_OUTPUT_DIR=/home/fedora/src/bottlerocket/build/repos/default/bottlerocket-1.15.0-013a6c84" "-e" "PUBLISH_AMI_NAME_DEFAULT=bottlerocket-aws-k8s-1.23-x86_64-v1.15.0-013a6c84" "-e" "BUILDSYS_KMOD_KIT=aws-k8s-1.23-x86_64-kmod-kit-v1.15.0.tar.xz" "-e" "BUILDSYS_KMOD_KIT_PATH=/home/fedora/src/bottlerocket/build/images/x86_64-aws-k8s-1.23/1.15.0-013a6c84/aws-k8s-1.23-x86_64-kmod-kit-v1.15.0.tar.xz" "-e" "BUILDSYS_OVA=bottlerocket-aws-k8s-1.23-x86_64-v1.15.0.ova" "-e" "BUILDSYS_OVA_PATH=/home/fedora/src/bottlerocket/build/images/x86_64-aws-k8s-1.23/1.15.0-013a6c84/bottlerocket-aws-k8s-1.23-x86_64-v1.15.0.ova" "-e" "BUILDSYS_OVF_TEMPLATE=/home/fedora/src/bottlerocket/variants/aws-k8s-1.23/template.ovf" "-e" "VMWARE_VM_NAME_DEFAULT=bottlerocket-aws-k8s-1.23-x86_64-v1.15.0-013a6c84" "-e" "BOOT_CONFIG_INPUT=/home/fedora/src/bottlerocket/bootconfig-input" "-e" "BOOT_CONFIG=/home/fedora/src/bottlerocket/bootconfig.data" "-e" "CARGO_MAKE_TESTSYS_KUBECONFIG_ARG=" "-e" "CARGO_MAKE_TESTSYS_ARGS=" "-e" "CARGO_HOME=/home/fedora/src/bottlerocket/.cargo" "build-package", kill_on_drop: false }' was unsuccessful, exit code 1
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.

How to reproduce the problem:

Try to build a single package for a variant like, fails for any package I have tried:

$ cargo make -e BUILDSYS_VARIANT=aws-k8s-1.23 -e PACKAGE=kernel-5_10 build-package

The offending commit where it all starts breaking seems to be 3a4dadc
I am not quite sure what is happening. Trying to chase the path where this is breaking is slow for me as I am not deeply familiar with cargo intricacies.

As a saving grace: Building full variants is not affected by this issue. Confirmed with @markusboehme that this is not just my dev host, and is affecting his dev environment as well.

[foersleo]

There seems to be some similar (not same) issues to be discussed on cargo upstream. I am however not quite sure on the interactions between cargo, buildsys, and twoliter.

Example issues upstream:

rust-lang/cargo#12108
rust-lang/cargo#12358

[foersleo]

Running the build with RUST_BACKTRACE=1 gives some more information that may help pin point this down (see below). However, trying to chase through this myself I believe I do not have enough context on how cargo works internally. I am not quite sure how the resolver version plays into this either.

thread 'main' panicked at 'activated_features for invalid package: features did not find PackageId { name: "kernel-5_10", version: "0.1.0", source: "/home/fedora/src/bottlerocket/packages/kernel-5.10" } NormalOrDev', src/tools/cargo/src/cargo/core/resolver/features.rs:321:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/result.rs:1687:5
   3: <cargo::ops::cargo_compile::unit_generator::UnitGenerator>::proposals_to_units
   4: <cargo::ops::cargo_compile::unit_generator::UnitGenerator>::generate_root_units
   5: cargo::ops::cargo_compile::create_bcx
   6: cargo::ops::cargo_compile::compile_ws
   7: cargo::ops::cargo_compile::compile
   8: cargo::commands::build::exec
   9: cargo::cli::main
  10: cargo::main

[markusboehme]

Running cargo make -e PACKAGE=libfoo build-package results in running cargo build --manifest-path variants/Cargo.toml --package libfoo. This reproduces the error a little faster.

The key to understanding the error lies in the new Cargo feature resolver v2 separating activated features by both package and where/when it is going to be used. When checking for activated features of a package, Cargo takes both into account. It bails when it unexpectedly cannot find a combination of {package, where to use it}.

The following is my current understanding of the matter, but it is to be taken with a grain proper amount of salt:
We are using dependencies and build-dependencies in the Cargo manifests for specifying runtime and buildtime dependencies both for the target architecture which is a bit different from what Cargo uses these for. Specifying a dependency in the build-dependencies section of a manifest, I believe, activates features in this dependency package as HostDep. When:

• a variant is built, the roots of all (RPM) package trees to pull into the final image are likewise defined as build-dependencies (e.g. in variants/metal-dev/Cargo.toml). When resolving the feature set for dependencies upon building these top-level packages (already known as HostDeps), Cargo looks for and finds their (and their transitive) dependencies also as HostDep.
• when just building a single package it is looking for activated features in a NormalOrDev context for the single target to be built (e.g. libfoo)--while still knowing libfoo and all of its dependencies as HostDep only.

I'm not sure what would a good way to resolve this would look like. Perhaps reverting to the feature resolver v1 and accepting warnings is the lesser evil for now, so we can avoid the problem by having the unified feature namespace.

[stmcginnis]

This appears to have been resolved with #3376 and future work captured with bottlerocket-os/twoliter#45 - closing this issue.