update twoliter and remove tools

[webern]

TODO

• Figure out what's wrong with cargo make clean
• merge import tools history twoliter#37
• merge add bottlerocket tools to twoliter binary twoliter#47
• merge use cargo dist for binary releases twoliter#49
• Create a Twoliter tag v0.0.3
• Update this PR to use v0.0.3

Issue number:

Related to #2669 and bottlerocket-os/twoliter#14

Description of changes:

Use a binary release of Twoliter that has Bottlerocket's build tools embedded in it.

Testing done:

On both x86_64 and aarch64 build hosts:

• x86_64 build host
  • cargo make build
  • cargo make ami
  • cargo make repo
  • cargo test --help
  • cargo make clean
• aarch64 build host
  • cargo make build
  • cargo make ami
  • cargo make repo
  • cargo test --help
  • cargo make clean

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[webern]

https://github.com/bottlerocket-os/bottlerocket/compare/dae22a7d8fa74f9efc2cdf5415310ef8cd50d7dc..3ceff6ccf08c4075369f8a8bbabeb4bd979d05f7

^ has an important change. A script and logic flow suggested by Ben.

https://github.com/bottlerocket-os/bottlerocket/compare/043e10d749886990d7aba058969178a3b2017245..b49f5326c299eaf1e588e555a5b75ee0e9c60ebb

^ rebase on develop

[stmcginnis]

I have an aarch64 host handy, so decided to try some of those test cases.

cargo make build worked fine, no errors.

I then tried cargo make ami. That failed, without any good breadcrumbs pointing to the reason:

$ cargo make ami
[cargo-make] INFO - cargo make 0.37.0
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: ami
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: install-twoliter
Found twoliter v0.0.3-rc1 installed.
Skipping installation.
[cargo-make] INFO - Execute Command: "/home/stmcg/src/bottlerocket-os/bottlerocket/tools/twoliter/twoliter" "--log-level=info" "make" "ami" "--project-path=/home/stmcg/src/bottlerocket-os/bottlerocket/Twoliter.toml" "--cargo-home=/home/stmcg/src/bottlerocket-os/bottlerocket/.cargo" "--"
[cargo-make][1] INFO - Build File: /tmp/.tmplElH8h/Makefile.toml
[cargo-make][1] INFO - Task: ami
[cargo-make][1] INFO - Profile: development
[cargo-make][1] INFO - Running Task: setup
[cargo-make][1] INFO - Running Task: setup-build
[cargo-make][1] INFO - Running Task: fetch-sources
[cargo-make][1] INFO - Running Task: ami
/home/stmcg/src/bott : decoded 2147483648 bytes                                
/home/stmcg/src/bott : decoded 1073741824 bytes                                
Failed to build AMI: Infra.toml is missing aws.regions
[cargo-make][1] ERROR - Error while executing command, exit code: 1
[cargo-make][1] WARN - Build Failed.
Error: Command 'Command { std: "cargo" "make" "--disable-check-for-updates" "--makefile" "/tmp/.tmplElH8h/Makefile.toml" "--cwd" "/home/stmcg/src/bottlerocket-os/bottlerocket" "-e" "BUILDSYS_ROOT_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket" "-e" "BUILDSYS_ARCH=aarch64" "-e" "BUILDSYS_BUILD_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build" "-e" "BUILDSYS_PACKAGES_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/rpms" "-e" "BUILDSYS_STATE_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/state" "-e" "BUILDSYS_IMAGES_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/images" "-e" "BUILDSYS_TOOLS_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/tools" "-e" "BUILDSYS_SOURCES_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/sources" "-e" "BUILDSYS_SBKEYS_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/sbkeys" "-e" "BUILDSYS_SBKEYS_PROFILE=local" "-e" "BUILDSYS_TIMESTAMP=1694639008" "-e" "BUILDSYS_VERSION_BUILD=b49f5326" "-e" "BUILDSYS_RELEASE_CONFIG_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/Release.toml" "-e" "BUILDSYS_VERSION_IMAGE=1.15.0" "-e" "BUILDSYS_VARIANT=aws-k8s-1.24" "-e" "BUILDSYS_NAME=bottlerocket" "-e" "BUILDSYS_PRETTY_NAME=Bottlerocket OS" "-e" "BUILDSYS_SDK_NAME=bottlerocket" "-e" "BUILDSYS_SDK_VERSION=v0.34.1" "-e" "BUILDSYS_REGISTRY=public.ecr.aws/bottlerocket" "-e" "PUBLISH_EXPIRATION_POLICY_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/tools/pubsys/policies/repo-expiration/2w-2w-1w.toml" "-e" "PUBLISH_WAVE_POLICY_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/sources/updater/waves/default-waves.toml" "-e" "PUBLISH_INFRA_CONFIG_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/Infra.toml" "-e" "PUBLISH_REPO=default" "-e" "PUBLISH_TUFTOOL_VERSION=0.10.0" "-e" "PUBLISH_SSM_TEMPLATES_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/tools/pubsys/policies/ssm/defaults.toml" "-e" "BUILDSYS_LICENSES_CONFIG_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/Licenses.toml" "-e" "REPO_VALIDATE_TARGETS=true" "-e" "REPO_METADATA_EXPIRING_WITHIN=3 days" "-e" "VMWARE_IMPORT_SPEC_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/tools/pubsys/support/vmware/import_spec.template" "-e" "BUILDSYS_UPSTREAM_SOURCE_FALLBACK=false" "-e" "BUILDSYS_ALLOW_FAILED_LICENSE_CHECK=false" "-e" "BUILDSYS_UPSTREAM_LICENSE_FETCH=false" "-e" "BUILDSYS_JOBS=8" "-e" "AMI_DATA_FILE_SUFFIX=amis.json" "-e" "SSM_DATA_FILE_SUFFIX=ssm-params.json" "-e" "TESTSYS_TEST=quick" "-e" "CARGO_MAKE_DEFAULT_TESTSYS_KUBECONFIG_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/testsys.kubeconfig" "-e" "TESTSYS_STARTING_VERSION=v1.14.3" "-e" "TESTSYS_STARTING_COMMIT=764e37e4" "-e" "TESTSYS_TESTS_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/tests" "-e" "TESTSYS_TEST_CONFIG_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/Test.toml" "-e" "BUILDSYS_SDK_IMAGE=public.ecr.aws/bottlerocket/bottlerocket-sdk-aarch64:v0.34.1" "-e" "BUILDSYS_TOOLCHAIN=public.ecr.aws/bottlerocket/bottlerocket-toolchain-aarch64:v0.34.1" "-e" "CARGO_MAKE_CARGO_LIMIT_JOBS=--jobs 8" "-e" "CARGO_MAKE_CARGO_ARGS=--offline --locked" "-e" "BUILDSYS_OUTPUT_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/images/aarch64-aws-k8s-1.24" "-e" "BUILDSYS_VERSION_FULL=1.15.0-b49f5326" "-e" "BUILDSYS_NAME_VARIANT=bottlerocket-aws-k8s-1.24-aarch64" "-e" "BUILDSYS_NAME_VERSION=bottlerocket-1.15.0-b49f5326" "-e" "BUILDSYS_NAME_FULL=bottlerocket-aws-k8s-1.24-aarch64-1.15.0-b49f5326" "-e" "BUILDSYS_NAME_FRIENDLY=bottlerocket-aws-k8s-1.24-aarch64-v1.15.0" "-e" "BUILDSYS_VARIANT_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/images/aarch64-aws-k8s-1.24/1.15.0-b49f5326" "-e" "BUILDSYS_SBKEYS_PROFILE_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/sbkeys/local" "-e" "PUBLISH_REPO_ROOT_JSON=/home/stmcg/src/bottlerocket-os/bottlerocket/roles/default.root.json" "-e" "PUBLISH_REPO_KEY=/home/stmcg/src/bottlerocket-os/bottlerocket/keys/default.pem" "-e" "PUBLISH_REPO_BASE_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/repos" "-e" "PUBLISH_REPO_OUTPUT_DIR=/home/stmcg/src/bottlerocket-os/bottlerocket/build/repos/default/bottlerocket-1.15.0-b49f5326" "-e" "PUBLISH_AMI_NAME_DEFAULT=bottlerocket-aws-k8s-1.24-aarch64-v1.15.0-b49f5326" "-e" "BUILDSYS_KMOD_KIT=aws-k8s-1.24-aarch64-kmod-kit-v1.15.0.tar.xz" "-e" "BUILDSYS_KMOD_KIT_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/build/images/aarch64-aws-k8s-1.24/1.15.0-b49f5326/aws-k8s-1.24-aarch64-kmod-kit-v1.15.0.tar.xz" "-e" "BUILDSYS_OVA=bottlerocket-aws-k8s-1.24-aarch64-v1.15.0.ova" "-e" "BUILDSYS_OVA_PATH=/home/stmcg/src/bottlerocket-os/bottlerocket/build/images/aarch64-aws-k8s-1.24/1.15.0-b49f5326/bottlerocket-aws-k8s-1.24-aarch64-v1.15.0.ova" "-e" "BUILDSYS_OVF_TEMPLATE=/home/stmcg/src/bottlerocket-os/bottlerocket/variants/aws-k8s-1.24/template.ovf" "-e" "VMWARE_VM_NAME_DEFAULT=bottlerocket-aws-k8s-1.24-aarch64-v1.15.0-b49f5326" "-e" "BOOT_CONFIG_INPUT=/home/stmcg/src/bottlerocket-os/bottlerocket/bootconfig-input" "-e" "BOOT_CONFIG=/home/stmcg/src/bottlerocket-os/bottlerocket/bootconfig.data" "-e" "CARGO_MAKE_TESTSYS_KUBECONFIG_ARG=" "-e" "CARGO_MAKE_TESTSYS_ARGS=" "-e=CARGO_HOME=/home/stmcg/src/bottlerocket-os/bottlerocket/.cargo" "-e=TWOLITER_TOOLS_DIR=/tmp/.tmplElH8h" "ami", kill_on_drop: false }' was unsuccessful, exit code 1
[cargo-make] ERROR - Error while executing command, exit code: 1
[cargo-make] WARN - Build Failed.

[webern]

Failed to build AMI: Infra.toml is missing aws.regions

I often have this issue when my Infra.toml isn't right or I don't have one. Are you sure your invocation is correct?

[stmcginnis]

I often have this issue when my Infra.toml isn't right or I don't have one. Are you sure your invocation is correct?

I don't have one. Are we no longer allowing that?

[webern]

I often have this issue when my Infra.toml isn't right or I don't have one. Are you sure your invocation is correct?

I don't have one. Are we no longer allowing that?

I have always had this issue with cargo make ami, as long as I can remember. You have to pass PUBLISH_REGIONS in the absence of an Infra.toml

# You can also set PUBLISH_REGIONS to override the list of regions from
# Infra.toml for AMI and SSM commands; it's a comma-separated list like

[webern]

https://github.com/bottlerocket-os/bottlerocket/compare/b49f5326c299eaf1e588e555a5b75ee0e9c60ebb..88988e61d4c0bbf13adfb0fa603c5232af6223c4

Copy commented code and restore script modes to 755.

[stmcginnis]

Ah, I always us an alias to run these, so forgot I had that in there. That said, the error when omitting that pre-twoliter is a lot more helpful.

Rather than copy/pasting the "to test" commands, I used by alias to run the build and ami targets. Both completed successfully.

[webern]

Ah, I always us an alias to run these, so forgot I had that in there. That said, the error when omitting that pre-twoliter is a lot more helpful.

Rather than copy/pasting the "to test" commands, I used by alias to run the build and ami targets. Both completed successfully.

I think the error message is the same, but I will get rid of that spew about the cargo make command. Hopefully that will make it easier to spot the "real" error.

[webern]

https://github.com/bottlerocket-os/bottlerocket/compare/d589150a840bc027899268d7fbb87a702685bbbf..cf77c9b3bc0898e94acb9ba15e6ea8d8d6fed8b3

Hopefully the last push. Use the v0.0.3 tag and I don't think I need to specify +nightly anymore because I have rust-toolchain.toml in Twoliter.