Various layer and test fixes for C++ client #2001

kacf · 2023-11-10T12:03:38Z

Plus a merge with master. Please start the review after the merge commit.

The app-updates Update Module provides an API for orchestrated application level updates, and currently two user implementation for docker-compose and k8s. The recipe installs the Update Module and dependencies for deploying such docker-compose and k8s artifacts. The implementation is in shell script and therefore architecture agnostic. Changelog: Title Ticket: MEN-6084 Signed-off-by: Fabio Tranchitella <fabio.tranchitella@northern.tech>

feat: add recipe for the Application Update Module

Bumps [pytest-html](https://github.com/pytest-dev/pytest-html) from 3.2.0 to 4.0.2. - [Release notes](https://github.com/pytest-dev/pytest-html/releases) - [Changelog](https://github.com/pytest-dev/pytest-html/blob/master/docs/changelog.rst) - [Commits](pytest-dev/pytest-html@v3.2.0...4.0.2) --- updated-dependencies: - dependency-name: pytest-html dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.0 to 7.4.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.4.0...7.4.2) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.12.3 to 3.12.4. - [Release notes](https://github.com/tox-dev/py-filelock/releases) - [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst) - [Commits](tox-dev/filelock@3.12.3...3.12.4) --- updated-dependencies: - dependency-name: filelock dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

chore: dependabot updates for client acceptance tests

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.16 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.16...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

…/pip/tests/acceptance/urllib3-1.26.17 chore: bump urllib3 from 1.26.16 to 1.26.17 in /tests/acceptance

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>

…/pip/tests/acceptance/urllib3-1.26.18

The IMAGE_FEATURE "splash" is unconditionally added to the build if the meta-mender-demo layer is included. This is at least a bad practice, and should be considered a bug for many types of devices that do not have displays. Generally, a layer.conf should not modify the build just by being included. Changelog: Title Ticket: None Signed-off-by: TheYoctoJester <jester@theyoctojester.info>

In order to save time for test pipeline runs, the images to be built should be as small as possible. To facilitate this, settings for MENDER_STORAGE_TOTAL_SIZE_MB_DEFAULT and IMAGE_OVERHEAD_FACTOR have been placed in the layer.conf of meta-mender-demo. This way they affect all builds that have the layer included, even those that do not build a mender-style image. Therefore, move these variable assignments into the test scripts. Changelog: Title Ticket: None Signed-off-by: TheYoctoJester <jester@theyoctojester.info>

…-layer Clean demo layer

coreutils 9.0 introduces a new behavior in the "install" tool. When used to install a sparse file, it now looks for holes and may insert new holes. Using "mv" instead of install ensures that images do not lose significant information. This fix is beneficial to the use of bmaptool to write images to a device, especially if the image contains a FAT filesystem. Changelog: Title Ticket: None Signed-off-by: Pascal Hache <hacpa@touchtunes.com>

Changelog: Add recipe for mender-connect 2.1.1 Changelog: Add recipe for mender-artifact 3.10.2 Signed-off-by: Fabio Tranchitella <fabio.tranchitella@northern.tech>

…ct-2.1.1 Mender connect 2.1.1

fix: mv image files instead of install, to avoid corrupting the image.

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.2 to 7.4.3. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.4.2...7.4.3) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [filelock](https://github.com/tox-dev/py-filelock) from 3.12.4 to 3.13.1. - [Release notes](https://github.com/tox-dev/py-filelock/releases) - [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst) - [Commits](tox-dev/filelock@3.12.4...3.13.1) --- updated-dependencies: - dependency-name: filelock dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

…ndencies Test dependencies updates

mender-test-bot · 2023-11-10T12:03:48Z

@kacf, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

my commands and options

You can trigger a pipeline on multiple prs with:

mentioning me and start pipeline --pr mender/127 --pr mender-connect/255

You can start a fast pipeline, disabling full integration tests with:

mentioning me and start pipeline --fast

You can trigger GitHub->GitLab branch sync with:

mentioning me and sync

You can cherry pick to a given branch or branches with:

mentioning me and:

 cherry-pick to:
 * 1.0.x
 * 2.0.x

oleorhagen

Good job here!

oleorhagen · 2023-11-20T09:05:23Z

meta-mender-core/recipes-mender/mender-artifact/mender-artifact_git.bb

@@ -29,6 +29,8 @@ def mender_branch_from_preferred_version(d):
        # If the preferred version is some kind of version, use the branch name
        # for that one (1.0.x style).
        return match.group(0) + "x"
+    elif version.endswith("-git%"):


Q: When are we unifying this functionality somewhere?

My "unification muscle" twitched a bit as well. I don't promise to do it now, but let me try and see if it causes problems.

Bah! There is a SHA in there which is specific to each recipe. It can be unified, but I'm leaving it alone for now. A task for another day.

At least we tried :)

meta-mender-core/recipes-mender/mender-client/mender-client_3.5.0.bb

meta-mender-core/recipes-mender/mender-client/mender-client-cpp.inc

oleorhagen · 2023-11-20T09:14:04Z

meta-mender-demo/recipes-mender/mender-client/mender-client-append.inc


-MENDER_UPDATE_CONTROL_MAP_EXPIRATION_TIME_SECONDS:mender-client-install = "90"
-MENDER_UPDATE_CONTROL_MAP_BOOT_EXPIRATION_TIME_SECONDS:mender-client-install = "45"
+MENDER_UPDATE_CONTROL_MAP_EXPIRATION_TIME_SECONDS:mender-update-install = "90"


Might as well remove these or?

They are still applicable to the old client versions. This file should not be be picked up by the C++ client, since it has the mender-client name in it.

lluiscampos

Quite a bit of work here 💪

meta-mender-commercial/conditional/mender-monitor/mender-monitor.inc

meta-mender-core/recipes-mender/mender-client/README-releasing.md

The build component is now called `mender` instead of `mender-client`, because it contains the two independent `mender-auth` and `mender-update` runtime components. Other recipes should either use `DEPENDS = "mender"` for a build time dependency, or `RDEPENDS:${PN} = "mender-auth | mender-update"` for a runtime dependency. In addition the `mender-client-install` class has been renamed to `mender-install`. Changelog: Commit Ticket: None Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

It needs to use submodules, and we also need to avoid weak assignment because poky itself sets it to empty, so we have to override it. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

This didn't break anything, but just to be future proof. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Even though there is some merit to sharing things, it's getting to a point where it's quite messy and confusing. Even though it's more code it's better to make a clean split, and apply specific fixes to each without worrying about compatibility with the other. This can also make it easier to remove Golang support later. The downside is that some fixes may need to be applied twice. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Also remove the get_jwt_token test, since the new client does not automatically fetch a token anymore, so calling fetch_jwt_token is mandatory. GetJwtToken is also tested in the next test where we call the fetching first, so we rely on that from now on. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Changelog: The `mender-install` and `mender-install-ubi` Mender features have been removed. Do not use them in `MENDER_FEATURES` variables, but instead one of the alternatives (see Features under Yocto section in the Mender documentation). BREAKING CHANGE: The `mender-install` and `mender-install-ubi` Mender features have been removed Ticket: None Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

The default package config string really does not need to be feature scoped, since it is the default regardless. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

This is more in style with how Yocto package selection works. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Those versions don't exist in the repo anymore. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

We don't support logging directly to syslog in the new client, only indirectly through systemd. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

We do this by splitting mender client bbappend file into old and new name. Almost half of the content is different, so it does not really make sense to consolidate. Note the change from `RDEPENDS:${PN}` to `RDEPENDS:mender-auth`, which is the real fix here. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

mender-test-bot · 2023-11-21T08:06:03Z

Merging these commits will result in the following changelog entries:

Changelogs

meta-mender (test_fixes)

New changes in meta-mender since feature-c++-client:

Bug Fixes

move test specific settings from demo layer to tests
mv image files instead of install, to avoid corrupting the image.
remove splash IMAGE_FEATURE from demo layer

Features

add recipe for Application Updates Module
(MEN-6084)

Other

Add recipe for mender-artifact 3.10.2
Add recipe for mender-connect 2.1.1
The mender-install and mender-install-ubi
Mender features have been removed. Do not use them in
MENDER_FEATURES variables, but instead one of the alternatives (see
Features under Yocto section in the Mender documentation).
refac: Standardize on new names that follow the C++ client rewrite.

The build component is now called mender instead of mender-client,
because it contains the two independent mender-auth and
mender-update runtime components. Other recipes should either use
DEPENDS = "mender" for a build time dependency, or
RDEPENDS:${PN} = "mender-auth | mender-update" for a runtime
dependency.

In addition the mender-client-install class has been renamed to
mender-install.

kacf · 2023-11-21T09:52:56Z

Note to self: This is the official pipeline. Any other results is due to testing acceptance tests, which may make the PR red, but should not block it.

tranchitella and others added 20 commits September 26, 2023 12:53

Merge pull request mendersoftware#1979 from tranchitella/men-6084

1a7fe87

feat: add recipe for the Application Update Module

Merge pull request mendersoftware#1983 from lluiscampos/dependaoct

b0b5005

chore: dependabot updates for client acceptance tests

Merge pull request mendersoftware#1984 from mendersoftware/dependabot…

fb61bae

…/pip/tests/acceptance/urllib3-1.26.17 chore: bump urllib3 from 1.26.16 to 1.26.17 in /tests/acceptance

Merge pull request mendersoftware#1985 from mendersoftware/dependabot…

b486c30

…/pip/tests/acceptance/urllib3-1.26.18

Merge pull request mendersoftware#1986 from TheYoctoJester/clean-demo…

dd0f848

…-layer Clean demo layer

chore: Add recipes for release Mender 3.6.3

c5c88f4

Changelog: Add recipe for mender-connect 2.1.1 Changelog: Add recipe for mender-artifact 3.10.2 Signed-off-by: Fabio Tranchitella <fabio.tranchitella@northern.tech>

Merge pull request mendersoftware#1993 from tranchitella/mender-conne…

d49835c

…ct-2.1.1 Mender connect 2.1.1

Merge pull request mendersoftware#1992 from hacpa/mv_images

eef66ef

fix: mv image files instead of install, to avoid corrupting the image.

Merge pull request mendersoftware#1999 from lluiscampos/nov-test-depe…

988158a

…ndencies Test dependencies updates

kacf force-pushed the test_fixes branch 3 times, most recently from 8a0dfb6 to 87dad84 Compare November 15, 2023 09:51

kacf requested review from oleorhagen and TheYoctoJester November 15, 2023 10:40

kacf force-pushed the test_fixes branch from 87dad84 to 565ce4c Compare November 15, 2023 13:17

Merge remote-tracking branch 'origin/master' into feature-c++-client

da030e0

kacf force-pushed the test_fixes branch from 565ce4c to 1efb26a Compare November 16, 2023 08:34

oleorhagen approved these changes Nov 20, 2023

View reviewed changes

lluiscampos approved these changes Nov 20, 2023

View reviewed changes

meta-mender-commercial/conditional/mender-monitor/mender-monitor.inc Outdated Show resolved Hide resolved

meta-mender-core/recipes-mender/mender-client/README-releasing.md Outdated Show resolved Hide resolved

kacf added 15 commits November 21, 2023 08:53

chore: Fix incorrect SRC_URI for mender client.

3356b3a

It needs to use submodules, and we also need to avoid weak assignment because poky itself sets it to empty, so we have to override it. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Make it possible to use non-master branch names with git recipe.

188dfad

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Make sure we add to dependencies, not replace.

bef3d60

This didn't break anything, but just to be future proof. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Add several missing paths to mender-config's files.

0d8bafb

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Remove unnecessary package config.

ded8670

The default package config string really does not need to be feature scoped, since it is the default regardless. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Switch ca-certificates dependency to RRECOMMENDS.

5fa77fa

This is more in style with how Yocto package selection works. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Remove outdated component selection snippets.

5b459ef

Those versions don't exist in the repo anymore. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

test: Remove syslog test.

92b7432

We don't support logging directly to syslog in the new client, only indirectly through systemd. Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

chore: Install all update modules and make dependent on PACKAGECONFIG.

a4e565f

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

test: Adapt test_update_modules.py to new C++ client.

e0d565a

Signed-off-by: Kristian Amlie <kristian.amlie@northern.tech>

kacf force-pushed the test_fixes branch from 1efb26a to e0d565a Compare November 21, 2023 08:05

kacf merged commit b0599f9 into mendersoftware:feature-c++-client Nov 21, 2023
14 of 17 checks passed

kacf deleted the test_fixes branch November 21, 2023 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various layer and test fixes for C++ client #2001

Various layer and test fixes for C++ client #2001

kacf commented Nov 10, 2023

mender-test-bot commented Nov 10, 2023

oleorhagen left a comment

oleorhagen Nov 20, 2023

kacf Nov 20, 2023

kacf Nov 21, 2023 •

edited

oleorhagen Nov 21, 2023

oleorhagen Nov 20, 2023

kacf Nov 20, 2023

lluiscampos left a comment

mender-test-bot commented Nov 21, 2023 •

edited by jira bot

kacf commented Nov 21, 2023

Various layer and test fixes for C++ client #2001

Various layer and test fixes for C++ client #2001

Conversation

kacf commented Nov 10, 2023

mender-test-bot commented Nov 10, 2023

oleorhagen left a comment

Choose a reason for hiding this comment

oleorhagen Nov 20, 2023

Choose a reason for hiding this comment

kacf Nov 20, 2023

Choose a reason for hiding this comment

kacf Nov 21, 2023 • edited

Choose a reason for hiding this comment

oleorhagen Nov 21, 2023

Choose a reason for hiding this comment

oleorhagen Nov 20, 2023

Choose a reason for hiding this comment

kacf Nov 20, 2023

Choose a reason for hiding this comment

lluiscampos left a comment

Choose a reason for hiding this comment

mender-test-bot commented Nov 21, 2023 • edited by jira bot

Changelogs

meta-mender (test_fixes)

Bug Fixes

Features

Other

kacf commented Nov 21, 2023

kacf Nov 21, 2023 •

edited

mender-test-bot commented Nov 21, 2023 •

edited by jira bot