[SURE-5668] Fleet BundleDeployments does not reflect new labels from Bundles

[atsai1220]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

If we update an existing Fleet Bundle with additional labels then we can observe that BundleDeployments do not have the newly added labels unless we either:

• Delete the Bundle
• Manually patch the BundleDeployment in the cluster's Fleet namespace in the local cluster

This is problematic as we iterate on our Fleet deployments and leveraging the dependsOn feature. The dependent BundleDeployment will say this until we patch the BundleDeployment object with the new label.

No bundles matching labels x=y in namespace fleet-default

Expected Behavior

We expect BundleDeployments to be a reflection of Bundle objects, including from labels and annotations updates.

Steps To Reproduce

1. Create GitRepo A object with labels a=b inside fleet.yaml.
2. Observe Bundle and BundleDeployment objects created with labels a=b
3. Update GitRepo A with a new label x=y by updating fleet.yaml
4. Create second GtiRepo B object that dependsOn the GtiRepo A with the label x=y inside of fleet.yaml.
5. See that the cluster's BundleDeployment object from GitRepo B display this error:

No bundles matching labels x=y in namespace fleet-default

6. Manually patch BundleDeployment object from GitRepo A with label x=y (or delete Bundle objecet from GitRepo A but this is not ideal as this will cause the application to be reinstalled).

Environment

- Architecture: amd
- Fleet Version: 0.5.0
- Cluster:
  - Provider: RKE, RKE2, K3S
  - Kubernetes Version: v1.21.x

Logs

No response

Anything else?

No response

[manno]

I don't understand the reproduction steps.

Create GitRepo A object with labels a=b inside fleet.yaml.

I think labels are only used in fleet.yaml as selectors.

I'm under the impression that the propagation of labels from a gitrepo resource to a bundle was not actually implemented before 0.6.x-alpha. This feature only existed in the docs.

See #1007 for the implementation.

Edit: Actually the docs mention a labels field in the fleet.yaml, but I don't think it's part of the CRD?

# Bundles to match by label.  The labels are defined in the fleet.yaml
# labels field or from the GitRepo metadata.labels field

[manno]

This is the true fleet.yaml struct, which does support labels: https://github.com/rancher/fleet/blob/master/pkg/bundlereader/read.go#L129-L135

[manno]

When I try to set an additional label for step 3, I get this error in the status of the gitrepo:

time="2022-12-16T11:46:01Z" level=fatal msg="error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field bundleMeta.labels of type string"

y is expanded to boolean true in YAML. I usethis fleet.yaml instead:

labels:
  x: "y"

And the additional label shows up on the bundle.

[kourosh7]

The issue is the label is not added to the BundleDeployment object, and this blocks the dependsOn in the second GitRepo

[manno]

I could reproduce this (https://github.com/manno/fleet-experiments/tree/gitrepo-labels). The labels do not show up on the bundledeployment. However the dependsOn code in the agent looks at bundledeployments.

The labels are only copied over, from the bundle to the bundledeployment, the first time. Even if a later update changes the content and the labels, the labels won't update on the bundledeployment.

[raulcabello]

@rancher/qa

Additionnal QA

Problem

Updating labels in a Bundle does not change the labels in the BundleDeployments. BundleDeployment gets the label from the relevant Bundle when it is created, however it is not modified if labels are changed in the Bundle

Solution

Update BundleDeployments labels each time a Bundle changes.

Testing

Engineering Testing

integration tests added here

Manual Testing

• Create a GitRepo that contains a label a=b inside a fleet.yaml
• Verify Bundle and BundleDeployment are created with labels a=b
• Update the GitRepo with a new label x=y by updating the fleet.yaml
• Verify both Bundle and BundleDeployment contains the label x=y

[kkaempf]

See also #1488 (which should be fixed by this)

[sbulage]

QA TEST PLAN

Scenarios

Scenario	Test Case
1	As an admin user, add GitRepo having labels key1=value1 in fleet.yaml file. Verify that Bundle and BudleDeployment has same labels assigned to it.
2	Change the value of the label assigned in previous test no 1 in fleet.yaml to value2 similar to key1=value2. Verify that Bundle and BudleDeployment gets updated with updated value.
3	Remove the label from the fleet.yaml file. Verify that assigned label is not present on both Bundle and BudleDeployment.
4	Add new label newkey1=newvalue1 in fleet.yaml to . Verify that Bundle and BudleDeployment are having new label with newkey1=newvalue1.

[deniseschannon]

Can we also add the following test scenarios after it's been deployed?

1. Removing a label
2. Adding a new label

[sbulage]

From above test cases,

• First and second test cases is working as expected.

• But when remove label from the fleet.yaml file it didn't get reflected on bundles.

• Another, when add complete new label via fleet.yaml and update GitRepo it shows new label along with the old label on the bundles.

@raulcabello can you please take a look at it? Thanks 😄

[raulcabello]

@sbulage good catch! When a label is removed is not reflected in the Bundle or BundleDeployment. That's a different issue than this bug. This bug is about labels changes not being copied from Bundles into BundleDeployments. I would suggest creating a separate bug for this.

Please check labels for BundleDeployments too. I don't think this can be checked in the Rancher UI, so you need to use something like kubectl for that.

[sbulage]

@sbulage good catch! When a label is removed is not reflected in the Bundle or BundleDeployment. That's a different issue than this bug. This bug is about labels changes not being copied from Bundles into BundleDeployments. I would suggest creating a separate bug for this.

Sure, thanks for taking a look into it and I will file separate issue/bug.

Please check labels for BundleDeployments too. I don't think this can be checked in the Rancher UI, so you need to use something like kubectl for that.

Yes, I will confirm and update here.

[sbulage]

Verified first two test cases mentioned above QE Test Plan comment.

TEST RESULT

Scenarios

Scenario	Test Case	Result
1	As an admin user, add GitRepo having labels key1=value1 in fleet.yaml file. Verify that Bundle and BudleDeployment has same labels assigned to it.	✅
2	Change the value of the label assigned in previous test no 1 in fleet.yaml to value2 similar to key1=value2. Verify that Bundle and BudleDeployment gets updated with updated value.	✅
3	Remove the label from the fleet.yaml file. Verify that assigned label is not present on both Bundle and BudleDeployment.	X
4	Add new label newkey1=newvalue1 in fleet.yaml to . Verify that Bundle and BudleDeployment are having new label with newkey1=newvalue1.	X

REPRO STEPS

Scenario 1

1. Created a github repository: https://github.com/sbulage/test-fleet

• Above github repository has fleet.yaml which contains key1: value1 labels in it.

2. Navigated back to Rancher > Continuous Delivery.
3. Created a GitRepo using https://github.com/sbulage/test-fleet.
4. Verified that new label is added to Bundle and BundleDeployment.

Scenario 2

1. Using GitHub repo: https://github.com/sbulage/test-fleet
2. Update the label to key1: *value2
3. Navigated back to Rancher > Continuous Delivery.
4. Wait for the GitRepo to sync with GitHub repo to pull latest changes.
5. Verified that modified label value is being updated on Bundle and BundleDeployment.

Scenario 3

• Remove label is not working issue/bug link --> [Fleet] After removing labels from fleet.yaml not reflecting on Bundle or BundleDeployment. #1528.

Scenario 4

• New label after removing old one is being added by following the steps from Scenario 1.
• However, it shows both the labels i.e. old and new labels on the Bundle (Due to bug in scenario 3).

[sbulage]

Re-open: Understanding the corner cases around this issue.

[kkaempf]

Dicussed during weekly meeting and agreed to close it.