Add manifest bundle support #59

morvencao · 2024-03-07T09:24:00Z

ref: https://issues.redhat.com/browse/OCM-6367

morvencao · 2024-03-07T09:31:37Z

test results:

# make test-integration
CGO_ENABLED=1 GOEXPERIMENT=boringcrypto go install -ldflags="" ./cmd/maestro
OCM_ENV=testing gotestsum --format short-verbose -- -p 1 -ldflags -s -v -timeout 1h  \
		./test/integration
I0307 09:30:50.606715  386039 framework.go:76] Initializing testing environment
I0307 09:30:50.887469  386039 framework.go:161] Using Mock OCM Authz Client
I0307 09:30:50.890513  386039 framework.go:199] Disabling Sentry error reporting
I0307 09:30:50.898980  386039 grpc_server.go:74] Serving gRPC service without TLS at 8090
I0307 09:30:50.899286  386039 api_server.go:151] Serving without TLS at 8000
PASS test/integration.TestConsumerGet (5.06s)
PASS test/integration.TestConsumerPost (0.23s)
PASS test/integration.TestConsumerPaging (0.35s)
PASS test/integration.TestControllerRacing (1.76s)
PASS test/integration.TestControllerReconcile (1.38s)
PASS test/integration.TestControllerSync (1.30s)
PASS test/integration.TestPulseServer (5.31s)
PASS test/integration.TestResourceGet (0.19s)
PASS test/integration.TestResourcePost (5.20s)
PASS test/integration.TestResourcePatch (0.35s)
PASS test/integration.TestResourcePaging (0.39s)
PASS test/integration.TestResourceListSearch (0.26s)
PASS test/integration.TestUpdateResourceWithRacingRequests (2.29s)
PASS test/integration.TestResourceFromGRPC (2.28s)
E0307 09:31:17.272039  386039 logger.go:129]   Unable to get all maestro instances: pq: relation "server_instances" does not exist
E0307 09:31:17.273825  386039 logger.go:129]   Unable to get all maestro instances: pq: relation "server_instances" does not exist
E0307 09:31:17.275218  386039 logger.go:129]   Unable to get all maestro instances: pq: relation "server_instances" does not exist
E0307 09:31:17.276482  386039 logger.go:129]   Unable to get all maestro instances: pq: relation "server_instances" does not exist
PASS test/integration

DONE 14 tests in 30.024s

skeeey · 2024-03-08T02:08:25Z

cmd/maestro/server/grpc_server.go

-			},
-		},
+
+	for _, obj := range objs {


check/filter event type matching?

do you mean single or bundle match?

yes, you can get the event data type from the resync request event, then you can only response the type matched, right?

thank! done.

skeeey · 2024-03-08T02:09:33Z

pkg/api/resource_types.go

+	resourceVersionInt := int64(0)
+	resourceVersion, err := cloudeventstypes.ToString(evtExtensions[cetypes.ExtensionResourceVersion])
+	if err != nil {
+		resourceVersionIntVal, err := cloudeventstypes.ToInteger(evtExtensions[cetypes.ExtensionResourceVersion])


only use ToInteger?

right, we now ensure the resource version is integer.
Updated all the occurrences.

skeeey · 2024-03-08T02:27:42Z

And cloud you also append the e2e test result?

morvencao · 2024-03-08T04:46:37Z

e2e test results:

# make e2e-test
ginkgo --output-dir="/home/oasis/workspace/openshift-online-maestro/test/e2e/report" --json-report=report.json --junit-report=report.xml \
/home/oasis/workspace/openshift-online-maestro/test/e2e/pkg -- -consumer_id="34617c63-5f52-4ae0-8428-35daf6b9a45f" \
-api-server=https://127.0.0.1:30080 -kubeconfig=/home/oasis/workspace/openshift-online-maestro/test/e2e/.kubeconfig
Ginkgo detected a version mismatch between the Ginkgo CLI and the version of Ginkgo imported by your packages:
  Ginkgo CLI Version:
    2.6.0
  Mismatched package versions found:
    2.13.2 used by pkg

  Ginkgo will continue to attempt to run but you may see errors (including flag
  parsing errors) and should either update your go.mod or your version of the
  Ginkgo CLI to match.

  To install the matching version of the CLI run
    go install github.com/onsi/ginkgo/v2/ginkgo
  from a path that contains a go.mod file.  Alternatively you can use
    go run github.com/onsi/ginkgo/v2/ginkgo
  from a path that contains a go.mod file to invoke the matching version of the
  Ginkgo CLI.

  If you are attempting to test multiple packages that each have a different
  version of the Ginkgo library with a single Ginkgo CLI that is currently
  unsupported.

Running Suite: End-to-End Test Suite - /home/oasis/workspace/openshift-online-maestro/test/e2e/pkg
============================================================================================
Random Seed: 1709873066

Will run 5 of 5 specs
•••••

Ran 5 of 5 Specs in 4.149 seconds
SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 8.021418251s
Test Suite Passed

morvencao · 2024-03-15T01:54:43Z

/assign @qiujian16 @clyang82

qiujian16 · 2024-03-18T06:56:54Z

pkg/api/resource_types.go

+
+// DecodeStatus converts a CloudEvent JSONMap representation of a resource status
+// into resource status (map[string]interface{}).
+func DecodeStatus(status datatypes.JSONMap) (map[string]interface{}, error) {


So when we store in DB, we store a cloudEvent structure right?

yes, we now store spec and status in cloudevent format in DB now. This function is used to decode cloudevent status to resource status in openapi format to restful client.

qiujian16 · 2024-03-18T06:57:54Z

pkg/db/migrations/202311151850_add_resources.go

@@ -13,6 +13,7 @@ func addResources() *gormigrate.Migration {
 		Source     string         `gorm:"index"`
 		ConsumerID string         `gorm:"index"`
 		Version    int            `gorm:"not null"`
+		Type       string         `gorm:"index"`


can we add some comments here? It is not clear to me what data structure is stored in DB.

added comments.

morvencao · 2024-03-18T13:16:45Z

@qiujian16 An example of resource status in DB:

# select jsonb_pretty(resources.status) from resources;
                              jsonb_pretty
--------------------------------------------------------------------
  {
      "id": "0fd2b15f-e4d5-4ecb-a5fb-455284d27e2b",
      "data": {
          "status": {
              "conditions": [
                  {
                      "type": "Applied",
                      "reason": "AppliedManifestComplete",
                      "status": "True",
                      "message": "Apply manifest complete",
                      "lastTransitionTime": "2024-03-18T12:57:58Z"
                  },
                  {
                      "type": "Available",
                      "reason": "ResourceAvailable",
                      "status": "True",
                      "message": "Resource is available",
                      "lastTransitionTime": "2024-03-18T12:57:58Z"
                  },
                  {
                      "type": "StatusFeedbackSynced",
                      "reason": "StatusFeedbackSynced",
                      "status": "True",
                      "message": "",
                      "lastTransitionTime": "2024-03-18T12:57:58Z"
                  }
              ],
              "resourceMeta": {
                  "kind": "Deployment",
                  "name": "nginx1",
                  "group": "apps",
                  "ordinal": 0,
                  "version": "v1",
                  "resource": "deployments",
                  "namespace": "default"
              },
              "statusFeedback": {
                  "values": [
                      {
                          "name": "status",
                          "fieldValue": {
                              "type": "JsonRaw",
                              "jsonRaw": "{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2024-03-18T12:58:04Z\",\"lastUpdateTime\":\"2024-03-18T12:58:04Z\",\"m
  essage\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2024-03-18T12:57:58Z\",\"lastUpdateTime\":\"2024-03-18T12:58:04Z\",\"message\":\"ReplicaSet \\\"nginx1-5d6b548959\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\
  ":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}"
                          }
                      }
                  ]
              }
          },
          "conditions": [
              {
                  "type": "Applied",
                  "reason": "AppliedManifestWorkComplete",
                  "status": "True",
                  "message": "Apply manifest work complete",
                  "lastTransitionTime": "2024-03-18T12:57:58Z"
              },
              {
                  "type": "Available",
                  "reason": "ResourcesAvailable",
                  "status": "True",
                  "message": "All resources are available",
                  "lastTransitionTime": "2024-03-18T12:57:58Z"
              }
          ]
      },
      "time": "2024-03-18T12:58:11.020848168Z",
      "type": "io.open-cluster-management.works.v1alpha1.manifests.status.update_request",
      "source": "b288a9da-8bfe-4c82-94cc-2b48e773fc46-work-agent",
      "resourceid": "dc970bd3-da6d-4a63-992b-dc0ae0419a7c",
      "sequenceid": "1769709885668200448",
      "clustername": "b288a9da-8bfe-4c82-94cc-2b48e773fc46",
      "specversion": "1.0",
      "originalsource": "maestro",
      "datacontenttype": "application/json",
      "resourceversion": "1"
  }
(1 row)

Signed-off-by: morvencao <lcao@redhat.com>

clyang82

LGTM

morvencao force-pushed the br_bundle_support branch from 746dfae to b2fbb39 Compare March 7, 2024 09:27

morvencao mentioned this pull request Mar 7, 2024

[WIP]support manifest bundle. #57

Closed

morvencao force-pushed the br_bundle_support branch 2 times, most recently from d43380b to 5331ebc Compare March 7, 2024 10:02

skeeey reviewed Mar 8, 2024

View reviewed changes

morvencao force-pushed the br_bundle_support branch 2 times, most recently from 63012f9 to 949dbc6 Compare March 14, 2024 13:23

morvencao force-pushed the br_bundle_support branch from 949dbc6 to efb0ae6 Compare March 18, 2024 02:34

qiujian16 reviewed Mar 18, 2024

View reviewed changes

morvencao added 6 commits March 20, 2024 00:57

support manifest bundle.

c49360e

Signed-off-by: morvencao <lcao@redhat.com>

update manifest in api resource.

4fe2b94

Signed-off-by: morvencao <lcao@redhat.com>

address comments.

1459082

Signed-off-by: morvencao <lcao@redhat.com>

fix testing.

27f549a

Signed-off-by: morvencao <lcao@redhat.com>

filter resource type for grpc resync respond.

ff67a24

Signed-off-by: morvencao <lcao@redhat.com>

add comments for resource schema.

18226fe

Signed-off-by: morvencao <lcao@redhat.com>

morvencao force-pushed the br_bundle_support branch from dbe9316 to 18226fe Compare March 20, 2024 01:02

clyang82 approved these changes Mar 20, 2024

View reviewed changes

clyang82 merged commit 95c5468 into openshift-online:main Mar 20, 2024
1 check was pending

morvencao deleted the br_bundle_support branch March 20, 2024 02:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add manifest bundle support #59

Add manifest bundle support #59

morvencao commented Mar 7, 2024

morvencao commented Mar 7, 2024

skeeey Mar 8, 2024

morvencao Mar 8, 2024

skeeey Mar 8, 2024

morvencao Mar 11, 2024

skeeey Mar 8, 2024

morvencao Mar 8, 2024

skeeey commented Mar 8, 2024

morvencao commented Mar 8, 2024

morvencao commented Mar 15, 2024

qiujian16 Mar 18, 2024

morvencao Mar 18, 2024

qiujian16 Mar 18, 2024

morvencao Mar 18, 2024

morvencao commented Mar 18, 2024

clyang82 left a comment

Add manifest bundle support #59

Add manifest bundle support #59

Conversation

morvencao commented Mar 7, 2024

morvencao commented Mar 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skeeey commented Mar 8, 2024

morvencao commented Mar 8, 2024

morvencao commented Mar 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morvencao commented Mar 18, 2024

clyang82 left a comment

Choose a reason for hiding this comment