Feat: SDK generating framework

[chivalryq]

Signed-off-by: Qiaozp qiaozhongpei.qzp@alibaba-inc.com

TL;DR

This PR introduce a framework which can generate KubeVela SDK from scaffolds, templates and X-Definitions.

Background

As KubeVela designs for separation of concerns, there are two roles of user: platform team and end user. They are dealing with Application in different ways.

For platform team

When platform team users assembling Application, filling in components, traits and workflow steps, they are actually assemble Application YAML. Unfortunately, due to the extensibility of KubeVela X-Definition system, the property structure of different component (also for traits, workflow steps, policies) varies. In the source code, we use a map[string]interface{} to accept all kinds of properties. The unstructured code make it hard examine, modify, insert values of nested properties.

Before KubeVela provide the SDK, the problem is mainly solved by providing OpenAPI schema to frontend and generate the form dynamically. There is also an enhancement scheme: ui-schema. After filling the schema, the JSON returned by frontend is already satisfied the specification of X-Definition properties. This is how VelaUX works when assembling Application.

For end user

Basically there are two ways for end user to use Application:

1. Through dashboard provided by platform team like VelaUX. The data is kept by the internal platform. In this pattern user's behavior can be better guided with typed form and UI indications.
2. Storing Application as part of the codebase. Then through some way (e.g. GitOps) the application is applied to cluster.

Benefits

In the aforementioned ways of using Application, there are some issues for both roles.

For platform team

There are some scenarios which use JOSN data passed from frontend is not enough:

1. Platform is responsible for some examination and modification on user's application:
   1. Adding some traits and policies which is taken care of by platform team. (e.g. observability)
   2. Execute some check which can't be done with OpenAPI schema and ui-schema. (e.g. call other services to access needed data)
2. Platform want to execute some offline batch jobs. For example, add observability trait to existing Applications

In these scenarios, it's painful to develop with map[string]interface{}. With SDK, platform team can benefits from the typed X-Definition properties and builds, examines, modifies application more easily.

For end user

If user is writing Application YAML, they have to refer to the doc. Although we have done some toolchain on this topic like vela show, a typed SDK also provide a possibility for end user to code Application in different languages.

Design and tools

Thanks to the OpenAPI schema and related tool openapi-generator in ecosystem, we can re-use the almost ready templates to generate code. The whole generation process is shown below.

Step0: The scaffold is written to destination if --init is specified. (omitted in pic)

Step1: Generate OpenAPI schema from CUE. The generated schema may not fully prepared as input of openapi-generator and it will be correct and adjust.

Step2: Call openapi-generator to generate SDK. After that some modification will be applied to make SDK API suited for KubeVela user.

Usage

This command requires java and mvn in PATH as we are leveraging openapi-generator which is a Java progarm.

# Initialize the SDK, generate API from all definitions, 
vela def gen-api -f /path/to/def/dir -o /path/to/sdk --init

# Incrementally generate API from definitions
vela def gen-api -f /path/to/def/dir -o /path/to/sdk

Future work

This PR only provide a framework on generating SDK and there are lots of jobs to do to provide
user-friendly SDKs in multiple languages.

1. Provide SDK in other languages. This is a LFX mentorship: [Feature] [LFX] Support auto generation of multiple languages SDK from CUE #5365
2. Allow user to customize the templates in Step2.

Preview

The generated SDK can work like below to build applications. This is a example written in

Build Application

Link

Modify Application

Link

Known Issues

There are some problems when generating:

1. apply-terraform-provider workflow step. Parameter are like below will cause the problem. Related issue: Error generating openapi: unsupported node string (*ast.Ident) cue-lang/cue#2259

basic: { info: string }
#A:{ 
  info
  a: string 
}
#B:{ 
  info
  b: string 
}
parameter: *#A|#B

4. label and annotation trait. openapi-generator doesn't support top-level schema to be a map. The two traits' properties is actually just map[string]string so it's easy to build it without using SDK.

I have:

• Read and followed KubeVela's contribution process.
• Related Docs updated properly. In a new feature or configuration option, an update to the documentation is necessary.
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Special notes for your reviewer

[codecov]

Codecov Report

Base: 61.34% // Head: 61.52% // Increases project coverage by +0.18% 🎉

Coverage data is based on head (2542692) compared to base (3a8746e).
Patch coverage: 75.43% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5431      +/-   ##
==========================================
+ Coverage   61.34%   61.52%   +0.18%     
==========================================
  Files         310      311       +1     
  Lines       47274    47919     +645     
==========================================
+ Hits        29001    29483     +482     
- Misses      15271    15383     +112     
- Partials     3002     3053      +51     

Flag	Coverage Δ
apiserver-e2etests	35.61% <ø> (+0.10%)	⬆️
apiserver-unittests	36.26% <ø> (ø)
core-unittests	55.62% <75.43%> (+0.47%)	⬆️
e2e-multicluster-test	18.83% <0.00%> (-0.24%)	⬇️
e2e-rollout-tests	20.97% <ø> (-0.05%)	⬇️
e2etests	26.51% <ø> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/definition/definition.go	73.07% <ø> (ø)
pkg/definition/gen_sdk/gen_sdk.go	62.05% <62.05%> (ø)
pkg/definition/gen_sdk/go.go	85.10% <85.10%> (ø)
pkg/apiserver/event/sync/worker.go	72.30% <0.00%> (-4.62%)	⬇️
pkg/apiserver/event/sync/cr2ux.go	47.19% <0.00%> (-4.50%)	⬇️
pkg/apiserver/event/sync/convert/convert.go	80.35% <0.00%> (-1.79%)	⬇️
pkg/addon/push.go	75.14% <0.00%> (-1.13%)	⬇️
pkg/multicluster/virtual_cluster.go	76.19% <0.00%> (-1.10%)	⬇️
...server/infrastructure/datastore/kubeapi/kubeapi.go	62.97% <0.00%> (-1.09%)	⬇️
pkg/controller/utils/capability.go	82.35% <0.00%> (-0.57%)	⬇️
... and 13 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[wonderflow]

End user assemble application, they can assemble through YAML or using VelaUX, when using YAML, the issue occurs.

Actually, the SDK is provided for end users.

[wonderflow]

It should be the sdk ways, the sdk can generate yamls out for GitOps or directly apply to cluster.

[wonderflow]

We need a section called "Language Caution" to describe the difference and notable points inside.

[chivalryq]

That would be great. And I think it should be part of kubevela.io

[wonderflow]

The zh doc is not necessary if time limits