Skip to content

Commit

Permalink
Feat: bootstrap sharding
Browse files Browse the repository at this point in the history 
Signed-off-by: Somefive <yd219913@alibaba-inc.com>
  • Loading branch information
@Somefive
Somefive committed Jan 29, 2023
1 parent f733d74 commit b0243a9
Show file tree
Hide file tree
Showing 29 changed files with 1,055 additions and 74 deletions.
4 changes: 3 additions & 1 deletion .github/workflows/apiserver-test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,9 @@ jobs:
make e2e-apiserver-test
- name: Stop kubevela, get profile
run: make end-e2e-core
run: |
make end-e2e-core
CORE_NAME=kubevela-shard sh ./hack/e2e/end_e2e_core.sh
- name: Upload coverage report
uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/e2e-multicluster-test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,9 @@ jobs:
make e2e-multicluster-test
- name: Stop kubevela, get profile
run: make end-e2e-core
run: |
make end-e2e-core
CORE_NAME=kubevela-shard sh ./hack/e2e/end_e2e_core.sh
- name: Upload coverage report
uses: codecov/codecov-action@d9f34f8cd5cb3b3eb79b3e4b5dae3a16df499a70
Expand Down
45 changes: 24 additions & 21 deletions charts/vela-core/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ helm install --create-namespace -n vela-system kubevela kubevela/vela-core --wai
| `featureGates.gzipApplicationRevision` | compress apprev using gzip (good) before being stored. This is reduces network throughput when dealing with huge apprevs. | `false` |
| `featureGates.zstdApplicationRevision` | compress apprev using zstd (fast and good) before being stored. This is reduces network throughput when dealing with huge apprevs. Note that zstd will be prioritized if you enable other compression options. | `true` |
| `featureGates.preDispatchDryRun` | enable dryrun before dispatching resources. Enable this flag can help prevent unsuccessful dispatch resources entering resourcetracker and improve the user experiences of gc but at the cost of increasing network requests. | `true` |
| `featureGates.validateComponentWhenSharding` | enable component validation in webhook when sharding mode enabled | `false` |


### MultiCluster parameters
Expand Down Expand Up @@ -132,27 +133,29 @@ helm install --create-namespace -n vela-system kubevela kubevela/vela-core --wai

### Common parameters

| Name | Description | Value |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `imagePullSecrets` | Image pull secrets | `[]` |
| `nameOverride` | Override name | `""` |
| `fullnameOverride` | Fullname override | `""` |
| `serviceAccount.create` | Specifies whether a service account should be created | `true` |
| `serviceAccount.annotations` | Annotations to add to the service account | `{}` |
| `serviceAccount.name` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template | `nil` |
| `nodeSelector` | Node selector | `{}` |
| `tolerations` | Tolerations | `[]` |
| `affinity` | Affinity | `{}` |
| `rbac.create` | Specifies whether a RBAC role should be created | `true` |
| `logDebug` | Enable debug logs for development purpose | `false` |
| `logFilePath` | If non-empty, write log files in this path | `""` |
| `logFileMaxSize` | Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. | `1024` |
| `kubeClient.qps` | The qps for reconcile clients, default is 100 | `100` |
| `kubeClient.burst` | The burst for reconcile clients, default is 200 | `200` |
| `authentication.enabled` | Enable authentication for application | `false` |
| `authentication.withUser` | Application authentication will impersonate as the request User | `true` |
| `authentication.defaultUser` | Application authentication will impersonate as the User if no user provided in Application | `kubevela:vela-core` |
| `authentication.groupPattern` | Application authentication will impersonate as the request Group that matches the pattern | `kubevela:*` |
| Name | Description | Value |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- |
| `imagePullSecrets` | Image pull secrets | `[]` |
| `nameOverride` | Override name | `""` |
| `fullnameOverride` | Fullname override | `""` |
| `serviceAccount.create` | Specifies whether a service account should be created | `true` |
| `serviceAccount.annotations` | Annotations to add to the service account | `{}` |
| `serviceAccount.name` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template | `nil` |
| `nodeSelector` | Node selector | `{}` |
| `tolerations` | Tolerations | `[]` |
| `affinity` | Affinity | `{}` |
| `rbac.create` | Specifies whether a RBAC role should be created | `true` |
| `logDebug` | Enable debug logs for development purpose | `false` |
| `logFilePath` | If non-empty, write log files in this path | `""` |
| `logFileMaxSize` | Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. | `1024` |
| `kubeClient.qps` | The qps for reconcile clients, default is 100 | `100` |
| `kubeClient.burst` | The burst for reconcile clients, default is 200 | `200` |
| `authentication.enabled` | Enable authentication for application | `false` |
| `authentication.withUser` | Application authentication will impersonate as the request User | `true` |
| `authentication.defaultUser` | Application authentication will impersonate as the User if no user provided in Application | `kubevela:vela-core` |
| `authentication.groupPattern` | Application authentication will impersonate as the request Group that matches the pattern | `kubevela:*` |
| `sharding.enabled` | When sharding enabled, the controller will run as master mode. Refer to https://github.com/kubevela/kubevela/blob/master/design/vela-core/sharding.md for details. | `false` |
| `sharding.schedulableShards` | The shards available for scheduling. If empty, dynamic discovery will be used. | `""` |


## Uninstallation
Expand Down
37 changes: 36 additions & 1 deletion charts/vela-core/templates/kubevela-controller.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ rules:
- apiGroups: ["authorization.k8s.io"]
resources: ["subjectaccessreviews"]
verbs: ["*"]

---

apiVersion: rbac.authorization.k8s.io/v1
Expand Down Expand Up @@ -85,6 +84,34 @@ subjects:
namespace: {{ .Release.Namespace }}

{{ end }}


{{ if and .Values.sharding.enabled .Values.authentication.enabled }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "kubevela.fullname" . }}:shard-scheduler
namespace: {{ .Release.Namespace }}
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ include "kubevela.fullname" . }}:shard-scheduler
namespace: {{ .Release.Namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ include "kubevela.fullname" . }}:shard-scheduler
subjects:
- kind: ServiceAccount
name: {{ include "kubevela.serviceAccountName" . }}
{{ end }}

---
# permissions to do leader election.
apiVersion: rbac.authorization.k8s.io/v1
Expand Down Expand Up @@ -183,6 +210,9 @@ spec:
metadata:
labels:
{{- include "kubevela.selectorLabels" . | nindent 8 }}
{{ if .Values.sharding.enabled }}
controller.core.oam.dev/shard-id: master
{{ end }}
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
Expand Down Expand Up @@ -282,6 +312,11 @@ spec:
- "--authentication-default-user={{ .Values.authentication.defaultUser }}"
- "--authentication-group-pattern={{ .Values.authentication.groupPattern }}"
{{ end }}
{{ if .Values.sharding.enabled }}
- "--enable-sharding"
- "--schedulable-shards={{ .Values.sharding.schedulableShards }}"
- "--feature-gates=ValidateComponentWhenSharding={{- .Values.featureGates.validateComponentWhenSharding | toString -}}"
{{ end }}
image: {{ .Values.imageRegistry }}{{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ quote .Values.image.pullPolicy }}
resources:
Expand Down
8 changes: 8 additions & 0 deletions charts/vela-core/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ optimize:
##@param featureGates.gzipApplicationRevision compress apprev using gzip (good) before being stored. This is reduces network throughput when dealing with huge apprevs.
##@param featureGates.zstdApplicationRevision compress apprev using zstd (fast and good) before being stored. This is reduces network throughput when dealing with huge apprevs. Note that zstd will be prioritized if you enable other compression options.
##@param featureGates.preDispatchDryRun enable dryrun before dispatching resources. Enable this flag can help prevent unsuccessful dispatch resources entering resourcetracker and improve the user experiences of gc but at the cost of increasing network requests.
##@param featureGates.validateComponentWhenSharding enable component validation in webhook when sharding mode enabled
##@param
featureGates:
enableLegacyComponentRevision: false
Expand All @@ -124,6 +125,7 @@ featureGates:
gzipApplicationRevision: false
zstdApplicationRevision: true
preDispatchDryRun: true
validateComponentWhenSharding: false

## @section MultiCluster parameters

Expand Down Expand Up @@ -268,3 +270,9 @@ authentication:
withUser: true
defaultUser: kubevela:vela-core
groupPattern: kubevela:*

## @param sharding.enabled When sharding enabled, the controller will run as master mode. Refer to https://github.com/kubevela/kubevela/blob/master/design/vela-core/sharding.md for details.
## @param sharding.schedulableShards The shards available for scheduling. If empty, dynamic discovery will be used.
sharding:
enabled: false
schedulableShards: ""
10 changes: 4 additions & 6 deletions cmd/core/app/options/options.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,19 @@ limitations under the License.
package options

import (
"flag"
"strconv"
"time"

ctrlrec "github.com/kubevela/pkg/controller/reconciler"
pkgmulticluster "github.com/kubevela/pkg/multicluster"
utillog "github.com/kubevela/pkg/util/log"
wfTypes "github.com/kubevela/workflow/pkg/types"
utilfeature "k8s.io/apiserver/pkg/util/feature"
cliflag "k8s.io/component-base/cli/flag"
"k8s.io/klog/v2"

standardcontroller "github.com/oam-dev/kubevela/pkg/controller"
commonconfig "github.com/oam-dev/kubevela/pkg/controller/common"
"github.com/oam-dev/kubevela/pkg/controller/sharding"
"github.com/oam-dev/kubevela/pkg/oam"
"github.com/oam-dev/kubevela/pkg/resourcekeeper"

Expand Down Expand Up @@ -165,11 +165,9 @@ func (s *CoreOptions) Flags() cliflag.NamedFlagSets {
pkgmulticluster.AddFlags(fss.FlagSet("multicluster"))
ctrlrec.AddFlags(fss.FlagSet("controllerreconciles"))
utilfeature.DefaultMutableFeatureGate.AddFlag(fss.FlagSet("featuregate"))

sharding.AddFlags(fss.FlagSet("sharding"))
kfs := fss.FlagSet("klog")
local := flag.NewFlagSet("klog", flag.ExitOnError)
klog.InitFlags(local)
kfs.AddGoFlagSet(local)
utillog.AddFlags(kfs)

if s.LogDebug {
_ = kfs.Set("v", strconv.Itoa(int(commonconfig.LogDebug)))
Expand Down

0 comments on commit b0243a9

Please sign in to comment.