Skip to content

Allows use of Istio and adds multi-cloud global strong scripts #133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Allows use of Istio and adds multi-cloud global strong scripts #133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b6923a5
Add istio replication to operator
alaye-ms Jul 11, 2025
c7c958a
move scripts to proper folder
alaye-ms Nov 4, 2025
1461441
update mutli region script
alaye-ms Nov 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions documentdb-kubectl-plugin/cmd/status.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,11 @@ func (o *statusOptions) run(ctx context.Context, cmd *cobra.Command) error {
if err != nil {
return fmt.Errorf("failed to read spec.clusterReplication.primary: %w", err)
}
clusterNames, found, err := unstructured.NestedStringSlice(document.Object, "spec", "clusterReplication", "clusterList")
clusterListRaw, found, err := unstructured.NestedSlice(document.Object, "spec", "clusterReplication", "clusterList")
if err != nil {
return fmt.Errorf("failed to read spec.clusterReplication.clusterList: %w", err)
}
if !found || len(clusterNames) == 0 {
if !found || len(clusterListRaw) == 0 {
return errors.New("DocumentDB spec.clusterReplication.clusterList is empty")
}

Expand All @@ -120,8 +120,9 @@ func (o *statusOptions) run(ctx context.Context, cmd *cobra.Command) error {
}
fmt.Fprintln(cmd.OutOrStdout())

statuses := make([]clusterStatus, 0, len(clusterNames))
for _, cluster := range clusterNames {
statuses := make([]clusterStatus, 0, len(clusterListRaw))
for _, clusterObj := range clusterListRaw {
cluster := clusterObj.(map[string]interface{})["name"].(string)
role := "Replica"
if cluster == primaryCluster {
role = "Primary"
Expand Down
6 changes: 5 additions & 1 deletion documentdb-kubectl-plugin/cmd/status_run_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,11 @@ func TestStatusRunRendersClusterTable(t *testing.T) {
docName := "documentdb-sample"

hubDoc := newDocument(docName, namespace, "cluster-a", "Ready")
if err := unstructured.SetNestedStringSlice(hubDoc.Object, []string{"cluster-a", "cluster-b"}, "spec", "clusterReplication", "clusterList"); err != nil {
clusterList := []interface{}{
map[string]interface{}{"name": "cluster-a"},
map[string]interface{}{"name": "cluster-b"},
}
if err := unstructured.SetNestedSlice(hubDoc.Object, clusterList, "spec", "clusterReplication", "clusterList"); err != nil {
t.Fatalf("failed to set clusterList: %v", err)
}
if err := unstructured.SetNestedField(hubDoc.Object, "PrimaryConn", "status", "connectionString"); err != nil {
Expand Down
12 changes: 12 additions & 0 deletions documentdb-playground/aws-setup/scripts/delete-cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ REGION="us-west-2"
DELETE_CLUSTER="${DELETE_CLUSTER:-true}"
DELETE_OPERATOR="${DELETE_OPERATOR:-true}"
DELETE_INSTANCE="${DELETE_INSTANCE:-true}"
SKIP_CONFIRMATION="false"

# Parse command line arguments
while [[ $# -gt 0 ]]; do
Expand All @@ -37,6 +38,10 @@ while [[ $# -gt 0 ]]; do
REGION="$2"
shift 2
;;
-y|--yes)
SKIP_CONFIRMATION="true"
shift
;;
-h|--help)
echo "Usage: $0 [OPTIONS]"
echo ""
Expand All @@ -45,12 +50,14 @@ while [[ $# -gt 0 ]]; do
echo " --instance-and-operator Delete instances and operator (keep cluster)"
echo " --cluster-name NAME EKS cluster name (default: documentdb-cluster)"
echo " --region REGION AWS region (default: us-west-2)"
echo " -y, --yes Skip confirmation prompt"
echo " -h, --help Show this help message"
echo ""
echo "Examples:"
echo " $0 # Delete everything (default)"
echo " $0 --instance-only # Delete only DocumentDB instances"
echo " $0 --instance-and-operator # Delete instances and operator, keep cluster"
echo " $0 --yes # Delete everything without confirmation"
exit 0
;;
*)
Expand Down Expand Up @@ -86,6 +93,11 @@ error() {

# Confirmation prompt
confirm_deletion() {
if [ "$SKIP_CONFIRMATION" == "true" ]; then
log "Skipping confirmation (--yes flag provided)"
return 0
fi

echo ""
echo "======================================="
echo " DELETION WARNING"
Expand Down
32 changes: 16 additions & 16 deletions documentdb-playground/multi-clould-setup/multi-cloud-deployment-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ possible and will be documented as they are tested.
- Two kubernetes clusters that are network connected to each other. For example using
- [Azure VPN Gatway](https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways)
- [Azure ExpressRoute](https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction)
- ENV variables `$AZURE_MEMBER` and `$ON_PREM_MEMBER` with the kubectl context names for your clusters
- ENV variables `$CLOUD_MEMBER` and `$ON_PREM_MEMBER` with the kubectl context names for your clusters
- (e.g. "azure-documentdb-cluster", "k3s-cluster-context")

## Architecture Overview
Expand Down Expand Up @@ -94,7 +94,7 @@ Run `kubectl get membercluster -A` again and see `True` under `JOINED` to confir

```bash
# Install on primary
kubectl config use-context $AZURE_MEMBER
kubectl config use-context $CLOUD_MEMBER
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true
Expand Down Expand Up @@ -251,7 +251,7 @@ kubectl apply -f ./documentdb-base.yaml
After a few seconds, ensure that the operator is running on both of the clusters

```sh
kubectl config use-context $AZURE_MEMBER
kubectl config use-context $CLOUD_MEMBER
kubectl get deployment -n documentdb-operator
kubectl config use-context $ON_PREM_MEMBER
kubectl get deployment -n documentdb-operator
Expand All @@ -274,21 +274,21 @@ Physical replication provides high availability and disaster recovery capabiliti
```bash
kubectl config use-context $ON_PREM_MEMBER
kubectl create configmap cluster-name -n kube-system --from-literal=name=on-prem-cluster-name
kubectl config use-context $AZURE_MEMBER
kubectl create configmap cluster-name -n kube-system --from-literal=name=azure-cluster-name
kubectl config use-context $CLOUD_MEMBER
kubectl create configmap cluster-name -n kube-system --from-literal=name=cloud-cluster-name
```

OR

```bash
cat <<EOF > azure-cluster-name.yaml
cat <<EOF > cloud-cluster-name.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-name
namespace: kube-system
data:
name: "azure-cluster-name"
name: "cloud-cluster-name"
EOF

cat <<EOF > on-prem-name.yaml
Expand All @@ -301,7 +301,7 @@ data:
name: "on-prem-cluster-name"
EOF

kubectl config use-context $AZURE_MEMBER
kubectl config use-context $CLOUD_MEMBER
kubectl apply -f ./primary-name.yaml
kubectl config use-context $ON_PREM_MEMBER
kubectl apply -f ./replica-name.yaml
Expand Down Expand Up @@ -332,10 +332,10 @@ spec:
storage:
pvcSize: 10Gi
clusterReplication:
primary: azure-cluster-name
primary: cloud-cluster-name
clusterList:
- azure-cluster-name
- on-prem-cluster-name
- name: cloud-cluster-name
- name: on-prem-cluster-name
exposeViaService:
serviceType: ClusterIP

Expand Down Expand Up @@ -364,7 +364,7 @@ kubectl apply -f ./documentdb-resource.yaml
After a few seconds, ensure that the operator is running on both of the clusters

```sh
kubectl config use-context $AZURE_MEMBER
kubectl config use-context $CLOUD_MEMBER
kubectl get pods -n documentdb-operator-ns
kubectl config use-context $ON_PREM_MEMBER
kubectl get pods -n documentdb-operator-ns
Expand All @@ -374,16 +374,16 @@ Output:

```text
NAME READY STATUS RESTARTS AGE
azure-cluster-name-1 2/2 Running 0 3m33s
cloud-cluster-name-1 2/2 Running 0 3m33s
```

## Testing and Verification

1. Test connection to DocumentDB:

```bash
# Get the service IP from primary (azure)
kubectl config use-context $AZURE_MEMBER
# Get the service IP from primary (cloud)
kubectl config use-context $CLOUD_MEMBER
service_ip=$(kubectl get service documentdb-service-documentdb-preview -n documentdb-preview-ns -o jsonpath="{.status.loadBalancer.ingress[0].ip}")

# Connect using mongosh
Expand All @@ -410,7 +410,7 @@ kubectl config use-context hub
kubectl patch documentdb documentdb-preview -n documentdb-preview-ns \
--type='json' -p='[
{"op": "replace", "path": "/spec/clusterReplication/primary", "value":"on-prem-cluster-name"},
{"op": "replace", "path": "/spec/clusterReplication/clusterList", "value":["on-prem-cluster-name"]}
{"op": "replace", "path": "/spec/clusterReplication/clusterList", "value":[{"name": "on-prem-cluster-name"}]}
]'
```

Expand Down
34 changes: 29 additions & 5 deletions operator/documentdb-helm-chart/crds/db.microsoft.com_documentdbs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,36 @@ spec:
description: ClusterList is the list of clusters participating
in replication.
items:
type: string
properties:
environment:
description: |-
EnvironmentOverride is the cloud environment of the member cluster.
Will default to the global setting
enum:
- eks
- aks
- gke
type: string
name:
description: Name is the name of the member cluster.
type: string
storageClass:
description: StorageClassOverride specifies the storage
class for DocumentDB persistent volumes in this member
cluster.
type: string
required:
- name
type: object
type: array
enableFleetForCrossCloud:
description: EnableFleetForCrossCloud determines whether to use
KubeFleet mechanics for the replication
type: boolean
crossCloudNetworkingStrategy:
description: CrossCloudNetworking determines which type of networking
mechanics for the replication
enum:
- AzureFleet
- Istio
- None
type: string
highAvailability:
description: Whether or not to have replicas on the primary cluster.
type: boolean
Expand Down
2 changes: 1 addition & 1 deletion operator/documentdb-helm-chart/templates/03_documentdb_wal_replica.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,4 +171,4 @@ metadata:
labels:
app.kubernetes.io/name: wal-replia-manager
app.kubernetes.io/managed-by: kustomize
{{- end }}
{{- end }}
18 changes: 15 additions & 3 deletions operator/src/api/preview/documentdb_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,16 +81,28 @@ type StorageConfiguration struct {
}

type ClusterReplication struct {
// EnableFleetForCrossCloud determines whether to use KubeFleet mechanics for the replication
EnableFleetForCrossCloud bool `json:"enableFleetForCrossCloud,omitempty"`
// CrossCloudNetworking determines which type of networking mechanics for the replication
// +kubebuilder:validation:Enum=AzureFleet;Istio;None
CrossCloudNetworkingStrategy string `json:"crossCloudNetworkingStrategy,omitempty"`
// Primary is the name of the primary cluster for replication.
Primary string `json:"primary"`
// ClusterList is the list of clusters participating in replication.
ClusterList []string `json:"clusterList"`
ClusterList []MemberCluster `json:"clusterList"`
// Whether or not to have replicas on the primary cluster.
HighAvailability bool `json:"highAvailability,omitempty"`
}

type MemberCluster struct {
// Name is the name of the member cluster.
Name string `json:"name"`
// EnvironmentOverride is the cloud environment of the member cluster.
// Will default to the global setting
// +kubebuilder:validation:Enum=eks;aks;gke
EnvironmentOverride string `json:"environment,omitempty"`
// StorageClassOverride specifies the storage class for DocumentDB persistent volumes in this member cluster.
StorageClassOverride string `json:"storageClass,omitempty"`
}

type ExposeViaService struct {
// ServiceType determines the type of service to expose for DocumentDB.
// +kubebuilder:validation:Enum=LoadBalancer;ClusterIP
Expand Down
17 changes: 16 additions & 1 deletion operator/src/api/preview/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

34 changes: 29 additions & 5 deletions operator/src/config/crd/bases/db.microsoft.com_documentdbs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,36 @@ spec:
description: ClusterList is the list of clusters participating
in replication.
items:
type: string
properties:
environment:
description: |-
EnvironmentOverride is the cloud environment of the member cluster.
Will default to the global setting
enum:
- eks
- aks
- gke
type: string
name:
description: Name is the name of the member cluster.
type: string
storageClass:
description: StorageClassOverride specifies the storage
class for DocumentDB persistent volumes in this member
cluster.
type: string
required:
- name
type: object
type: array
enableFleetForCrossCloud:
description: EnableFleetForCrossCloud determines whether to use
KubeFleet mechanics for the replication
type: boolean
crossCloudNetworkingStrategy:
description: CrossCloudNetworking determines which type of networking
mechanics for the replication
enum:
- AzureFleet
- Istio
- None
type: string
highAvailability:
description: Whether or not to have replicas on the primary cluster.
type: boolean
Expand Down
10 changes: 5 additions & 5 deletions operator/src/internal/cnpg/cnpg_cluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
)

func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, documentdb_image string, serviceAccountName string, log logr.Logger) *cnpgv1.Cluster {
func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, documentdb_image, serviceAccountName, storageClass string, log logr.Logger) *cnpgv1.Cluster {
sidecarPluginName := documentdb.Spec.SidecarInjectorPluginName
if sidecarPluginName == "" {
sidecarPluginName = util.DEFAULT_SIDECAR_INJECTOR_PLUGIN
Expand All @@ -31,9 +31,9 @@ func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, docu
}

// Configure storage class - use specified storage class or nil for default
var storageClass *string
if documentdb.Spec.Resource.Storage.StorageClass != "" {
Copy link
Collaborator

@hossain-rayhan hossain-rayhan Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are removing this line means for single cluster setup in AWS or GKE it will fail unless we are passing it right. From the controller I see we are only passing replicationContext.StorageClass but not from spec.

Copy link
Collaborator Author

@alaye-ms alaye-ms Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never tried AWS without a storage class, I just ran it that way because your sample scripts had a special storage class. GKE doesn't need a non-default storage class. This is so that each member cluster can have their own unique class in the membercluster list, single cluster operations shouldn't be changed by this

Copy link
Collaborator

@hossain-rayhan hossain-rayhan Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For customers who want to pass a specific storageClass for single cloud/cluster setup, they will use this documentdb.Spec.Resource.Storage.StorageClass spec config. Where are we evaluating that then?

Copy link
Collaborator Author

@alaye-ms alaye-ms Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In replication_context.go line 48

storageClass = &documentdb.Spec.Resource.Storage.StorageClass
var storageClassPointer *string
if storageClass != "" {
storageClassPointer = &storageClass
}

return &cnpgv1.Cluster{
Expand All @@ -56,7 +56,7 @@ func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, docu
Instances: documentdb.Spec.InstancesPerNode,
ImageName: documentdb_image,
StorageConfiguration: cnpgv1.StorageConfiguration{
StorageClass: storageClass, // Use configured storage class or default
StorageClass: storageClassPointer, // Use configured storage class or default
Size: documentdb.Spec.Resource.Storage.PvcSize,
},
InheritedMetadata: getInheritedMetadataLabels(documentdb.Name),
Expand Down
Loading
Loading