E2E deploy test: rewrite in go tests (#814)

* ✨ add tests/e2e package Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✨ use stream apis Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✨ add GetObject & SearchByID Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ♻️ improve validation Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 update e2e-deploy Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 add libhdf5 for e2e-deploy test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 fix vald client Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 fix NewPortforward interface Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔥 commented out logging search results Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ improve e2e test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🍱 update values-ci manifest Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 simplify Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 add config Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 inc replicas Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 fix loop Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔊 add log Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔍 improve logs Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 add keepalive Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 add Cassandra E2E test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 revise config Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 revise config Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 cassandra consistency = one Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 trigger e2e deploy test using chatops Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ use correct version of containers Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔊 print HELM_EXTRA_OPTIONS Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 remove redundant condition checks Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ⬆️ upgrade setup-k3d Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ just a workaround for strange mysql table definition Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🚚 move values into .github/helm/values Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ add sidecar test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 fix Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 fix config for agent-sidecar Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 reduce number of data Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 add createindex phase Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 remove useless config Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 remove useless lines Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 fix sidecar config Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ add tests for mysql deleteMeta Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Add options for which data is used for test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🎨 format yamls Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 📄 Update license header Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ⬆️ Upgrade to v1 APIs Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Fix Helm values for E2E test Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 Fix method call Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Reduce number of data Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 Fix to use v1 vald API Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Reduce number of Scylla cluster member Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🚸 Add a variable for scylla deploy task Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Add minio deploy task Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ♻️ Use kubectl wait instead for sh loop Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> ✏️ Fix typo Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ➕ Add cli-runtime to go.mod.default / Remove duplicated client-go entry Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ♻️ use assoc list for checking pr-xxx image existences Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ⬆️ Update go.sum Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 Fix build Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 tuning: add sleep Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 🔧 Tuning number of requests Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * 💚 Fix e2e tests code Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com> * ✅ Update e2e tests Signed-off-by: Rintaro Okamura <rintaro.okamura@gmail.com>
vdaas · Jan 20, 2021 · 7a78959 · 7a78959
1 parent 6adeefb
commit 7a78959
Show file tree

Hide file tree

Showing 24 changed files with 2,407 additions and 120 deletions.
diff --git a/.github/chatops_commands.md b/.github/chatops_commands.md
@@ -4,3 +4,4 @@
 - :white_check_mark: `/gen-test` - generate test codes
 - :label: `/label` - add labels
 - :rewind: `/rebase` - rebase master
+- :end: :two: :end: `/label actions/e2e-deploy` - run E2E deploy & integration test
diff --git a/.github/helm/values/values-agent-sidecar.yaml b/.github/helm/values/values-agent-sidecar.yaml
@@ -0,0 +1,89 @@
+#
+# Copyright (C) 2019-2021 vdaas.org vald team <vald@vdaas.org>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+defaults:
+  logging:
+    level: info
+
+agent:
+  minReplicas: 1
+  podManagementPolicy: Parallel
+  hpa:
+    enabled: false
+  resources:
+    requests:
+      cpu: 100m
+      memory: 50Mi
+  volumes:
+    - name: ngt-index
+      emptyDir: {}
+  volumeMounts:
+    - name: ngt-index
+      mountPath: /var/ngt
+  ngt:
+    auto_index_duration_limit: 3m
+    auto_index_check_duration: 1m
+    auto_index_length: 1000
+    dimension: 784
+    index_path: /var/ngt/index
+    enable_in_memory_mode: false
+  sidecar:
+    enabled: true
+    initContainerEnabled: true
+    env:
+      - name: AWS_ACCESS_KEY
+        value: ACCESSKEY
+      - name: AWS_SECRET_ACCESS_KEY
+        value: SECRETKEY
+    resources:
+      requests:
+        cpu: 100m
+        memory: 100Mi
+    config:
+      filename: vald-agent-ngt-index
+      blob_storage:
+        storage_type: "s3"
+        bucket: "vald-minio"
+        s3:
+          endpoint: "http://minio.default.svc.cluster.local:9000"
+          region: "us-east-1"
+          force_path_style: true
+
+gateway:
+  vald:
+    enabled: false
+  lb:
+    enabled: false
+  backup:
+    enabled: false
+  meta:
+    enabled: false
+
+discoverer:
+  enabled: false
+
+manager:
+  compressor:
+    enabled: false
+
+  backup:
+    enabled: false
+
+  index:
+    enabled: false
+
+meta:
+  enabled: false
diff --git a/charts/vald/values-ci.yaml → .github/helm/values/values-redis-mysql.yaml b/charts/vald/values-ci.yaml → .github/helm/values/values-redis-mysql.yaml
@@ -14,8 +14,15 @@
 # limitations under the License.
 #
 
+defaults:
+  logging:
+    level: info
+
 gateway:
   vald:
+    enabled: false
+  lb:
+    enabled: true
     minReplicas: 1
     hpa:
       enabled: false
@@ -25,6 +32,24 @@ gateway:
         memory: 50Mi
     gateway_config:
       index_replica: 3
+  backup:
+    enabled: true
+    minReplicas: 1
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
+  meta:
+    enabled: true
+    minReplicas: 1
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
 
 agent:
   minReplicas: 3
@@ -37,10 +62,10 @@ agent:
       cpu: 100m
       memory: 50Mi
   ngt:
-    auto_index_duration_limit: 60s
-    auto_index_check_duration: 5s
-    auto_index_length: 10
-    dimension: 6
+    auto_index_duration_limit: 3m
+    auto_index_check_duration: 1m
+    auto_index_length: 1000
+    dimension: 784
 
 discoverer:
   minReplicas: 1
@@ -53,7 +78,7 @@ discoverer:
 
 manager:
   compressor:
-    minReplicas: 1
+    minReplicas: 3
     hpa:
       enabled: false
     resources:
@@ -64,7 +89,7 @@ manager:
       compress_algorithm: gob
 
   backup:
-    minReplicas: 1
+    minReplicas: 3
     hpa:
       enabled: false
     resources:
@@ -80,7 +105,7 @@ manager:
         memory: 30Mi
 
 meta:
-  minReplicas: 1
+  minReplicas: 3
   hpa:
     enabled: false
   resources:

diff --git a/.github/helm/values/values-scylla.yaml b/.github/helm/values/values-scylla.yaml
@@ -0,0 +1,152 @@
+#
+# Copyright (C) 2019-2021 vdaas.org vald team <vald@vdaas.org>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+defaults:
+  logging:
+    level: info
+
+gateway:
+  vald:
+    enabled: false
+  lb:
+    enabled: true
+    minReplicas: 1
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
+    gateway_config:
+      index_replica: 3
+  backup:
+    enabled: true
+    minReplicas: 1
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
+  meta:
+    enabled: true
+    minReplicas: 1
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
+
+agent:
+  minReplicas: 3
+  maxReplicas: 10
+  podManagementPolicy: Parallel
+  hpa:
+    enabled: false
+  resources:
+    requests:
+      cpu: 100m
+      memory: 50Mi
+  ngt:
+    auto_index_duration_limit: 3m
+    auto_index_check_duration: 1m
+    auto_index_length: 1000
+    dimension: 784
+
+discoverer:
+  minReplicas: 1
+  hpa:
+    enabled: false
+  resources:
+    requests:
+      cpu: 100m
+      memory: 50Mi
+
+manager:
+  compressor:
+    minReplicas: 3
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 50Mi
+    compress:
+      compress_algorithm: gob
+
+  backup:
+    minReplicas: 3
+    hpa:
+      enabled: false
+    resources:
+      requests:
+        cpu: 100m
+        memory: 30Mi
+    image:
+      repository: vdaas/vald-manager-backup-cassandra
+    initContainers:
+      - type: wait-for-cassandra
+        name: wait-for-scylla
+        image: cassandra:latest
+        cassandra:
+          hosts:
+            - vald-scylla-cluster-dc0-rack0-0.scylla.svc.cluster.local
+        sleepDuration: 2
+    env: []
+    mysql:
+      enabled: false
+    cassandra:
+      enabled: true
+      config:
+        hosts:
+          - vald-scylla-cluster-dc0-rack0-0.scylla.svc.cluster.local
+        consistency: one
+
+  index:
+    replicas: 1
+    resources:
+      requests:
+        cpu: 100m
+        memory: 30Mi
+
+meta:
+  minReplicas: 3
+  hpa:
+    enabled: false
+  resources:
+    requests:
+      cpu: 100m
+      memory: 30Mi
+  image:
+    repository: vdaas/vald-meta-cassandra
+  initContainers:
+    - type: wait-for-cassandra
+      name: wait-for-scylla
+      image: cassandra:latest
+      cassandra:
+        hosts:
+          - vald-scylla-cluster-dc0-rack0-0.scylla.svc.cluster.local
+      sleepDuration: 2
+  env: []
+  redis:
+    enabled: false
+  cassandra:
+    enabled: true
+    config:
+      hosts:
+        - vald-scylla-cluster-dc0-rack0-0.scylla.svc.cluster.local
+      consistency: one