Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to create new service backed by an operator #2739

Merged
merged 5 commits into from Apr 10, 2020
Merged

Conversation

dharmit
Copy link
Member

@dharmit dharmit commented Mar 20, 2020

What type of PR is this?
/kind feature

What does does this PR do / why we need it:
This PR creates operator backed services.

Which issue(s) this PR fixes:
Fixes a part of #2613

How to test changes / Special notes to the reviewer:

  • On a CRC or OCP 4.x cluster, create a project using developer user: odo project create myproject
  • Log into the OpenShift Web Console as kubeadmin user and install etcd operator in the myproject namespace.
  • Go to the cli and confirm that the operator is installed in myproject namespace
    $ export ODO_EXPERIMENTAL=true
    $ odo login https://api.crc.testing:6443 -u developer -p developer
    $ odo catalog list services
    Operators available through Operator Hub
    NAME                       CRDs
    etcdoperator.v0.9.4        EtcdCluster, EtcdBackup, EtcdRestore
  • Create the operator backed service:
    odo service create etcdoperator.v0.9.4 --crd EtcdCluster
    Deploying service etcdoperator.v0.9.4 of type: etcdoperator.v0.9.4
    example
     ✓  Deploying service [6ms]
     ✓  Service 'etcdoperator.v0.9.4' was created
    
    Progress of the provisioning will not be reported and might take a long time
    You can see the current status by executing 'odo service list'
    Optionally, link etcdoperator.v0.9.4 to your component by running: 'odo link <component-name>'
  • Verify that pods have come up (odo doesn't support listing operator backed services yet so use oc CLI for this):
    $ oc get pods
    NAME                               READY   STATUS    RESTARTS   AGE
    etcd-operator-59c7bf9dfd-sdtl2     3/3     Running   0          20m
    example-8jgjmhq4m2                 1/1     Running   0          31s
    example-d6p2l2s7f8                 1/1     Running   0          103s
    example-rrnm4sz7rs                 1/1     Running   0          55s

@openshift-ci-robot openshift-ci-robot added the kind/feature Categorizes issue as a feature request. For PRs, that means that the PR is the implementation label Mar 20, 2020
@dharmit
Copy link
Member Author

dharmit commented Mar 23, 2020

Not related to this PR. Maybe a flake? Running 4.3 tests once more.
/test v4.3-integration-e2e-benchmark

[Fail] odo preference and config command tests when using --now with config command [It] should successfully set and unset variables 
/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

Ran 164 of 164 Specs in 3868.372 seconds
FAIL! -- 163 Passed | 1 Failed | 0 Pending | 0 Skipped

Ginkgo ran 1 suite in 1h4m33.522299631s
Test Suite Failed
make: *** [test-integration] Error 1
[Fail] odo preference and config command tests when using --now with config command [It] should successfully set and unset variables 
/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34 

@dharmit
Copy link
Member Author

dharmit commented Mar 24, 2020

Failed with same trace again. Doesn't look like a flake:

odo preference and config command tests
/go/src/github.com/openshift/odo/tests/integration/cmd_pref_config_test.go:20
  when using --now with config command
  /go/src/github.com/openshift/odo/tests/integration/cmd_pref_config_test.go:325
    should successfully set and unset variables [It]

[Fail] odo preference and config command tests when using --now with config command [It] should successfully set and unset variables 
/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34

Ran 165 of 165 Specs in 3402.247 seconds
FAIL! -- 164 Passed | 1 Failed | 0 Pending | 0 Skipped

Ginkgo ran 1 suite in 56m54.092548505s
Test Suite Failed
make: *** [test-integration] Error 1

@dharmit
Copy link
Member Author

dharmit commented Mar 24, 2020

Since it's a flake, triggering again hoping it passes. 😉 🤞

/test v4.3-integration-e2e-benchmark

@codecov
Copy link

codecov bot commented Mar 24, 2020

Codecov Report

Merging #2739 into master will decrease coverage by 0.49%.
The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2739      +/-   ##
==========================================
- Coverage   43.67%   43.18%   -0.50%     
==========================================
  Files          95       95              
  Lines        8762     8839      +77     
==========================================
- Hits         3827     3817      -10     
- Misses       4571     4653      +82     
- Partials      364      369       +5     
Impacted Files Coverage Δ
pkg/kclient/deployments.go 36.66% <0.00%> (-3.58%) ⬇️
pkg/kclient/kclient.go 24.13% <0.00%> (-2.79%) ⬇️
pkg/kclient/operators.go 0.00% <0.00%> (ø)
pkg/odo/cli/service/create.go 14.20% <0.00%> (-5.80%) ⬇️
pkg/service/service.go 67.75% <0.00%> (-1.91%) ⬇️
pkg/watch/watch.go 63.52% <0.00%> (-5.89%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00262d8...6f909d3. Read the comment docs.

@dharmit
Copy link
Member Author

dharmit commented Mar 25, 2020

Created dir: /tmp/030013783
Creating a new project: kfwdgfteka
Running odo with args [odo project create kfwdgfteka -w -v4]
[odo]  •  Waiting for project to come up  ...
[odo] I0325 05:14:03.867137    2341 occlient.go:547] Status of creation of project kfwdgfteka is Active
[odo] I0325 05:14:03.867173    2341 occlient.go:552] Project kfwdgfteka now exists
[odo] I0325 05:14:03.871808    2341 occlient.go:587] Status of creation of service account &ServiceAccount{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:default,GenerateName:,Namespace:kfwdgfteka,SelfLink:/api/v1/namespaces/kfwdgfteka/serviceaccounts/default,UID:7172b53d-6e57-11ea-95a4-42010a1e0044,ResourceVersion:7399,Generation:0,CreationTimestamp:2020-03-25 05:14:03 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Secrets:[{  default-token-wf9df    } {  default-dockercfg-r9bbv    }],ImagePullSecrets:[{default-dockercfg-r9bbv}],AutomountServiceAccountToken:nil,} is ready
 ✓  Waiting for project to come up [811ms]
[odo]  ✓  Project 'kfwdgfteka' is ready for use
[odo]  ✓  New project created and now using project: kfwdgfteka
[odo] I0325 05:14:03.878314    2341 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)
Created dir: /tmp/480965834
Created dir: /tmp/433769889
Current working dir: /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog
Running odo with args [odo service create --app qphldq -w dh-postgresql-apb --project kfwdgfteka --plan dev -p postgresql_user=luke -p postgresql_password=secret -p postgresql_database=my_data -p postgresql_version=9.6]
[odo] Deploying service dh-postgresql-apb of type: dh-postgresql-apb
[odo]  •  Deploying service  ...
 ✓  Deploying service [28ms]
[odo]  •  Waiting for service to come up  ...
 ✓  Waiting for service to come up [40s]
[odo]  ✓  Service 'dh-postgresql-apb' is ready for use
[odo] Optionally, link dh-postgresql-apb to your component by running: 'odo link <component-name>'
Running oc with args [oc get serviceinstance -o name -n kfwdgfteka]
[oc] serviceinstance.servicecatalog.k8s.io/dh-postgresql-apb
Running odo with args [odo app delete qphldq -f]
[odo]  ✗  qphldq app does not exists
• Failure [42.140 seconds]
odo service command tests
/home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:15
  When the application is deleted
  /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:439
    should delete the service(s) in the application as well [It]
    /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:444
    No future change is possible.  Bailing out early after 0.117s.
    Running odo with args [odo app delete qphldq -f]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0
    /home/travis/gopath/src/github.com/openshift/odo/tests/helper/helper_run.go:34

I don't understand why it can't find the app qphldq at the time of deletion. @amitkrout is it possible that this is happening because of parallel tests and that test has switched to a different OpenShift namespace by the time we execute odo app delete?

@dharmit
Copy link
Member Author

dharmit commented Mar 25, 2020

/test v4.2-integration-e2e-benchmark related to flake with --now test

/test v4.3-integration-e2e-benchmark
Looks like infra issue

level=info msg="Cluster operator monitoring Progressing is True with RollOutInProgress: Rolling out the stack."
level=error msg="Cluster operator monitoring Degraded is True with UpdatingAlertmanagerFailed: Failed to rollout the stack. Error: running task Updating Alertmanager failed: waiting for Alertmanager Route to become ready failed: waiting for RouteReady of alertmanager-main: Get https://172.30.0.1:443/apis/route.openshift.io/v1/namespaces/openshift-monitoring/routes/alertmanager-main: http2: server sent GOAWAY and closed the connection; LastStreamID=1533, ErrCode=NO_ERROR, debug=\"\""
level=info msg="Cluster operator monitoring Available is False with : "
level=info msg="Cluster operator openshift-apiserver Available is False with _OpenShiftAPICheckFailed: Available: \"oauth.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAvailable: \"quota.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)\nAvailable: \"security.openshift.io.v1\" is not ready: 503 (the server is currently unable to handle the request)"
level=fatal msg="failed to initialize the cluster: Some cluster operators are still updating: authentication, monitoring, openshift-apiserver, openshift-samples"

@dharmit
Copy link
Member Author

dharmit commented Mar 25, 2020

Travis fails for the same reason as mentioned in previous comment:

Created dir: /tmp/790277868
Creating a new project: pcfqaetfch
Running odo with args [odo project create pcfqaetfch -w -v4]
[odo]  •  Waiting for project to come up  ...
[odo] I0325 09:24:38.460136   24252 occlient.go:547] Status of creation of project pcfqaetfch is Active
[odo] I0325 09:24:38.460169   24252 occlient.go:552] Project pcfqaetfch now exists
 ✓  Waiting for project to come up [1s]
[odo]  ✓  Project 'pcfqaetfch' is ready for use
[odo] I0325 09:24:38.462518   24252 occlient.go:587] Status of creation of service account &ServiceAccount{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:default,GenerateName:,Namespace:pcfqaetfch,SelfLink:/api/v1/namespaces/pcfqaetfch/serviceaccounts/default,UID:72925942-6e7a-11ea-b369-42010a1e0102,ResourceVersion:4596,Generation:0,CreationTimestamp:2020-03-25 09:24:37 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Secrets:[{  default-dockercfg-xqjqx    } {  default-token-xlv2c    }],ImagePullSecrets:[{default-dockercfg-xqjqx}],AutomountServiceAccountToken:nil,} is ready
[odo]  ✓  New project created and now using project: pcfqaetfch
[odo] I0325 09:24:38.472493   24252 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)
Created dir: /tmp/303022427
Created dir: /tmp/553400574
Current working dir: /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog
Running odo with args [odo service create --app wkaxxd -w dh-postgresql-apb --project pcfqaetfch --plan dev -p postgresql_user=luke -p postgresql_password=secret -p postgresql_database=my_data -p postgresql_version=9.6]
[odo] Deploying service dh-postgresql-apb of type: dh-postgresql-apb
[odo]  •  Deploying service  ...
 ✓  Deploying service [55ms]
[odo]  •  Waiting for service to come up  ...
 ✓  Waiting for service to come up [44s]
[odo]  ✓  Service 'dh-postgresql-apb' is ready for use
[odo] Optionally, link dh-postgresql-apb to your component by running: 'odo link <component-name>'
Running oc with args [oc get serviceinstance -o name -n pcfqaetfch]
[oc] serviceinstance.servicecatalog.k8s.io/dh-postgresql-apb
Running odo with args [odo app delete wkaxxd -f]
[odo]  ✗  wkaxxd app does not exists
• Failure [47.560 seconds]
odo service command tests
/home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:15
  When the application is deleted
  /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:439
    should delete the service(s) in the application as well [It]
    /home/travis/gopath/src/github.com/openshift/odo/tests/integration/servicecatalog/cmd_service_test.go:444
    No future change is possible.  Bailing out early after 0.548s.
    Running odo with args [odo app delete wkaxxd -f]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0
    /home/travis/gopath/src/github.com/openshift/odo/tests/helper/helper_run.go:34

Going to try and modify the particular test case to also have project info so that it does: odo app delete <app-name> -f --project <project-name> and hope that works. 🤞

@dharmit
Copy link
Member Author

dharmit commented Mar 25, 2020

Failing for same --now flake #2642

  • 4.2 link
    [Fail] odo preference and config command tests when using --now with config command [It] should successfully set and unset variables
    
  • 4.1 link
     [Fail] odo preference and config command tests when using --now with config command [It] should successfully set and unset variables 
    

/test v4.1-integration-e2e-benchmark
/test v4.2-integration-e2e-benchmark

@amitkrout
Copy link
Contributor

@dharmit I am not getting the expected result. I am using a 4.4 cluster hosted on AWS resources.

Let me know if i am missing anything

$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

$ git fetch upstream pull/2739/head:pull_2739
remote: Enumerating objects: 21, done.
[...]

$ git checkout pull_2739
Switched to branch 'pull_2739'

$ make
go build -ldflags="-w -X github.com/openshift/odo/pkg/version.GITCOMMIT=8d7f6b48a" cmd/odo/odo.go

$ sudo cp odo /usr/local/bin/
Password:

$ make configure-installer-tests-cluster. ./scripts/configure-installer-tests-cluster.sh++ HTPASSWD_FILE=./htpass
++ USERPASS=developer
++ HTPASSWD_SECRET=htpasswd-secret
++ SETUP_OPERATORS_41=./scripts/setup-operators-41.sh
++ SETUP_OPERATORS=./scripts/setup-operators.sh
+++ pwd
++ DEFAULT_INSTALLER_ASSETS_DIR=/Users/amit/go/src/github.com/openshift/odo
++ KUBEADMIN_USER=kubeadmin
++ KUBEADMIN_PASSWORD_FILE=/Users/amit/go/src/github.com/openshift/odo/auth/kubeadmin-password
++ OC_STABLE_LOGIN=false
++ CI_OPERATOR_HUB_PROJECT=ci-operator-hub-project
++ export KUBECONFIG=/Users/amit/go/src/github.com/openshift/odo/auth/kubeconfig
++ KUBECONFIG=/Users/amit/go/src/github.com/openshift/odo/auth/kubeconfig
++ USERS='developer odonoprojectattemptscreate odosingleprojectattemptscreate odologinnoproject odologinsingleproject1'
++ '[' -z ']'
++ '[' '!' -f /Users/amit/go/src/github.com/openshift/odo/auth/kubeadmin-password ']'
++ '[' '!' -f /Users/amit/go/src/github.com/openshift/odo/auth/kubeconfig ']'
+++ cat /Users/amit/go/src/github.com/openshift/odo/auth/kubeadmin-password
++ KUBEADMIN_PASSWORD=P6F6L-Guw7W-e5xm5-AEtaF
++ oc login -u kubeadmin -p P6F6L-Guw7W-e5xm5-AEtaF
Login successful.

You have access to 59 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
++ oc new-project ci-operator-hub-project
Now using project "ci-operator-hub-project" on server "https://api.moregaju344installer2603.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app ruby~https://github.com/sclorg/ruby-ex.git

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node

++ oc adm policy add-role-to-user edit developer
Unable to connect to the server: net/http: TLS handshake timeout
+++ oc version -o yaml
+++ tail
+++ awk /gitVersion/
++ CI_K8S_VERSION='  gitVersion: v1.17.1'
++ [[   gitVersion: v1.17.1 =~ 1.13 ]]
++ sh ./scripts/setup-operators.sh
+ CI_OPERATOR_HUB_PROJECT=ci-operator-hub-project
+ oc create -f -
subscription.operators.coreos.com/mongodb-enterprise created
+ oc create -f -
operatorgroup.operators.coreos.com/ci-operator-hub-project-qbfbw created
+ oc create -f -
subscription.operators.coreos.com/etcd created
++ '[' -f ./htpass ']'
++ rm -rf ./htpass
++ HTPASSWD_CREATED=' -c '
+++ echo developer odonoprojectattemptscreate odosingleprojectattemptscreate odologinnoproject odologinsingleproject1
++ for i in '`echo $USERS`'
++ htpasswd -b -c ./htpass developer developer
Adding password for user developer
++ HTPASSWD_CREATED=
++ for i in '`echo $USERS`'
++ htpasswd -b ./htpass odonoprojectattemptscreate developer
Adding password for user odonoprojectattemptscreate
++ HTPASSWD_CREATED=
++ for i in '`echo $USERS`'
++ htpasswd -b ./htpass odosingleprojectattemptscreate developer
Adding password for user odosingleprojectattemptscreate
++ HTPASSWD_CREATED=
++ for i in '`echo $USERS`'
++ htpasswd -b ./htpass odologinnoproject developer
Adding password for user odologinnoproject
++ HTPASSWD_CREATED=
++ for i in '`echo $USERS`'
++ htpasswd -b ./htpass odologinsingleproject1 developer
Adding password for user odologinsingleproject1
++ HTPASSWD_CREATED=
++ oc apply -n openshift -f https://raw.githubusercontent.com/openshift/library/master/arch/x86_64/community/wildfly/imagestreams/wildfly-centos7.json
E0326 14:03:44.339765   72801 request.go:924] Unexpected error when reading response body: context deadline exceeded (Client.Timeout exceeded while reading body)
imagestream.image.openshift.io/wildfly created
++ oc get secret htpasswd-secret -n openshift-config
++ '[' 1 -eq 0 ']'
++ oc create secret generic htpasswd-secret --from-file=htpasswd=./htpass -n openshift-config
secret/htpasswd-secret created
++ oc apply -f -
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
oauth.config.openshift.io/cluster configured
++ for i in '{1..40}'
++ oc login -u developer -p developer
++ '[' 1 -eq 0 ']'
++ sleep 3
++ for i in '{1..40}'
++ oc login -u developer -p developer
++ '[' 0 -eq 0 ']'
++ OC_STABLE_LOGIN=true
++ for j in '{1..25}'
++ oc whoami
[...]
++ '[' true == true ']'
++ break
++ '[' true == false ']'
++ oc new-project myproject
Now using project "myproject" on server "https://api.moregaju344installer2603.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app ruby~https://github.com/sclorg/ruby-ex.git

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node

++ sleep 4
++ oc version
Client Version: 4.4.0-rc.2
Kubernetes Version: v1.17.1

$ export ODO_EXPERIMENTAL=true

$ odo login -u developer -p developer https://api.moregaju344installer2603.devcluster.openshift.com:6443
Connecting to the OpenShift cluster

The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Login successful.

You have one project on this server: "myproject"

Using project "myproject".

$ odo catalog list services
Operators available through Operator Hub
NAME                          CRDs
mongodb-enterprise.v1.2.4     MongoDB, MongoDBUser, MongoDBOpsManager

$ odo service create mongodb-enterprise.v1.2.4 --crd MongoDBDeploying service mongodb-enterprise.v1.2.4 of type: mongodb-enterprise.v1.2.4
 ◒  Deploying servicesample-replica-set
 ✓  Deploying service [910ms]
 ✓  Service 'mongodb-enterprise.v1.2.4' was created

Progress of the provisioning will not be reported and might take a long time
You can see the current status by executing 'odo service list'
Optionally, link mongodb-enterprise.v1.2.4 to your component by running: 'odo link <component-name>'

$ oc get pods
No resources found in myproject namespace.

Copy link
Contributor

@amitkrout amitkrout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dharmit Integration tests are missing. You can just replicate ``How to test``` through integration test script

@dharmit
Copy link
Member Author

dharmit commented Mar 27, 2020

@amitkrout I'm guessing that you're expecting MongoDB pods to spin up. Correct me if I'm wrong but I'm not able to infer anything else from the code blocks you've mentioned. The thing is that you need to test an operator that's got a valid alm example in its yaml definition. And that's not the case with MongoDB. So, please try with etcd operator's EtcdCluster type. There's also an operator called Nexus which worked while I was testing this during development.

In upcoming PRs, we will address this issue to an extent. You can follow #2723 if you'd like to.

Integration tests are missing. You can just replicate ``How to test``` through integration test script

Sure. Thanks for the tip. I'll add it. 👍

@openshift-ci-robot openshift-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. Required by Prow. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. Required by Prow. labels Mar 27, 2020
@dharmit
Copy link
Member Author

dharmit commented Mar 30, 2020

Doesn't look related to the PR. Link to error log.
/test v4.1-integration-e2e-benchmark

error: could not run steps: step integration-e2e-benchmark failed: template pod "integration-e2e-benchmark" failed: the pod ci-op-tpxcn7yv/integration-e2e-benchmark failed after 2h7m11s (failed containers: test): ContainerFailed one or more containers exited

@dharmit
Copy link
Member Author

dharmit commented Mar 30, 2020

Both seems to be infra issues.

/test v4.1-integration-e2e-benchmark

 ---
Installing from release registry.svc.ci.openshift.org/ci-op-0b6c09hd/release@sha256:e1568f9540326dfb4d47e00e684b1fc73ae2e5b085b9b069bba4c5c566b32acc
AWS region: us-west-2 (zones: us-west-2a us-west-2b)
level=fatal msg="failed to fetch Terraform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Config\": failed to fetch dependency of \"Master Machines\": failed to generate asset \"Platform Credentials Check\": validate AWS credentials: checking install permissions: error simulating policy: Throttling: Rate exceeded\n\tstatus code: 400, request id: 44ba86a9-e261-4e61-90a2-6617c17c5e0a" 

/test v4.2-integration-e2e-benchmark

 2020/03/30 15:03:27 Ran for 4h0m8s
2020/03/30 15:03:27 Submitted failure event to sentry (id=ab86d8a9007043c991f2dbaa87342e4e)
error: could not run steps: some steps failed:
  * execution cancelled
  * step [release:latest] failed: release "release-latest" failed: pod was deleted while ci-operator step was waiting for it
{"component":"entrypoint","file":"prow/entrypoint/run.go:242","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process gracefully exited before 15m0s grace period","time":"2020-03-30T15:03:27Z"}

@dharmit
Copy link
Member Author

dharmit commented Mar 31, 2020

/test v4.1-integration-e2e-benchmark

 error: could not run steps: some steps failed:
  * execution cancelled
  * step integration-e2e-benchmark failed: template pod "v4.1-integration-e2e-benchmark" failed: pod was deleted while ci-operator step was waiting for it
{"component":"entrypoint","file":"prow/entrypoint/run.go:242","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process gracefully exited before 15m0s grace period","time":"2020-03-30T20:04:36Z"} 

/test v4.2-integration-e2e-benchmark

 error: could not run steps: some steps failed:
  * execution cancelled
  * step integration-e2e-benchmark failed: template pod "v4.2-integration-e2e-benchmark" failed: the pod ci-op-clhvj3j1/v4.2-integration-e2e-benchmark was deleted without completing after 3h53m17s (failed containers: )
{"component":"entrypoint","file":"prow/entrypoint/run.go:242","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process gracefully exited before 15m0s grace period","time":"2020-03-30T20:04:21Z"} 

@dharmit
Copy link
Member Author

dharmit commented Mar 31, 2020

I'm seeing same errors repeating in the tests.

  • 4.1 integration tests
     error: could not run steps: some steps failed:
      * execution cancelled
      * step integration-e2e-benchmark failed: template pod "v4.2-integration-e2e-benchmark" failed: the pod ci-op-qhsj4p66/v4.2-integration-e2e-benchmark was deleted without completing after 3h56m17s (failed containers: )
    {"component":"entrypoint","file":"prow/entrypoint/run.go:242","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process gracefully exited before 15m0s grace period","time":"2020-03-31T09:48:50Z"}
    
  • 4.2 integration tests
     error: could not run steps: some steps failed:
      * execution cancelled
      * step integration-e2e-benchmark failed: template pod "v4.2-integration-e2e-benchmark" failed: the pod ci-op-qhsj4p66/v4.2-integration-e2e-benchmark was deleted without completing after 3h56m17s (failed containers: )
    {"component":"entrypoint","file":"prow/entrypoint/run.go:242","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process gracefully exited before 15m0s grace period","time":"2020-03-31T09:48:50Z"} 
    

@droslean
Copy link

/retest

@droslean
Copy link

/test 4.1-integration-e2e-benchmark
/test v4.2-integration-e2e-benchmark

@droslean
Copy link

/test v4.1-integration-e2e-benchmark

@droslean
Copy link

/test v4.1-integration-e2e-benchmark
/test v4.2-integration-e2e-benchmark

@dharmit
Copy link
Member Author

dharmit commented Apr 1, 2020

/test v4.1-integration-e2e-benchmark
4.1 failure seems related to infra error

error: could not run steps: step integration-e2e-benchmark failed: failed to acquire lease: resources not found

/test v4.3-integration-e2e-benchmark
4.3 looks like a flake to me

 odo generic
/go/src/github.com/openshift/odo/tests/integration/generic_test.go:15
  creating component with an application and url
  /go/src/github.com/openshift/odo/tests/integration/generic_test.go:140
    should create the component in default application [It]
[Fail] odo generic creating component with an application and url [It] should create the component in default application 
/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34
Ran 172 of 172 Specs in 4111.608 seconds
FAIL! -- 171 Passed | 1 Failed | 0 Pending | 0 Skipped
Ginkgo ran 1 suite in 1h8m39.017141689s
Test Suite Failed
make: *** [test-integration] Error 1 


helper.CmdShouldPass("odo", "service", "create", etcdOperator, "--crd", "EtcdCluster")

pods := helper.CmdShouldPass("oc", "get", "pods")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass namespace along with rest argument, For example

helper.CmdShouldPass("oc", "get", "pods", "-n", project)

otherwise it may leads to flaky output in parallel run

// Look for pod with example name because that's the name etcd will give to the pods.
etcdPod := regexp.MustCompile(`example-.[a-z0-9]*`).FindString(pods)

ocArgs := []string{"get", "pods", etcdPod, "-o", "template=\"{{.status.phase}}\""}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass namespace as a argument to avoid active namespace collision in parallel run

operators := helper.CmdShouldPass("odo", "catalog", "list", "services")
etcdOperator := regexp.MustCompile(`etcdoperator\.*[a-z][0-9]\.[0-9]\.[0-9]`).FindString(operators)

helper.CmdShouldPass("odo", "service", "create", etcdOperator, "--crd", "EtcdCluster")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete the service at the end of the spec to make sure odo service delete works in this scenario too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we're doing this by helper.CmdShouldPass("oc", "delete", "EtcdCluster", "example") since there's no way to delete operator backed services right now. This reminds me that I need to create a user story for service deletion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #2792 to track this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!!!

@dharmit
Copy link
Member Author

dharmit commented Apr 1, 2020

/test v4.3-integration-e2e-benchmark

Copy link
Contributor

@amitkrout amitkrout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dharmit Integration test looks good to me, however UTs are missing for the changes you made. Can please add them or do you have any other plan ?

@dharmit
Copy link
Member Author

dharmit commented Apr 6, 2020

@dharmit , if the experimental mode is enabled and then I try to do odo service create etcdoperator.v0.9.4 then wouldn't it try to consider that as a request for service catalog service?

@girishramnani
Yes, it's doing that. And it's failing:

$ odo service create etcdoperator.v0.9.4
 ✗  unable to search classes by name (clusterserviceclasses.servicecatalog.k8s.io is forbidden: User "developer" cannot list resource "clusterserviceclasses" in API group "servicecatalog.k8s.io" at the cluster scope)

I'll have to add a check for this so that if user has only specified operator name and not a crd, it fails; otherwise it continues to look for the service in Service Catalog.

I have a tangential question related to the behaviour of errors.Wrap, which I hope you can help me answer. Above command is failing at 2e5f03c#diff-47ac2191bd75f4970f6c210349223189R237-R239.

But I don't see the message unable to create service because Service Catalog is not enabled in your cluster in that error printed above. Does errors.Wrap wrap err.Error() on top of this message? If err.Error() was empty string, it would have printed the Service Catalog related message instead, is it so?

@girishramnani
Copy link
Contributor

I found two things

  • if err i.e the first argument to Wrap is nil then the whole error becomes nil hence Wrap returns nil
  • and for err being "" (empty string), that would give you <msg> : as per this line https://github.com/pkg/errors/blob/master/errors.go#L244 , but I don’t think someone ever returns an empty error instead of a nil.

@girishramnani
Copy link
Contributor

@dharmit should we close this PR as it seems its been superseded by this #2810

@girishramnani
Copy link
Contributor

@dharmit can we close this PR?

@dharmit
Copy link
Member Author

dharmit commented Apr 9, 2020

/retest

@dharmit
Copy link
Member Author

dharmit commented Apr 9, 2020

/test v4.2-integration-e2e-benchmark
Maybe a flake?

 odo supported images e2e tests
/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:14
  odo supported images deployment
  /go/src/github.com/openshift/odo/tests/e2escenarios/e2e_images_test.go:137
    Should be able to verify the nodejs-8-rhel7 image [It]
[Fail] odo supported images e2e tests odo supported images deployment [It] Should be able to verify the nodejs-8-rhel7 image 
/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34
Ran 23 of 23 Specs in 934.581 seconds
FAIL! -- 22 Passed | 1 Failed | 0 Pending | 0 Skipped 

@girishramnani
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. Required by Prow. label Apr 9, 2020
@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

12 similar comments
@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 5f25c90 into redhat-developer:master Apr 10, 2020
@dharmit dharmit deleted the fix-2613 branch May 15, 2020 04:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. Required by Prow. kind/feature Categorizes issue as a feature request. For PRs, that means that the PR is the implementation lgtm Indicates that a PR is ready to be merged. Required by Prow.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants