Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

‘make test-e2e-beta‘ failed with permission error #3165

Closed
zhengxiaomei123 opened this issue May 13, 2020 · 19 comments
Closed

‘make test-e2e-beta‘ failed with permission error #3165

zhengxiaomei123 opened this issue May 13, 2020 · 19 comments
Projects
Sprint 184

Comments

@zhengxiaomei123
Copy link
Contributor

zhengxiaomei123 commented May 13, 2020 •
edited by amitkrout

Steps to reproduce: (Copy pasted from slack conversation)

The env I am using is OCP 4.2 created by IBM Z UPI .
I cloned the odo project into GORoot and build the binary by make cross .
I created some pvs with nfs.
I changed the directory to odo/ and run make test-e2e-betato run e2e core beta tests.

I ran the command make test-e2e-beta to run e2e tests on OCP 4.2 Z platform. I got the error below:

odo core beta flow
/root/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_beta_test.go:13
  when component is in the current directory
  /root/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_beta_test.go:95
    create local java component and push code [It]
    /root/go/src/github.com/openshift/odo/tests/e2escenarios/e2e_beta_test.go:116

    No future change is possible.  Bailing out early after 29.932s.
    Running odo with args [odo push]
    Expected
        <int>: 1
    to match exit code:
        <int>: 0

    /root/go/src/github.com/openshift/odo/tests/helper/helper_run.go:34
@zhengxiaomei123
Copy link
Contributor Author

The detail log is

Created dir: /tmp/279047315
Creating a new project: anjfgllnxu
Running odo with args [odo project create anjfgllnxu -w -v4]
[odo]  •  Waiting for project to come up  ...
[odo] I0512 05:28:34.979481   28613 occlient.go:531] Status of creation of project anjfgllnxu is Active
[odo] I0512 05:28:34.979579   28613 occlient.go:536] Project anjfgllnxu now exists
[odo] I0512 05:28:34.983657   28613 occlient.go:571] Status of creation of service account &ServiceAccount{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:default,GenerateName:,Namespace:anjfgllnxu,SelfLink:/api/v1/namespaces/anjfgllnxu/serviceaccounts/default,UID:f3a4f4cf-9432-11ea-9b6d-025539722618,ResourceVersion:11539496,Generation:0,CreationTimestamp:2020-05-12 05:28:34 -0400 EDT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Secrets:[{  default-token-b74qd    } {  default-dockercfg-zksfp    }],ImagePullSecrets:[{default-dockercfg-zksfp}],AutomountServiceAccountToken:nil,} is ready
 ✓  Waiting for project to come up [511ms]
[odo]  ✓  Project 'anjfgllnxu' is ready for use
[odo] I0512 05:28:34.993868   28613 odo.go:80] Could not get the latest release information in time. Never mind, exiting gracefully :)
[odo]  ✓  New project created and now using project: anjfgllnxu
Current working dir: /root/go/src/github.com/openshift/odo/tests/e2escenarios
Setting current dir to: /tmp/279047315
Running oc with args [oc get is -n openshift -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}']
[oc] 'cli
[oc] cli-artifacts
[oc] installer
[oc] installer-artifacts
[oc] java
[oc] must-gather
[oc] nodejs
[oc] php
[oc] redhat-openjdk18-openshift
[oc] tests
[oc] 'Running oc with args [oc get is java -n openshift -o jsonpath='{range .spec.tags[*]}{.name}{"\n"}{end}']
[oc] '11
[oc] 8
[oc] latest
[oc] 'Running odo with args [odo component create redhat-openjdk18-openshift:1.8 mycomponent --app myapp --project anjfgllnxu]
[odo] Validation
[odo] Warning: redhat-openjdk18-openshift:1.8 is not fully supported by odo, and it is not guaranteed to work
[odo]  •  Validating component  ...
 ✓  Validating component [8ms]
[odo]
[odo] Please use `odo push` command to create the component with source deployed
Running odo with args [odo config set --env FOO=bar]
[odo]  ✓  Environment variables were successfully updated
[odo]
[odo] Run `odo push --config` command to apply changes to the cluster
Running odo with args [odo url create --port 8080]
[odo]  ✓  URL mycomponent-8080 created for component: mycomponent
[odo]
[odo] To apply the URL configuration changes, please use `odo push`
Running odo with args [odo push]
[odo] Validation
[odo]  •  Checking component  ...
 ✓  Checking component [12ms]
[odo]
[odo] Configuration changes
[odo]  ✓  Initializing component
[odo]  •  Creating component  ...
 ✓  Creating component [106ms]
[odo]
[odo] Applying URL changes
[odo]  ✓  URL mycomponent-8080: http://mycomponent-8080-myapp-anjfgllnxu.apps.ocp-test.redhat.com created
[odo]
[odo] Pushing to component mycomponent of type local
[odo]  •  Checking files for pushing  ...
 ✓  Checking files for pushing [3ms]
[odo]  •  Waiting for component to start  ...
 ✓  Waiting for component to start [29s]
[odo]  •  Syncing files to the component  ...
 ✓  Syncing files to the component [296ms]
[odo]  •  Building component  ...
[odo]  ✗  Building component [306ms]
[odo]  ✗
[odo] Unable to exec command [/opt/odo/bin/assemble-and-restart]:
[odo] + set -eo pipefail
[odo] + export ODO_UTILS_DIR=/opt/odo
[odo] + ODO_UTILS_DIR=/opt/odo
[odo] + export ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] + ODO_IMAGE_MAPPINGS_FILE=/opt/odo/language-scripts/image-mappings.json
[odo] ++ /opt/odo/bin/getlanguage
[odo] + IMAGE_LANG=java
[odo] + '[' '!' -z /home/jboss ']'
[odo] + '[' /tmp '!=' /home/jboss ']'
[odo] + '[' -n /opt/app-root/src-backup ']'
[odo] + '[' '!' -d /opt/app-root/src-backup ']'
[odo] + rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/.dummy.UjB3Bc" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/.gitignore.uepFZe" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/.pom.xml.u835lh" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/.odo/.config.yaml.QJ32hk" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/.odo/.odo-file-index.json.ALg3Hq" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/src/main/java/.AnotherMessageProducer.java.gP788w" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/src/main/java/.Application.java.grzzwD" failed: Permission denied (13)
[odo] rsync: mkstemp "/opt/app-root/src-backup/src/src/main/java/.MessageProducer.java.oxieSJ" failed: Permission denied (13)
[odo] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]
[odo]
[odo]  ✗  command terminated with exit code 23
Setting current dir to: /root/go/src/github.com/openshift/odo/tests/e2escenarios
Deleting project: anjfgllnxu
Running odo with args [odo project delete anjfgllnxu -f]
[odo] This project contains the following applications, which will be deleted
[odo] Application myapp
[odo] This application has following components that will be deleted
[odo] component named mycomponent
[odo] This component has following urls that will be deleted with component
[odo] URL named mycomponent-8080 with host mycomponent-8080-myapp-anjfgllnxu.apps.ocp-test.redhat.com having protocol http at port 8080
[odo] No services / could not get services
[odo]  •  Deleting project anjfgllnxu  ...
 ✓  Deleting project anjfgllnxu [28s]
[odo]  ✓  Deleted project : anjfgllnxu
Deleting dir: /tmp/279047315

@zhengxiaomei123
Copy link
Contributor Author

zhengxiaomei123 commented May 13, 2020 •
edited

I created the PersistentVolume for tests with yaml file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvnfs001
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  nfs:
    path: /nfs01/
    server: 9.x.x.x
  persistentVolumeReclaimPolicy: Retain

@zhengxiaomei123
Copy link
Contributor Author

zhengxiaomei123 commented May 13, 2020 •
edited

Hi , I reproduced the permission error. Could you please help check this issue? I am not sure if it is the PersistentVolume definition error or not. For upstream CI, how to provide the PV for those tests? Thanks.

@zhengxiaomei123
Copy link
Contributor Author

The nfs exports is

/nfs06 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)
/nfs05 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)
/nfs04 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)
/nfs03 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)
/nfs02 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)
/nfs01 *(rw,sync,no_wdelay,root_squash,insecure,fsid=0)

@jichenjc
Copy link
Contributor

@zhengxiaomei123 this is the same issue to the PV permission that discussed before?
I think they requested default project for some reason ,we followed that?

@zhengxiaomei123
Copy link
Contributor Author

this is the same issue to the PV permission that discussed before?

The PV definition is same as the before one. Maybe the tests cases need the different PV definition.

@zhengxiaomei123
Copy link
Contributor Author

I think they requested default project for some reason ,we followed that?

Sorry , What's the meaning?

@amitkrout amitkrout added this to For consideration in Sprint 184 via automation May 13, 2020
@amitkrout
Copy link
Contributor

Slack conversation:

On AWS SC is configured by default. In our test environment we uses cluster hosted on AWS, that is why we did not hit the issue you are facing with OCP 4.2 created by IBM Z UPI .

To make your PVC binding to a PV,

  1. First you need a storage server somewhere which is accessible from/to your cluster.
  2. Then follow docs [1] to create PVs pointing to above storage servers.
  3. Create storage class to group above PVs, which is optional.
    The example links to an NFS one, you can choose NFS/iSCSI/local volume or any one appropriate.
    Also the simplest HostPath PV [2] can be used for test purpose.
    [1]
    https://docs.openshift.com/container-platform/4.2/storage/persistent-storage/persistent-storage-nfs.html
    [2]
    https://docs.openshift.com/container-platform/4.2/storage/persistent-storage/persistent-storage-hostpath.html

Even these above steps did not work for @zhengxiaomei123. Actually this looks more like devops kind stuff. @girishramnani can you take a look if it is a miss from odo side by any chance ?

@zhengxiaomei123
Copy link
Contributor Author

@amitkrout Hi, I found it is related to the s2i image for applications. The error occured while there is warning "Warning: redhat-openjdk18-openshift:1.8 is not fully supported by odo, and it is not guaranteed to work".

@zhengxiaomei123
Copy link
Contributor Author

The image used is "registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.8" . It can not worked for odo.

@zhengxiaomei123
Copy link
Contributor Author

The error occurred not only with the imagestream for java but also with the imagestream for NodeJS.
The same image stream worked well for make test-e2e-beta, but failed for make test-integration .

@zhengxiaomei123
Copy link
Contributor Author 

[jboss@mycomponent-myapp-1-7jpnr src]$ rsync -rlO /tmp/src/. /opt/app-root/src-backup/src/
rsync: recv_generator: mkdir "/opt/app-root/src-backup/src/src" failed: Permission denied (13)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/opt/app-root/src-backup/src/.dummy.xrXXmC" failed: Permission denied (13)
rsync: mkstemp "/opt/app-root/src-backup/src/.gitignore.y3wCy4" failed: Permission denied (13)
rsync: mkstemp "/opt/app-root/src-backup/src/.pom.xml.bFy6Qx" failed: Permission denied (13)
rsync: mkstemp "/opt/app-root/src-backup/src/.odo/.config.yaml.imJ930" failed: Permission denied (13)
rsync: mkstemp "/opt/app-root/src-backup/src/.odo/.odo-file-index.json.jMCPbu" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]

@zhengxiaomei123
Copy link
Contributor Author

zhengxiaomei123 commented May 25, 2020 •
edited 

[jboss@mycomponent-myapp-1-7jpnr src]$ ls /opt/app-root/src-backup/src/ -all
total 24
drwxr-xr-x. 3 1001940000 65534 4096 May 13 07:32 .
drwxr-xr-x. 3 1001940000 65534 4096 May 13 07:32 ..
-rw-r--r--. 1 1001940000 65534    0 May 13 07:32 .dummy
-rw-------. 1 1001940000 65534   25 May 13 07:32 .gitignore
drwxr-xr-x. 2 1001940000 65534 4096 May 13 07:32 .odo
-rw-r--r--. 1 1001940000 65534  329 May 13 07:32 package.json
-rw-r--r--. 1 1001940000 65534  543 May 13 07:32 server.js

$id
uid=1005250000 (jboss) gid=0(root) groups=0(root),1004830000

ls -lah /opt/
total 4.0K
drwxrwxr-x. 1        185 root         46 May 25 07:51 .
drwxr-xr-x. 1 root       root         39 May 25 07:51 ..
drwxr-xr-x. 6 1001940000      65534 4.0K May 13 07:32 app-root
drwxrwxr-x. 1        185 root         23 Apr 21 11:52 jboss
lrwxrwxrwx. 1        185 root         28 Apr 21 11:55 jolokia -> /opt/jboss/container/jolokia
drwxrwsrwx. 5 root       1005250000   53 May 25 07:51 odo
drwxr-xr-x. 3 root       root         24 Jun 18  2018 rh
drwxr-xr-x. 2        185 root        166 Apr 21 11:55 run-java

@jichenjc
Copy link
Contributor

@zhengxiaomei123
Are those on s390x or on x86 platform?

@zhengxiaomei123
Copy link
Contributor Author

@zhengxiaomei123
Are those on s390x or on x86 platform?

s390x, occurred sometimes.

@zhengxiaomei123
Copy link
Contributor Author

zhengxiaomei123 commented May 25, 2020 •
edited

The user group is 1005250000 but the app-root user permission is 65534 (nfsnobody).

@zhengxiaomei123
Copy link
Contributor Author 

ls -lah /nfs12
total 16K
drwxrwxrwx.  4 root       root      4.0K May 18 02:26 .
dr-xr-xr-x. 32 root       root      4.0K May 12 23:10 ..
drwxr-xr-x.  6 1001760000 nfsnobody 4.0K May 13 03:29 app-root
drwxrwxrwx.  2 nfsnobody  nfsnobody 4.0K May 18 02:27 deployment

@zhengxiaomei123
Copy link
Contributor Author

I am not sure if it is related to the PVs. I change the definition of PV and then haven't met this error again.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvnfs009
spec:
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  - ReadWriteMany
  - ReadOnlyMany
  nfs:
    path: /nfs/odotest/09
    server: 192.168.127.1
  persistentVolumeReclaimPolicy: Recycle

@zhengxiaomei123
Copy link
Contributor Author

I will close this issue and If I meet this error again I will reopen it.

Sprint 184 automation moved this from For consideration to Done Jun 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
No open projects
Sprint 184
  
Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dharmit @zhengxiaomei123 @jichenjc @amitkrout