New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot find supported RGW runtime. Disabling RGW fsal build #126

Closed
lnsyyj opened this Issue Nov 4, 2016 · 16 comments

Comments

Projects
None yet
4 participants
@lnsyyj

lnsyyj commented Nov 4, 2016

ceph version 10.2.3
ubuntu 14.04 server
nfs-ganesha 2.4.1
ntirpc 1.4.3

cmake -DUSE_FSAL_RGW=ON ../src/

-- Found rgw libraries: /usr/lib
-- Could NOT find RGW: Found unsuitable version ".", but required is at least "1.1" (found /usr)
CMake Warning at CMakeLists.txt:571 (message):
Cannot find supported RGW runtime. Disabling RGW fsal build

Hello, everyone, Will the nfs-ganesha in ceph 10.2.3 version available?

@dang

This comment has been minimized.

Show comment
Hide comment
@dang

dang Nov 4, 2016

Contributor

The 2.4 version of ganesha needs a newer Ceph than 10.2.3. I believe the relevant parts have been backported and will be in 10.2.4; otherwise, you'll need to stick with the 2.3.x versions of ganesha.

Contributor

dang commented Nov 4, 2016

The 2.4 version of ganesha needs a newer Ceph than 10.2.3. I believe the relevant parts have been backported and will be in 10.2.4; otherwise, you'll need to stick with the 2.3.x versions of ganesha.

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 5, 2016

@dang Thank you very much.

lnsyyj commented Nov 5, 2016

@dang Thank you very much.

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 7, 2016

Hi @dang
Will if I used nfs-ganesha 2.3.3 , ntirpc need to use what version?

lnsyyj commented Nov 7, 2016

Hi @dang
Will if I used nfs-ganesha 2.3.3 , ntirpc need to use what version?

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 7, 2016

Oh, I use ntirpc 1.3.1

lnsyyj commented Nov 7, 2016

Oh, I use ntirpc 1.3.1

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 7, 2016

I use nfs-ganesha 2.3.3, ntirpc 1.3.1
Compiler is no problem, but start using the following order
Ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/nfs-ganesha/nfs-ganesha.log -NIV_DEBUG

Failure log

07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :INFO :Cache Inode library successfully initialized
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :DEBUG :Now building IP/name cache
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] load_fsal :NFS STARTUP :DEBUG :Loading FSAL RGW with /usr/lib/ganesha/libfsalrgw.so
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init :FSAL :DEBUG :RGW module registering.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_config :FSAL :DEBUG :RGW module setup.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)

lnsyyj commented Nov 7, 2016

I use nfs-ganesha 2.3.3, ntirpc 1.3.1
Compiler is no problem, but start using the following order
Ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/nfs-ganesha/nfs-ganesha.log -NIV_DEBUG

Failure log

07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :INFO :Cache Inode library successfully initialized
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :DEBUG :Now building IP/name cache
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :INFO :IP/name cache successfully initialized
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] load_fsal :NFS STARTUP :DEBUG :Loading FSAL RGW with /usr/lib/ganesha/libfsalrgw.so
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init :FSAL :DEBUG :RGW module registering.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_config :FSAL :DEBUG :RGW module setup.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)

@dang

This comment has been minimized.

Show comment
Hide comment
@dang

dang Nov 7, 2016

Contributor

EIO means that the RGW client couldn't communicate properly with the cluster. Can you do stand-alone RGW operations from that host with the same credentials?

Contributor

dang commented Nov 7, 2016

EIO means that the RGW client couldn't communicate properly with the cluster. Can you do stand-alone RGW operations from that host with the same credentials?

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 9, 2016

root@B-80-7:/etc/ganesha# ps -ef | grep rados
root 32688 32604 0 11:25 pts/11 00:00:00 tailf radosgw.log
root 32724 1 0 11:25 ? 00:00:48 /usr/bin/radosgw -n client.radosgw.gateway
root 33071 32665 0 14:12 pts/13 00:00:00 grep --color=auto rados

Nfs-ganesha and rados gateway are on the same machine

lnsyyj commented Nov 9, 2016

root@B-80-7:/etc/ganesha# ps -ef | grep rados
root 32688 32604 0 11:25 pts/11 00:00:00 tailf radosgw.log
root 32724 1 0 11:25 ? 00:00:48 /usr/bin/radosgw -n client.radosgw.gateway
root 33071 32665 0 14:12 pts/13 00:00:00 grep --color=auto rados

Nfs-ganesha and rados gateway are on the same machine

@dang

This comment has been minimized.

Show comment
Hide comment
@dang

dang Nov 10, 2016

Contributor

@mattbenjamin Any ideas?

Contributor

dang commented Nov 10, 2016

@mattbenjamin Any ideas?

@mattbenjamin

This comment has been minimized.

Show comment
Hide comment
@mattbenjamin

mattbenjamin Nov 10, 2016

Contributor

@insyyj, @dang The most common root cause would be a failure to authenticate with the supplied AWS credentials.

  1. first check the values of User_ID, Access_Key_Id, and Secret_Access_Key in the EXPORT, FSAL block
  2. then, consider adding
    init_args = "-d --debug-rgw=16" to the RGW block, for additional debugging output (lots)
Contributor

mattbenjamin commented Nov 10, 2016

@insyyj, @dang The most common root cause would be a failure to authenticate with the supplied AWS credentials.

  1. first check the values of User_ID, Access_Key_Id, and Secret_Access_Key in the EXPORT, FSAL block
  2. then, consider adding
    init_args = "-d --debug-rgw=16" to the RGW block, for additional debugging output (lots)
@mattbenjamin

This comment has been minimized.

Show comment
Hide comment
@mattbenjamin

mattbenjamin Nov 10, 2016

Contributor

@lnsyyj above

Contributor

mattbenjamin commented Nov 10, 2016

@lnsyyj above

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 11, 2016

@mattbenjamin @dang
Thank you, I checked the accesskey and secretkey, and aws the s3 browser client to connect my rgw, everything is normal.
My rgw debug level is 20.
Start nfs-ganesha, it seems nfs-ganesha not connected to rgw, rgw no output log.
In nfs-ganesha on the side of this report on the following log:

07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_config :FSAL :DEBUG :RGW module setup.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)

lnsyyj commented Nov 11, 2016

@mattbenjamin @dang
Thank you, I checked the accesskey and secretkey, and aws the s3 browser client to connect my rgw, everything is normal.
My rgw debug level is 20.
Start nfs-ganesha, it seems nfs-ganesha not connected to rgw, rgw no output log.
In nfs-ganesha on the side of this report on the following log:

07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] init_config :FSAL :DEBUG :RGW module setup.
07/11/2016 16:21:41 : epoch 58203995 : B-80-7 : nfs-ganesha-176649[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)

@mattbenjamin

This comment has been minimized.

Show comment
Hide comment
@mattbenjamin

mattbenjamin Nov 11, 2016

Contributor

@lnsyyj with the elevated level of rgw debugging, you may have an indication of the cause of failure;
and in fact, I should restate, we haven't attempted to use the AWS credentials yet, it's just trying to start an internal radosgw instance; does ganesha need a non-default path to a correct (rgw-aware) ceph.conf file? as @dang says, it's trying to start up with the cluster

Contributor

mattbenjamin commented Nov 11, 2016

@lnsyyj with the elevated level of rgw debugging, you may have an indication of the cause of failure;
and in fact, I should restate, we haven't attempted to use the AWS credentials yet, it's just trying to start an internal radosgw instance; does ganesha need a non-default path to a correct (rgw-aware) ceph.conf file? as @dang says, it's trying to start up with the cluster

@mattbenjamin

This comment has been minimized.

Show comment
Hide comment
@mattbenjamin

mattbenjamin Nov 11, 2016

Contributor

@lnsyyj am trying to imply here, will look at logs if you can provide them; It is the case that in the other cases I've seen where librgw_create() failed, nfs-ganesha was running on a machine w/invalid ceph.conf and keyring. As that's not the case here, trying to look for something like wrong ceph.conf being used.

Contributor

mattbenjamin commented Nov 11, 2016

@lnsyyj am trying to imply here, will look at logs if you can provide them; It is the case that in the other cases I've seen where librgw_create() failed, nfs-ganesha was running on a machine w/invalid ceph.conf and keyring. As that's not the case here, trying to look for something like wrong ceph.conf being used.

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 11, 2016

@dang @mattbenjamin Thank you. I look at the configuration file.

lnsyyj commented Nov 11, 2016

@dang @mattbenjamin Thank you. I look at the configuration file.

@lnsyyj

This comment has been minimized.

Show comment
Hide comment
@lnsyyj

lnsyyj Nov 11, 2016

This is my ceph.conf
root@B-80-7:~# cat /etc/ceph/ceph.conf
[global]
fsid = d503e95e-0c64-45b0-9e1a-29ed83df9ef8
public_network = 192.168.250.0/24
cluster_network = 192.168.122.0/24
mon_initial_members = B-80-7
mon_host = 192.168.250.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 1
[client.radosgw.gateway]
host = B-80-7
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /tmp/radosgw.sock
log file = /var/log/ceph/radosgw.log
rgw_admin_entry = admin
debug rgw = 20
#debug rgw = 16

This is my ganesha.conf
root@B-80-7:~# cat /etc/ganesha/ganesha.conf
EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
NFS_Protocols = 4;
Transport_Protocols = TCP;
FSAL {
Name = RGW;
User_Id = "yujiang";
Access_Key_Id ="5DV34OCY7850RPX5ISOB";
Secret_Access_Key = "0eKzFuCwffo1cTSLU75Udq4PCXhLIY3hJoJPG6m7";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
}

This is the keyring file permissions
root@B-80-7:~# ll /etc/ceph/
total 24
drwxr-xr-x 2 root root 4096 Nov 11 16:03 ./
drwxr-xr-x 100 root root 4096 Nov 3 17:48 ../
-rw-r--r-- 1 ceph ceph 63 Oct 20 16:38 ceph.client.admin.keyring
-rw-r--r-- 1 root root 1534 Nov 11 16:03 ceph.conf
-rw-r--r-- 1 root root 121 Oct 20 16:58 keyring.radosgw.gateway
-rw-r--r-- 1 root root 92 Sep 21 06:43 rbdmap
-rw------- 1 root root 0 Oct 20 16:38 tmpUpY4HP

This is ceph auth list
root@B-80-7:~/BBQ0.9-release# ceph auth list
installed auth entries:
osd.0
key: AQCygghYcK6zDBAAm6L7gHx/5et6YZKuyjmoTQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQDAgghYDsLbAhAAK2CIxMuuDwNGe+hSWXOXYw==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQB0gghYGUagHxAAjw6SxA/9MWUP8rHhVpo3pg==
caps: [mds] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQB1gghYSlT1AxAAKwh+AX7xbxCXzutBGXpcyg==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
key: AQB0gghYcN4RKhAAAGf93siYbXIN9K0efScmAQ==
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rgw
key: AQB0gghYqe3+NBAAE1Jv92fVnVQW8jOo29jqHw==
caps: [mon] allow profile bootstrap-rgw
client.radosgw.gateway
key: AQAthwhYjFFBERAA2XPTJuOvPZSaUvJrNTXx0g==
caps: [mon] allow rwx
caps: [osd] allow rwx

This is the contents of the ceph.client.admin.keyring file
root@B-80-7:~# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQB0gghYGUagHxAAjw6SxA/9MWUP8rHhVpo3pg==

This is the contents of the keyring.radosgw.gateway file
[client.radosgw.gateway]
key = AQAthwhYjFFBERAA2XPTJuOvPZSaUvJrNTXx0g==
caps mon = "allow rwx"
caps osd = "allow rwx"

lnsyyj commented Nov 11, 2016

This is my ceph.conf
root@B-80-7:~# cat /etc/ceph/ceph.conf
[global]
fsid = d503e95e-0c64-45b0-9e1a-29ed83df9ef8
public_network = 192.168.250.0/24
cluster_network = 192.168.122.0/24
mon_initial_members = B-80-7
mon_host = 192.168.250.79
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 1
[client.radosgw.gateway]
host = B-80-7
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /tmp/radosgw.sock
log file = /var/log/ceph/radosgw.log
rgw_admin_entry = admin
debug rgw = 20
#debug rgw = 16

This is my ganesha.conf
root@B-80-7:~# cat /etc/ganesha/ganesha.conf
EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
NFS_Protocols = 4;
Transport_Protocols = TCP;
FSAL {
Name = RGW;
User_Id = "yujiang";
Access_Key_Id ="5DV34OCY7850RPX5ISOB";
Secret_Access_Key = "0eKzFuCwffo1cTSLU75Udq4PCXhLIY3hJoJPG6m7";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
}

This is the keyring file permissions
root@B-80-7:~# ll /etc/ceph/
total 24
drwxr-xr-x 2 root root 4096 Nov 11 16:03 ./
drwxr-xr-x 100 root root 4096 Nov 3 17:48 ../
-rw-r--r-- 1 ceph ceph 63 Oct 20 16:38 ceph.client.admin.keyring
-rw-r--r-- 1 root root 1534 Nov 11 16:03 ceph.conf
-rw-r--r-- 1 root root 121 Oct 20 16:58 keyring.radosgw.gateway
-rw-r--r-- 1 root root 92 Sep 21 06:43 rbdmap
-rw------- 1 root root 0 Oct 20 16:38 tmpUpY4HP

This is ceph auth list
root@B-80-7:~/BBQ0.9-release# ceph auth list
installed auth entries:
osd.0
key: AQCygghYcK6zDBAAm6L7gHx/5et6YZKuyjmoTQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQDAgghYDsLbAhAAK2CIxMuuDwNGe+hSWXOXYw==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQB0gghYGUagHxAAjw6SxA/9MWUP8rHhVpo3pg==
caps: [mds] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQB1gghYSlT1AxAAKwh+AX7xbxCXzutBGXpcyg==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
key: AQB0gghYcN4RKhAAAGf93siYbXIN9K0efScmAQ==
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rgw
key: AQB0gghYqe3+NBAAE1Jv92fVnVQW8jOo29jqHw==
caps: [mon] allow profile bootstrap-rgw
client.radosgw.gateway
key: AQAthwhYjFFBERAA2XPTJuOvPZSaUvJrNTXx0g==
caps: [mon] allow rwx
caps: [osd] allow rwx

This is the contents of the ceph.client.admin.keyring file
root@B-80-7:~# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQB0gghYGUagHxAAjw6SxA/9MWUP8rHhVpo3pg==

This is the contents of the keyring.radosgw.gateway file
[client.radosgw.gateway]
key = AQAthwhYjFFBERAA2XPTJuOvPZSaUvJrNTXx0g==
caps mon = "allow rwx"
caps osd = "allow rwx"

@ffilz

This comment has been minimized.

Show comment
Hide comment
@ffilz

ffilz Jan 17, 2017

Member

Have we resolved this one?

Member

ffilz commented Jan 17, 2017

Have we resolved this one?

@ffilz ffilz closed this Feb 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment