NFS Ganesha Debug Logging

DEBUG logs help in understanding the code flow and narrowing down/solving a problem. NFS Ganesha supports various levels of debug logging for its components. This helps in enabling debug log (of desired level) for one or more or all components.

NFS Ganesha component list

Component Name	Description
ALL	Setting this turns on debug for all log components, it generates a lot of log messages and has a huge performance impact
LOG	Useful for debugging logging related issues
MEM_ALLOC	Useful for tracking memory allocations
MEMLEAKS / LEAKS	Useful for debugging/tracking memory leaks
EXPORT	Useful for debugging export configuration issues and problems with client access to exports
FSAL	Useful for debugging issues with Ganesha’s interface to FSAL ( like GPFS/SCALE. CephFS, VFS etc)
NFS_V4 / NFS4	Useful alongside the other components for knowing what NFSv4 operation is in progress
CACHE_INODE / INODE / MDCACHE	Useful to diagnose problems with attribute refresh and memory use
CONFIG	Useful to diagnose problems with configuration
NFS_READDIR	Useful to diagnose problems with directory listing
STATE	Useful to diagnose problems with opens and byte range locks
NFS3	Useful for debugging NFSv3 protocol understanding and related issues
NFSPROTO	Useful for debugging generic NFS protocol in general
FILEHANDLE / FH	Useful for debugging filehandle related issues
DISPATCH / DISP
CACHE_INODE_LRU / MDCACHE_LRU / INODE_LRU	Useful for debugging cache management related issues
HASHTABLE / HT	Useful for debugging hashing / hashtable related issues
HASHTABLE_CACHE / HT_CACHE	Useful for debugging hashtable caching issues
DUPREQ	Useful for debugging duplicate requests related issues
INIT	Useful for debugging NFS server initialization issues
NFS_STARTUP	Useful for debugging NFS server start-up issues
MAIN	Helps in NFS server main function debugging
IDMAPPER	Useful for debugging ID mapping related issues
NFS_V4_LOCK / NFS4_LOCK	Useful for debugging NFSv4 locking/unlocking related issues
CLIENTID	Useful for debugging client ID related issues (NFSv4)
SESSIONS	Useful for debugging client sessions (NFSv4)
PNFS	Useful for debugging pNFS
RW_LOCK	Useful for debugging read/write locking related issues
NLM	Useful for debugging NLM related issues
RPC	Useful for debugging RPC related issues
TIRPC	Useful for debugging NTIRPC library related issues
NFS_CB	Useful for debugging call back related issues
THREAD	Useful for debugging threading issue in NFS server
NFS_V4_ACL / NFS4_ACL	Useful for debugging NFSv4 ACL related issues
_9P	Useful for debugging 9P protocol
_9P_DISPATCH / _9P_DISP	Useful for debugging 9P protocol dispatch related issues
FSAL_UP	Useful for debugging FSAL upcalls
DBUS	Useful for debugging DBUS interface with NFS server

NFS Ganesha supports various levels of debug log for every component : NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG / M_DBG, and FULL_DEBUG / F_DBG.

Below sections talks more about NFS Ganesha in Ceph setup

Enabling NFS Ganesha debug logging dynamically (without restart)

In Ceph (version 7), the NFS Ganesha is deployed in container. The /etc/ganesha/ganesha.conf file in container is exported from the host system. So if one modifies the file on the host, NFS Ganesha in container sees the changes/updates. This is useful for enabling debug logging dynamically.

Follow below steps to enable / disable logging
On the container host, the config file is located at: /var/lib/ceph/${FSID}/nfs.${DAEMON_ID}/etc/ganesha/ganesha.conf
Edit above  ganesha.conf file and  add the stanza for logging
Get the PID of the Ganesha process, and send SIGHUP.
This will reread the ganesha.conf file and new logging levels will be in effect.

[root@ceph-mani-cst2nb-node2 ganesha]# ps -eaf | grep ganesha
root     1103935 1103933  0 04:39 ?        00:00:00 /run/podman-init -- /usr/bin/ganesha.nfsd -F -L STDERR -N NIV_EVENT
root     1103937 1103935  0 04:39 ?        00:00:02 /usr/bin/ganesha.nfsd -F -L STDERR -N NIV_EVENT   <== Send SIGHUP
root     1121642 1092342  0 05:36 pts/1    00:00:00 grep --color=auto ganesha
[root@ceph-mani-cst2nb-node2 ganesha]# kill -SIGHUP 1103937

To go back to previous logging level, again edit the conf file & send SIGHUP signal as described above.  

Enabling NFS Ganesha debug logging on restart

To enable logging, use command “ceph nfs cluster config set ..”.
Ceph nfs cluster config set <cluster_id> -i <config_file>
Eg: to change LOG level to DEBUG update config with

LOG {
    COMPONENTS {
        NFS4 = FULL_DEBUG;
    }
}

This creates an additional rados object for the user configuration
This object is added to the rados object holding information on all exports

[ceph: root@ceph-mani-cst2nb-node1-installer /]# rados get -N c_ganesha -p ".nfs" conf-nfs.c_ganesha /tmp/conf
[ceph: root@ceph-mani-cst2nb-node1-installer /]# cat /tmp/conf
 %url "rados://.nfs/c_ganesha/export-1"
 ...
 ...
 %url "rados://.nfs/c_ganesha/userconf-nfs.c_ganesha"
[ceph: root@ceph-mani-cst2nb-node1-installer /]# rados get -N c_ganesha -p ".nfs" userconf-nfs.c_ganesha /tmp/user_conf
[ceph: root@ceph-mani-cst2nb-node1-installer /]# cat /tmp/user_conf
 LOG {
      COMPONENTS {
         NFS4 = FULL_DEBUG;
     }
   }

To return logging to normal, run command “ceph nfs cluster config reset ..”.
Note that NFS Ganesha restarts when one sets/resets the debug logging.