Unable to find tracefs in different kernel so kernel tracing fails

[honggyukim]

The current way of finding tracefs for kernel tracing was implemented in #1476.

It assumes that /proc/<pid>/mountinfo looks as follows.

36 35 98:0 /mnt1 /mnt2 rw,noatime master:1 - ext3 .... ....

So it detects the file system type from the /proc/self/mountinfo. For example, the following input can be properly parsed and successfully finds mount_point and fs_type as written in find_tracing_dir.

$ cat /proc/self/mountinfo | grep tracefs
40 23 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:18 - tracefs tracefs rw

However, it fails detecting fs_type from the following my qemu rootfs and it makes kernel tracing disabled. The following line has one less word as shown as shared:18 in the above example.

root@qemux86-64:~# cat /proc/self/mountinfo | grep tracefs
26 22 0:10 / /sys/kernel/debug/tracing rw,relatime - tracefs tracefs rw

As I mentioned at #1471 (comment), I think we should use getmntent to find the mount point.

[honggyukim]

As written at #1471 (comment), the following example shows the clear usage.

$ cat getmntent.c
#include <stdio.h>
#include <stdlib.h>
#include <mntent.h>

int main(void)
{
  struct mntent *ent;
  FILE *aFile;

  aFile = setmntent("/proc/mounts", "r");
  if (aFile == NULL) {
    perror("setmntent");
    exit(1);
  }
  while (NULL != (ent = getmntent(aFile))) {
    printf("%s %s\n", ent->mnt_fsname, ent->mnt_dir);
  }
  endmntent(aFile);
}

It can simply be compiled as follows.

$ gcc getmntent.c

Then the output shows it find tracefs correctly.

$ ./a.out | grep tracefs
tracefs /sys/kernel/tracing

It also successfully finds the tracefs in my qemu rootfs.

root@qemux86-64:~# ./a.out | grep tracefs
tracefs /sys/kernel/debug/tracing

[gichoel]

I've seen issues #797, #1476, #1471(Comments-1) and the following is a list that needs to be checked again in the previous issues:

1. Does Android support mntent.h? (in #1476)
   There was a question about whether mntent.h is available in Android, and it's confirmed in #1471 (comments-2), so I don't think it's a problem.

2. Which of /proc/self/mountinfo or mntent.h was introduced first? (in #1476)
   a) According to proc_manpage, /proc/self/mountinfo can be introduced starting with Kernel 2.6.26.
   b) It is unknown when mntent.h was introduced into the kernel, but according to getmntent_manpage, it is a structure and code defined in the Standard C Library (libc), so it seems that each libc implementation must be checked.

   • glibc first confirmed getmntent() in 2.0.2 (link)
   • uClibc first confirmed getmntent() in 0.9.8 (link)

In conclusion, the above method using struct mntent will not be a problem, and I think the method using struct mntent is a better solution because it is an approach using structured data structures.

If there is no problem after reviewing the above, I'll try it.

[honggyukim]

Hi @gichoel,

Thanks a lot for the analysis. I've just tested that there is no problem compiling setmntent/getmntent in Android build. So please go ahead replacing the logic with them. Thanks.

[gichoel]

In order to solve this Issue, I thought that three conditions must be met in the code.

1. #797 states that debugfs can also be used, so you should support both tracefs and debugfs instead of just using tracefs.

   • I don't know much about the Linux kernel yet, but since it's constantly evolving, I thought that features that didn't exist before could come out later.
   • Therefore, we wanted to consider scalability in case the Linux kernel provides additional special file systems such as tracefs or debugfs.

2. Reduce unnecessary movements within while (NULL != (ent = getmntent(aFile))) { ... }

   • When you use the while statement to read the /proc/mounts file line by line, you must exit if you find the desired string (for example, tracefs path).
   • However, the first problem is that if you use strcmp to create a if statement to find debugfs or tracefs, tracefs will not appear when debugfs come first. (Debugfs is Plan B, so I wanted to use tracefs first if I had tracefs.) And the second problem is that if we use strcmp to find only tracefs paths, we don't satisfy item 1.
   • Therefore, if using assign priority to a path, exit the while statement and prevent subsequent rows from being read from the file to reduce unnecessary activity.

3. Declare variables and write code with minimal performance overhead

   • I always try to satisfy this part, but it may not be because I'm still inexperienced. 😅
   • So I'd like to hear other people's opinions on my code if I post a PR on this part soon (maybe within 1 day).

---

My code based on the above conditions are as follows.

$ cat tracefs_unit_test.c
#define _GNU_SOURCE //for asprintf()
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <mntent.h>

#define true 1
#define false 0

#define PROC_MOUNTS_PATH "/proc/mounts"
#define TRACEFS_STR		 "tracefs"
#define DEBUGFS_STR		 "debugfs"

static char *TRACING_DIR = NULL;

enum TRACE_DIR_PRIORITY {
	PRIORITY_FIRST = 0,
	PRIORITY_SECOND,
	PRIORITY_THIRD,
	PRIORITY_NOT_ASSIGNED,
	PRIORITY_END
};

int get_tracing_dir_priority(const char *tracing_dir_name) {
	if( !strcmp(tracing_dir_name, TRACEFS_STR) ) 	   { return PRIORITY_FIRST; }
	else if ( !strcmp(tracing_dir_name, DEBUGFS_STR) ) { return PRIORITY_SECOND; }
	
	return PRIORITY_NOT_ASSIGNED;
}

static int find_tracing_dir(void) {
	struct mntent *ent;
	FILE *fp;

	/* open a file with the currently mounted list */
	fp = setmntent(PROC_MOUNTS_PATH, "r");
	if( fp == NULL ) {
		printf("setmntent() return NULL\n");
		return false;
	}

	/* compare if "tracefs" or "debugfs" exist in list mounted until EOF(End of File) */
	while( NULL != (ent = getmntent(fp)) ) {
		printf("%s : %s\n", ent->mnt_fsname, ent->mnt_dir);
		if( !strcmp(ent->mnt_fsname, TRACEFS_STR) ||
			!strcmp(ent->mnt_fsname, DEBUGFS_STR) )
		{
			if(TRACING_DIR != NULL) {
				free(TRACING_DIR);
			}

			asprintf(&TRACING_DIR, "%s", ent->mnt_dir);
		}

		if( get_tracing_dir_priority(ent->mnt_fsname) == PRIORITY_FIRST ){
			break;
		}
	}

	/* close mntent fp */
	endmntent(fp);

	if(TRACING_DIR != NULL)
		return true;

	return false;
}

int main() {
	if(find_tracing_dir() == true){
		printf("\n");
		printf("TRACING_DIR : %s\n", TRACING_DIR);
		printf("\n");
	}

	return 0;
}

It can simply be compiled as follows.

$ gcc tracefs_unit_test.c

In my case (maybe it's because the results may vary depending on the OS), when I run the program, I get the result of finding tracefs normally even if debugfs exists first as shown below.

$ ./a.out
sysfs : /sys
proc : /proc
udev : /dev
devpts : /dev/pts
tmpfs : /run
/dev/sda3 : /
securityfs : /sys/kernel/security
tmpfs : /dev/shm
tmpfs : /run/lock
cgroup2 : /sys/fs/cgroup
pstore : /sys/fs/pstore
bpf : /sys/fs/bpf
systemd-1 : /proc/sys/fs/binfmt_misc
hugetlbfs : /dev/hugepages
mqueue : /dev/mqueue
debugfs : /sys/kernel/debug
tracefs : /sys/kernel/tracing

TRACING_DIR : /sys/kernel/tracing

[honggyukim]

Hi @gichoel, thanks for your work. But I started thinking that if it's really needed to make the logic complicated.

As shown in the commit torvalds/linux@cc31004, it says that the location of tracefs is set to /sys/kernel/tracing so better to use this fixed location. This patch was in since Linux v4.1 so most of modern systems will use this.

I think we better make the logic simpler as follows.

1. check if /sys/kernel/tracing/tracing_on file exists.
2. If exits, then the TRACING_DIR is set to /sys/kernel/tracing.
3. Otherwise, find debugfs using setmntent/getmntent.
4. if debugfs is found, then check if debugfs + /tracing/tracing_on file exits.
5. if the file is found, set TRACING_DIR to debugfs + /tracing.
6. Otherwise, make it fail.

[gichoel]

Hi @honggyukim, Thanks for suggesting a better logic.

But I did not understand that tracefs, which is the content of the Linux kernel patch is fixed to /sys/kernel/tracing.
Because in the comments, the path of tracefs was saying as follows.

root@qemux86-64:~# ./a.out | grep tracefs
tracefs /sys/kernel/debug/tracing

But after a few additional searches, I was able to understand the results of qemu rootfs and realized that I didn't know about the relationship between debugfs and tracefs.

• According to the ftrace.txt in kernel.org, there was no tracefs prior to Linux kernel v4.1 and subpath /sys/kernel/debug/tracing in debugfs replaces the tracingfs role.
  • on systems running Linux kernel v4.1 or later, tracefs mounts /sys/kernel/debug/tracing if debugfs is mounted for backward compatibility.
  • tracefs will eventually have the same list of files as the /sys/kernel/debug/tracing folder, which is the subpath of debugfs. (of course, the list of files in the folder is the same as the results checked by my system.)

So I understand why setmntent and getmntent are used to get the debugfs path if the /sys/kernel/tracing/tracing_on file does not exist.

If /sys/kernel/tracing/tracing_on exists, I think the proposed new logic is more efficient because do not proceed with the additional path find process.

I will test the unit test code according to the logic you suggested, and finally I will run it in utrace in the local environment.

[honggyukim]

That's exactly what I was going to explain. Please send a PR when you're ready. Thanks.

[honggyukim]

The commit torvalds/linux@cc31004 says

When tracefs is configured, have the directory /sys/kernel/tracing appear
just like /sys/kernel/debug appears when debugfs is configured.

This will give a consistent place for system admins to mount tracefs.

The author of ftrace and tracefs, Steven Rostedt, mentioned that the location /sys/kernel/tracing will be a "consistent place" so I think we can trust that the location won't be changed.

[namhyung]

The common way to check tracefs is:

1. call statfs with /sys/kernel/tracing and check if f_type is TRACEFS_MAGIC.
2. do the same with /sys/kernel/debug/tracing.
3. parse /proc/mounts.

[gichoel]

Hi @namhyung, thanks for suggesting new ideas.

Since I don't know about statfs(), I did some research.

In the corresponding statfs_manpage, the Magic Number for tracefs and debugfs was confirmed as follows.

TRACEFS_MAGIC 0x74726163
DEBUGFS_MAGIC 0x64626720

However, I have a questions.

• deprecated functions issue

  • since statfs() and fstatfs() are deprecated in the corresponding statfs_manpage, statvfs() and fstatvfs() are recommended as replacement functions.

  • Therefore, I am going to implement the recommended idea using statvfs(). Could this be a problem?

I like the fact that the logic is a bit simpler. Thank you

[gichoel]

I tried to implement the code using statvfs().

However, no matter how much I looked at the glibc code and the linux kernel code. I couldn't find a field that serves as the f_type of statfs() in statvfs().

Therefore, the unit test code using statfs() was written as follows.

• The reason for removing the part checked with the existing priority in find_tracing_path is as follows.

• According to the contents of the comment, it was judged that the priority is meaningless because tracefs and debugfs same file list.

cat tracefs_unit_test2.c
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#include <sys/vfs.h>
#include <mntent.h>

#define true 1
#define false 0

#define PROC_MOUNTS_PATH "/proc/mounts"

#define TRACEFS_DIR_PATH "/sys/kernel/tracing"
#ifndef TRACEFS_MAGIC
	#define TRACEFS_MAGIC 	 0x74726163
#endif

#define DEBUGFS_DIR_PATH "/sys/kernel/debug/tracing"
#ifndef DEBUGFS_MAGIC
	#define DEBUGFS_MAGIC 	 0x64626720
#endif

char *find_tracing_path(void) {
	struct mntent *ent;
	char *tracing_dir=NULL;
	FILE *fp;

	/* open a file with the currently mounted list */
	fp = setmntent(PROC_MOUNTS_PATH, "r");
	if( fp == NULL ) {
		printf("setmntent() return NULL\n");
		return NULL;
	}

	/* compare if "tracefs" or "debugfs" exist in list mounted until EOF(End of File) */
	while( NULL != (ent = getmntent(fp)) ) {
		if( !strcmp(ent->mnt_fsname, "tracefs") ||
			!strcmp(ent->mnt_fsname, "debugfs") )
		{
			asprintf(&tracing_dir, "%s", ent->mnt_dir);
		}
	}

	/* close mntent fp */
	endmntent(fp);

	if(tracing_dir != NULL) {
		return tracing_dir;
	}

	return NULL;
}

char *get_tracing_file(const char *name) {
	struct statfs fs;
	char *tracing_path = NULL;

	/* check default tracefs path is mounted */
	statfs(TRACEFS_DIR_PATH, &fs);
	if( (unsigned long) fs.f_type == TRACEFS_MAGIC ) {
		printf("[statfs result] f_type: %#lx\n",  (unsigned long) fs.f_type);

		asprintf(&tracing_path, "%s", TRACEFS_DIR_PATH);
		goto success;
	}

	/* check default debugfs path is mounted */
	statfs(DEBUGFS_DIR_PATH, &fs);
	if( (unsigned long) fs.f_type == DEBUGFS_MAGIC ) {
		printf("[statfs result] f_type: %#lx\n",  (unsigned long) fs.f_type);

		asprintf(&tracing_path, "%s", DEBUGFS_DIR_PATH);
		goto success;
	}

	/* last attempt with /proc/mounts */
	tracing_path = find_tracing_path();
	if( tracing_path != NULL ) {
		goto success;
	}

	return NULL;

success:
	asprintf(&tracing_path, "%s/%s", tracing_path, name);
	return tracing_path;
}

int main(void){
	char *tracing_path = NULL;

	tracing_path = get_tracing_file("tracing_on");
	if(tracing_path == NULL) {
		return 1;
	}

    printf("\n");
	printf("tracing file path : %s\n", tracing_path);
}

It can simply be compiled as follows.

$ gcc tracefs_unit_test2.c

The result of running my program is as follows

$ ./a.out

[statfs result] f_type: 0x74726163

tracing file path : /sys/kernel/tracing/tracing_on

• My doubts still exist.
  • Is it safe to use deprecated functions?

[honggyukim]

Hmm.. I think this is getting too much complicated unnecessarily. In most cases, the TRACING_DIR will be either /sys/kernel/tracing or /sys/kernel/debug/tracing. So I think we better make it simple as follows.

1. try /sys/kernel/tracing
2. try /sys/kernel/debug/tracing
3. otherwise, make it fail

I think we better make the logic simpler first as above then think about rewriting the fall back routine in a separate PR.