64-bit counters stopped working on 32-bit kernel after pull #298

[anphsw]

Facing problem with running pppd on linux 32-bit kernel releated to this commit:
81ad945

It seems like 32-bit kernel exporting only 32-bit counters to sysfs, and this commit changed default way of getting counters.

	if (get_ppp_stats_sysfs(u, stats)) {
	    func = get_ppp_stats_sysfs;
	    return 1;
	}
	warn("statistics falling back to ioctl which only supports 32-bit counters");
	func = get_ppp_stats_ioctl;
	TIMEOUT(ppp_stats_poller, (void*)(long)u, 25);
    }

Function get_ppp_stats_sysfs succeeds and then we rewrite default to get_ppp_stats_sysfs. But in 32-bit kernel it returns only 32-bit counters (at least for ppp) and we are not checking for overlap.
After removing get_ppp_stats_sysfs call, got 64-bit counters working again as get_ppp_stats_ioctl is overlap-friendly.
I think we must detect kernel at runtime (not compile time, as we can run 32-bit pppd on 64-bit kernel), and if it 32-bit, then do not use sysfs stats.

[Neustradamus]

@jkroonza, @paulusmack: Have you seen this ticket?

[jkroonza]

No, but there was a similar issue, I could never reproduce, and I can't find it now.

This ticket has more information than that one had, and is likely the same issue.

@anphsw

Under the note that overrun can occur within check time unless the check interval is frequent enough (at what rate are you not willing to get incorrect values?)

I do see code that's supposed to detect wraps in the ioctl() path, but this does not seem replicated to the sysfs path ... I'm thinking there's only really two solutions here:

1. On 32-bit kernels, just avoid the sysfs code completely and stick to ioctl().
2. Add the wrap detection code to sysfs.

I'm inclined to think that option 1 is probably best (no parsing required). I'm however not sure what the most reliable way to detect a running 32-bit kernel (a 32-bit process on a 64-bit kernel can still utilize the 64-bit counters directly I believe in an x32 setup, or multilib, definitely the latter, and x32 wasn't ever really a thing although I wish more semi-embedded setups would have embraced that).

uname(2) provides the following information on my machine at least, so the machine string is the only potentially useful bit:

$ ./uname
sysname: Linux
nodename: plastiekpoot
release: 6.18.2-uls
version: #1 SMP Wed Dec 31 09:22:39 SAST 2025
machine: x86_64

So from that perspective, I'm swinging towards option 2 - but starting yet another pointless timer just irks me.

[anphsw]

Under the note that overrun can occur within check time unless the check interval is frequent enough (at what rate are you not willing to get incorrect values?)

Current timer function have 25s interval and ~1.37gbps counter overlap. It's more than enough for such rare case.

1. On 32-bit kernels, just avoid the sysfs code completely and stick to ioctl().

Seems like more preferrable to me too, as 32-bit kernel may have another overlap conditions such as stuck (at 2^32) counter.
Is there any disadvantages of using ioctl at all?

uname(2) provides the following information on my machine at least, so the machine string is the only potentially useful bit:

I think it can also report a lot of other exotic machines which running (or not) in 32 bit mode.
So it must be generic way to determine such overlap.
Maybe to set some kernel counter to 2^32 + 1 and read it again?

[jkroonza]

Yea, I set that to 25s in order to cover 1Gbps.

So the real questions remains: How to detect we're running on 32-bit kernels? We can determine at compile time from kernel headers if it's 32 or 64 bit I'm hoping, but that doesn't stop certain scenarios where 32-bit userspace code can run on 64-bit code, I don't think the other way is possible, so do we just rely on kernel header at compile time?

[anphsw]

It seems like falling back to ioctl does not give any disadvantages.
So maybe using something like sizeof(int) is good enough to detect 32 bit, so we relay only on compiler internals, but not kernel header?

[jkroonza]

That certainly is an option, but I disagree in that for us inaccurate counters is a huge problem. And we now have some PPPoE connections pushing past 1Gbps at times. So in the (unusual) case of 32-bit userspace with 64-bit kernel where 64 bit counters is available, I'd prefer to utilize them.

That said, since all the systems I care about is 64/64 user/kernel, your proposed solution will work for us.

@paulusmack perhaps some insight from you? As it stands I'm thinking of doing

--- a/pppd/sys-linux.c
+++ b/pppd/sys-linux.c
@@ -2005,6 +2005,9 @@ ppp_stats_poller(void* u)
  */
 int get_ppp_stats(int u, struct pppd_stats *stats)
 {
+#if sizeof(int) == 4
+    return get_ppp_stats_ioctl();
+#else
     static int (*func)(int, struct pppd_stats*) = NULL;
 
     if (!func) {
@@ -2022,6 +2025,7 @@ int get_ppp_stats(int u, struct pppd_stats *stats)
     }
 
     return func(u, stats);
+#endif
 }
 
 /********************************************************************

[paulusmack]

Looking at the Linux kernel code, it seems to me that the sysfs counters should be 64-bit even on 32-bit kernels. The sysfs file is implemented in net/core/net-sysfs.c, and the stats come from struct rtnl_link_stats64, defined in include/uapi/linux/if_link.h, and as the name suggests, the fields in struct rtnl_link_stats64 are of type u64. I can't see any reason why ppp would be different from other network interfaces in this respect, but I'll keep looking. On a 32-bit ARM system I have here the stats are definitely 64-bit for the ethernet interface. I'll do an experiment to see what happens with a PPP interface.

@anphsw what kernel version are you using?

[anphsw]

@paulusmack

it seems to me that the sysfs counters should be 64-bit even on 32-bit kernels

Just checked with 3.2.102 on i386 arch: ppp interface reporting 32 bit counters by sysfs and ifconfig (wraps at 32 bit)
Maybe problem is ppp-releated? Ethernet counters showing correct data (several TBs at least), but ppp interface counter wraps each 4GB (on sysfs and ifconfig).
It seems like kernel module reporting only 32bit counters on such old (but actively used in embedded systems) kernels.

[jkroonza]

Gemeni says:

32-bit Architecture Transition: Supporting 64-bit counters on 32-bit CPUs took longer, with improvements for 64-bit time and counter handling continuing into the 5.x era.

So I'd wager we need something a bit more intelligent on the ppp side, but no, this isn't a pppd issue per sé.

Using uname(2) we can certainly detect the appropriate kernel version where 64-bit is definitely available even on 32-bit.

Possibly then something like:

     static int (*func)(int, struct pppd_stats*) = NULL;
 
     if (!func) {
+#if sizeof(int) == 4
+       struct utsname buf;
+       if (uname(&buf) == 0 && version_compare(buf.release, "4.???") < 0) {
+           func = get_ppp_stats_ioctl;
+           TIMEOUT(ppp_stats_poller, (void*)(long)u, 25);
+           return func(u, stats);
+       }
+#endif
        if (get_ppp_stats_rtnetlink(u, stats)) {
            func = get_ppp_stats_rtnetlink;
            return 1;

Where version_compare will need to be implemented with semantics similar strcmp(), and the appropriate version from where sysfs has 64 bit counters will need to be determined. We may want additional error handing on the uname() call too. So this is conceptual code at best, complete garbage at worst.

[paulusmack]

I transferred about 40GB over a PPP link to a 32-bit system running a Linux 6.8.8 kernel, and the RX bytes count on that system didn't wrap at 4GiB. So it does appear to be only a problem with really old kernels.

We already have a kernel_version variable in sys-linux.c, so how about adding this to get_ppp_stats() ?

    if (sizeof(int) == 4 && kernel_version < KVERSION(5,0,0))
        return get_ppp_stats_ioctl();

[anphsw]

Searched in kernel source and found that 64-bit counters first appeared in kernel version 3.8 (struct ppp_link_stats in drivers/net/ppp/ppp_generic.c line 151)
And it seems like on pre 3.8 kernels 32-bit counter will be used regardless of build arch.

[anphsw]

First encounter of RTM_NEWSTATS was on 4.8 kernel, so proper logic may be:
kernel < 3.8 - use ioctl
3.8 < kernel < 4.8 - try to use sysfs, then ioctl
kernel > 4.8 try to use all possible variants

[jkroonza]

Because the kernel we compile against, and the kernel we run against might be different.