-
Notifications
You must be signed in to change notification settings - Fork 259
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
time64 issue in simple program on 32-bit arm, linux 5.6 #96
Comments
You should probably raise this issue on the musl list, as it's almost certainly not something specific to the toolchain build system. It may be a kernel bug. Have you run strace to see if actual syscalls or vdso is being used? |
Also can you try with just two successive calls to clock_gettime in the same program? gettimeofday is just a thin wrapper around clock_gettime so if there's a kernel bug causing erratic results you should see it without bringing in gettimeofday. If it only happens with joint use of clock_gettime and gettimeofday there may be a gcc bug causing something to get miscompiled. |
Also, by qemu do you mean qemu user-level emulation, or system emulation? (If the latter, what kernel?) |
kernel bug: glibc is ok, so I'm assuming the kernel is basically ok. New program#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <sys/time.h>
#include <time.h>
int main(int argc, char**argv) {
struct timespec ts[2];
clock_gettime(CLOCK_REALTIME,&ts[0]);
clock_gettime(CLOCK_REALTIME,&ts[1]);
printf(
"0:%lld %lld\n"
"1:%lld %lld\n"
"-:%lld %lld\n",
(long long) ts[0].tv_sec,
(long long) ts[0].tv_nsec,
(long long) ts[1].tv_sec,
(long long) ts[1].tv_nsec,
(long long) ts[1].tv_sec - (long long) ts[0].tv_sec,
(long long) ts[1].tv_nsec - (long long) ts[0].tv_nsec
);
} ExpectedHere is a run on the program compiled by debian's gcc (with glibc):
On a 32-bit arm system using kernel 5.1 the output is similar (though no strace available). ActualBuilding with the same musl-cross-make toolchain (just to let you know the compiler setup is reproducible) $ ./output/bin/arm-pi-linux-musleabihf -o clock_gettime clock_gettime.c -static And then running on kernel 5.6 on 32 bit arm hardware: $ ./clock_gettime && strace ./clock_gettime
0:1597599551 811233074
1:1598270631 916673074
-:671080 105440000
execve("./clock_gettime", ["./clock_gettime"], 0xbed63dc0 /* 15 vars */) = 0
set_tls(0x25554) = 0
set_tid_address(0x25604) = 1270
ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
writev(1, [{iov_base="0:1599277252 74833074", iov_len=21}, {iov_base="\n", iov_len=1}], 20:1599277252 74833074
) = 22
writev(1, [{iov_base="1:1600116102 206633074", iov_len=22}, {iov_base="\n", iov_len=1}], 21:1600116102 206633074
) = 23
writev(1, [{iov_base="-:838850 131800000", iov_len=18}, {iov_base="\n", iov_len=1}], 2-:838850 131800000
) = 19
exit_group(0) = ?
+++ exited with 0 +++ |
glibc has 32-bit time, so "glibc is ok" doesn't say anything about whether the kernel is broken. |
And yes, your strace shows that vdso is being used, and apparently it's buggy and producing wrong results (scaling the clock counter by something way too large or accessing the 64-bit value in the wrong way or something similar). |
I wrote one that just uses the syscall pretty directly, and it appears to work well. Seconds go up by 1 about once per second :-) Program#include <stdio.h>
#include <unistd.h>
#include <time.h>
#include <syscall.h>
int main(int argc, char**argv) {
struct timespec ts[1];
syscall(SYS_clock_gettime64, CLOCK_REALTIME, ts);
printf("sc: %lld %lld\n",
(long long) ts->tv_sec, (long long) ts->tv_nsec
);
} Actual = ExpectedNotice seconds go up by one after waiting one second. $ ./syscall
sc: 1587855857 652015087
$ ./syscall
sc: 1587855858 711929367
$ strace ./syscall
strace ./syscall
execve("./syscall", ["./syscall"], 0xbe8e3dc0 /* 15 vars */) = 0
set_tls(0x2554c) = 0
set_tid_address(0x255fc) = 1419
clock_gettime64(CLOCK_REALTIME, {tv_sec=1587855990, tv_nsec=934739267}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
writev(1, [{iov_base="sc: 1587855990 934739267", iov_len=24}, {iov_base="\n", iov_len=1}], 2sc: 1587855990 934739267
) = 25
exit_group(0) = ?
+++ exited with 0 +++ I bet I've done something wrong since glibc crashes:
|
Do you think the kernel is putting wrong things in vdso or is musl misinterpreting the contents? Is there a simple way to help debug that? The kernel itself is more or less "make defconfig". |
VDSO is just functions the kernel exports. The kernel is definitely buggy. We need to figure out how this is going to get fixed in a way that buggy ones don't get used which will require coordination with kernel folks... |
Ok. I tried on a very default kernel for the hardware. Same behavior. I checked the hex version of the time, and didn't see any obvious patterns. Reproducible kernelkernel is: hardware is stock raspberry pi 4b rev 1.2 Basically, this is just following the instructions at https://www.raspberrypi.org/documentation/linux/kernel/building.md
Hex version of syscall vs vdsoI checked that it is probably not a bit-issue: #include <stdio.h>
#include <unistd.h>
#include <time.h>
#include <syscall.h>
int main(int argc, char**argv) {
struct timespec ts_sc[1];
struct timespec ts_vd[1];
syscall(SYS_clock_gettime64, CLOCK_REALTIME, ts_sc);
clock_gettime(CLOCK_REALTIME, ts_vd);
printf(
"vd: %lld %lld (%llx %llx)\n"
"sc: %lld %lld (%llx %llx)\n",
(long long) ts_vd->tv_sec, (long long) ts_vd->tv_nsec,
(long long) ts_vd->tv_sec, (long long) ts_vd->tv_nsec,
(long long) ts_sc->tv_sec, (long long) ts_sc->tv_nsec,
(long long) ts_sc->tv_sec, (long long) ts_sc->tv_nsec
);
} has output:
|
Yes, it's not at all surprising that the bug is present in the default kernel for the hardware. Nobody else (i.e. not glibc) is using 64-bit time so it's not expected that anyone else would have noticed the bug because the bug is in an interface they're not using. We're going to have to drop support for vdso clock_gettime on ARM until there's a fix that makes it possible to reliably detect that the vdso isn't broken (such a fix would require new symbol name or version when re-enabling it anyway, I think). You can apply this fix locally just by removing all of the |
Should be fixed in mcm by d6ded50. This still needs further action for kernel fix and upstream musl to avoid using broken vdso so I won't be closing the issue here until the report is acked/handed-off there. |
Thank you! The fix works for me. |
Thanks for confirming. |
Has this been reported to the kernel developers (linux-arm-kernel@lists.infradead.org)? I couldn't see anything in the archives. |
Thanks for the ping. I'll check. |
@jackschmidt can you bulid this code and run it on the kernel where you observed the issue and send us vdso.so:
also please confirm that the exact same kernel works in qemu-system-arm with musl (before the vdso usage was reverted) and the bug is only reproducible on rpi hw (which means this is likely a hw bug). |
@jackschmidt please explain what that patch is? |
@jackschmidt what makes you think it affects other hw? i don't see the problem on a cortex a72 with same toolchain and 5.6 arm 32bit kernel, i'm building the rpi kernel now, but i wont be able to test the exact same setup as that's rpi specific (but i dont see there anything that would change vdso time64 behaviour compared to upstream 5.6 kernel). |
vdso.so attached (gzip'd only because github required it) Let me know if you want me to change |
OK, @jackschmidt uncovered something useful even if not explaining it well. There's a function in the arm kernel code that kills the vdso functions if the virtual counter is missing. It's documented in comments as just being an optimization, to avoid going through a codepath that will fail. But I wonder if the code was actually broken. Without the above-cited patch from #96 (comment), (knaerzche/LibreELEC.tv@3efb847) the time64 version of the function was not suppressed. |
Yes, sorry. I suspect I was misreading @nsz-arm's messages. I don't use KODI, but it is a very popular arm OS, and libreelec is a popular way to run kodi on single board computers other than raspberry pi. They had a patch mentioned at raspberrypi/linux#3579 (comment) -- so credit to @popcornmix |
OK, that theory matches the vdso.so @jackschmidt shared. It has |
|
An issue was reported recently where a bug was found during successive reading of 64 bit time on arm32 platforms. Add a test for that. richfelker/musl-cross-make#96 Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> Acked-by: Arnd Bergmann <arnd@arndb.de> Reviewed-by: Cyril Hrubis <chrubis@suse.cz>
Using musl-cross-make 5086175 (Mar 6th 2020) the following program produces incorrect (erratic) results on arm hardware with linux kernel 5.6 (but correct results on 32-bit arm kernel 4.19, correct results in qemu, and correct results on 32-bit arm kernel 5.6 if compiled with debian's gcc+glibc).
Program
compiled with:
Expected
Something like this is produced in qemu and on arm hardware running kernel 4.19. When running on arm hardware with kernel 5.6, the actual results are erratic. Notice the first integers (seconds) agree, and the second integers are close, after multiplying the top by 1000 (microseconds vs nanoseconds).
Actual
When running on actual arm hardware with kernel 5.6, the following are illustrative. Notice the seconds don't agree with each other and running the program one second later, the seconds have not incremented by 1.
config.mak
I ran
make clean
and thenmake install
(and waited a while). The build host is a stock arch linux onx86_64
.The text was updated successfully, but these errors were encountered: