Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uv_uptime returns incorrect values on OpenVZ containers. #3068

Closed
schamberg97 opened this issue Dec 13, 2020 · 3 comments
Closed

uv_uptime returns incorrect values on OpenVZ containers. #3068

schamberg97 opened this issue Dec 13, 2020 · 3 comments

Comments

@schamberg97
Copy link
Contributor

schamberg97 commented Dec 13, 2020

  • Version: All current versions, to the best of my knowledge
  • Platform: Any Linux container in OpenVZ/Virtuozzo, e.g.: Linux vps1607355603 4.15.0 #1 SMP Tue Aug 25 11:59:26 MSK 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Related Node.js issue: os.uptime returns the wrong value nodejs/node#36244 (for our exploration of the issue)

libuv/src/unix/linux-core.c

Lines 606 to 628 in 9c6cec8

int uv_uptime(double* uptime) {
static volatile int no_clock_boottime;
struct timespec now;
int r;
/* Try CLOCK_BOOTTIME first, fall back to CLOCK_MONOTONIC if not available
* (pre-2.6.39 kernels). CLOCK_MONOTONIC doesn't increase when the system
* is suspended.
*/
if (no_clock_boottime) {
retry: r = clock_gettime(CLOCK_MONOTONIC, &now);
}
else if ((r = clock_gettime(CLOCK_BOOTTIME, &now)) && errno == EINVAL) {
no_clock_boottime = 1;
goto retry;
}
if (r)
return UV__ERR(errno);
*uptime = now.tv_sec;
return 0;
}

Hey there :) It appears that libuv in OpenVZ environments returns incorrect values. For instance, if one were to check the uptime using cat /proc/uptime and Node.js's os.uptime(), one would get the following results respectively: 3187731.82 and 6740816. In other words, libuv returns results that are higher than container's uptime. Same erroneous result is achieved with this code provided by @gireeshpunathil:

#include <time.h>
#include <errno.h>
#include <stdio.h>

int main() {
  struct timespec now;
  int r;

  r = clock_gettime(CLOCK_BOOTTIME, &now);
  if (r != 0) {
    fprintf(stderr, "clock_gettime failed with %d\n", errno);
    return errno;
  }
  fprintf(stderr, "system uptime: %ld\n", now.tv_sec);
  return 0;
}

To get into the root of the problem, we need to understand how OpenVZ operates. OpenVZ consists of a host Linux system with highly patched kernel, which then runs the containers. Unfortunately, all containers share the host kernel, but are provided with a separate simulation of /proc/ (procfs) filesystem for each container. Thus, the earlier mentioned 6740816 uptime from libuv coincides with the uptime of the physical host system.

In practice, this means that the clock_gettime is inherently unreliable on OpenVZ systems. It is suggested by @davedodo on a related node.js issue nodejs/node#36244 (comment) that sysinfo function works correctly with OpenVZ systems (like in Cygwin implementation). However, there might be other alternatives that I (we?) are unaware of.

It is my understanding that this issue concerns only OpenVZ/Virtuozo. Thus, in order to check, whether the system is an OpenVZ container, it appears sufficient to check that /proc/vz/veinfo file exists at launch time (it contains container ID, public IP of current container and current number of running processes).

I am not providing any PR at this point, because this issue may involve quite a bit of decision making on this issue, with a non-exhausting list of three alternatives:

  1. Rewrite uv_uptime entirely using sysinfo for all cases
  2. Check for OpenVZ at launch time to avoid clock_gettime in uv_uptime on OpenVZ and use sysinfo instead
  3. Ignore the issue
@bnoordhuis
Copy link
Member

Option 1 is probably off the table.

In 3a1be72 and 1c22b44 I switched some things from sysinfo() to /proc because of compatibility issues with lxc, another virtualization solution.

I think (not 100% sure) lxc intercepts clock_gettime() by remapping the vDSO - meaning it Just Works from libuv's perspective - but it does no such thing for sysinfo().

What could probably work - and is virtualization agnostic - is to read the first field from /proc/uptime if that file exists (the second field is the system idle time) with a fallback to clock_gettime(). Pull request welcome.

Parsing a file is much more expensive than a vDSO "system call" (it's usually serviced from the vDSO without making an actual system call) but I don't think that matters for uv_uptime(), it's not performance critical.

@schamberg97
Copy link
Contributor Author

Option 1 is probably off the table.

In 3a1be72 and 1c22b44 I switched some things from sysinfo() to /proc because of compatibility issues with lxc, another virtualization solution.

I think (not 100% sure) lxc intercepts clock_gettime() by remapping the vDSO - meaning it Just Works from libuv's perspective - but it does no such thing for sysinfo().

What could probably work - and is virtualization agnostic - is to read the first field from /proc/uptime if that file exists (the second field is the system idle time) with a fallback to clock_gettime(). Pull request welcome.

Parsing a file is much more expensive than a vDSO "system call" (it's usually serviced from the vDSO without making an actual system call) but I don't think that matters for uv_uptime(), it's not performance critical.

That’s how I saw the solution as well initially, but I had some aversion to it, as it is more expensive. I could make a PR today, if I am not disturbed by other matters. Otherwise, I would have no problem if someone else does it.

@bnoordhuis
Copy link
Member

I'd say go for it. :-)

schamberg97 added a commit to schamberg97/libuv that referenced this issue Dec 14, 2020
This change fixes libuv#3068 on OpenVZ Containers in Linux
santigimeno pushed a commit to santigimeno/libuv that referenced this issue Dec 28, 2020
First check `/proc/uptime`, then fallback to `clock_gettime()`.

Fixes: libuv#3068
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
santigimeno pushed a commit that referenced this issue Dec 28, 2020
First check `/proc/uptime`, then fallback to `clock_gettime()`.

Fixes: #3068
PR-URL: #3072
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
JeffroMF pushed a commit to JeffroMF/libuv that referenced this issue May 16, 2022
First check `/proc/uptime`, then fallback to `clock_gettime()`.

Fixes: libuv#3068
PR-URL: libuv#3072
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-by: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Santiago Gimeno <santiago.gimeno@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants