8259765: ZGC: Handle incorrect processor id reported by the operating system #124
Conversation
|
|
|
/cc hotspot-gc |
|
@pliden |
|
/cc hotspot-runtime |
|
@pliden |
Webrevs
|
src/hotspot/os/linux/os_linux.cpp
Outdated
| @@ -4746,19 +4746,46 @@ int os::active_processor_count() { | |||
| return active_cpus; | |||
| } | |||
|
|
|||
| static volatile int warn_invalid_processor_id = 1; | |||
There was a problem hiding this comment.
Maybe moving this var into the function, since it's only used inside it.
There was a problem hiding this comment.
Doing so will come with the cost of always having to run a pthread_once() in function entry.
There was a problem hiding this comment.
I'm was wrong here. Since the initialization if effectively const/constexp, there will not be any "pthread_once" overhead here. Moved the static variable inside the function in the latest commit.
src/hotspot/os/linux/os_linux.cpp
Outdated
| log_warning(os)("Invalid processor id reported by the operating system " | ||
| "(got processor id %d, valid processor id range is 0-%d)", | ||
| id, processor_count() - 1); | ||
| log_warning(os)("Falling back so assuming processor id is 0. " |
src/hotspot/os/linux/os_linux.cpp
Outdated
| // Some debuggers limit the processor count without limiting | ||
| // the returned processor ids. Fake the processor id. | ||
| return 0; | ||
| if (id >= 0 && id < processor_count()) { |
There was a problem hiding this comment.
Do we really need to check if the returned processor ID is negative? That seems a whole new level of environment screwup to me.
There was a problem hiding this comment.
I'm thinking we should make this safe to call in all cases. God knows what a broken environment might return.
There was a problem hiding this comment.
After some discussions, we agreed to not to check for negative processor ids.
There was a problem hiding this comment.
So we have to penalize all correctly functioning users because of one broken environment? Can we not detect this broken environment at startup and inject a workaround then?
Why is this an environment that is important enough that OpenJDK has to make changes to deal with a broken environment?
Cheers,
David
|
@pliden This change now passes all automated pre-integration checks. After integration, the commit message for the final commit will be: You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed. At the time when this comment was updated there had been 4 new commits pushed to the
Please see this link for an up-to-date comparison between the source branch of this pull request and the
|
|
What does the affinity mask look like at process startup? It should be possible to look at that and take the maximum CPU ID (plus 1) and It will not work with container deployments that dynamically alter affinity masks. Are there any? |
Not sure what you have in mind here? Having an indirect function call would not result in a lower overhead than the test/branch I've introduced. It's also not necessarily trivial to detect this error at startup, as you would need a reliable way to enumerate all processors (something that seems semi-broken in this environment, which is the root of the problem), bind the current thread to each of them and then check the processor id.
That's of course always judgement call/trade-off. I can't say I have a super good understanding of how common this environment it, but there's at least one "Java cloud provider" that uses this environment. |
|
It seems there have been e-mails sent that didn't show up here, so I'm answering on GitHub to hopefully re-attach the discussion to this PR. From the mailing list:
Glibc's tst-getcpu fails with some version of "getcpu results X should be Y". There seems to be a disconnect between CPU masks/affinity and what sched_getcpu() returns. Example (container with 1 CPU):
Another example (container with 2 CPUs):
It looks like CPUs are virtualized on some level, but not in sched_getcpu(). I'm guessing sched_getcpu() is returning the CPU id of the physical CPU, and not the virtual CPU, or something. So in the last example, maybe both virtual CPUs were scheduled on the same physical CPU. |
|
Mailing list message from David Holmes on hotspot-gc-dev: Hi Per, On 20/01/2021 11:16 pm, Per Liden wrote:
So it isn't that sysconf(_SC_NPROCESSORS_CONF) returns a too low number Thanks, |
(assuming you meant sched_setaffinity here...) You're seem to be right. sched_setaffinity() returns success, but a following call to sched_getaffinity() shows it had no effect.
Yep, that's what it looks like. |
It wasn't my intention to claim that sysconf() is the problem here. I just wanted to mention that it might be sysconf() that is the problem. The reason I mentioned that is the because of how Docker behaves. If you give a Docker container 2 CPUs, sysconf() will still return the number of CPUs available on the host system, e.g. 8, and sched_getcpu() will in that case return numbers in the 0-7 range. Of course, this was just an observation, Docker and OpenVZ could do things differently here.
We should be good, because libnuma will report that NUMA is not available, so we automatically disable UseNUMA if it's set. |
|
@dholmes-ora Do you still have questions or concerns here, or can I go ahead and integrate this? I've gone through all uses of sysconf(SC_NPROCESSORS*) and sched_getaffinity() we have, and they look fine. I've also looked at how the OSContainer stuff behaves in this environment, and it also looks fine. In summary, the only problem I can spot is related to sched_getcpu(). |
|
Mailing list message from David Holmes on hotspot-gc-dev: On 22/01/2021 9:21 pm, Per Liden wrote:
I remain concerned about the justification for putting in this
So IIUC what we suspect is that sched_getcpu is reporting physical id's I won't "block" this, but I'm not happy about it. Thanks, |
|
Ok, thanks all for reviewing. /integrate |
openjdk
bot
added
integrated
|
@pliden Since your change was applied there have been 16 commits pushed to the
Your commit was automatically rebased without conflicts. Pushed as commit e68eac9. |
Some environments (e.g. OpenVZ containers) incorrectly report a logical processor id that is higher than the number of processors available. This is problematic, for example, when implementing CPU-local data structures, where the processor id is used to index into an array of length processor_count().
We've received crash reports from Jelastic (a Virtuozzo/OpenVZ user) where they run into this problem. We can workaround the problem in the JVM, until the underlying problem is fixed. Without this workaround ZGC can't be used in this environment.
This is currently a ZGC-specific issue, since ZGC is currently the only part of HotSpot that is using CPU-local data structures, but that could change in the future.
Just to clarify. In a Virtuozzo/OpenZV environment, it seems the underlying problem is not necessarily that sched_getcpu() returns an incorrect processor id, but rather that sysconf(_SC_NPROCESSORS_CONF) returns a too low number. Either way, sched_getcpu() and syconf(_SC_NPROCESSORS_CONF) seems to have different views of the world. This is not an issue in container environments such as Docker.
This patch works around this problem by letting os::processor_id() on Linux detect incorrect processor ids, and convert them to processor id 0. As mentioned in the comment in the code, this is safe, but not optimal for performance if the system actually has more than one processor. There's also a warning printed the first time this happen.
Testing: Manual testing with various fake/incorrect values returned from sched_getcpu().
Progress
Issue
Reviewers
Download
$ git fetch https://git.openjdk.java.net/jdk16 pull/124/head:pull/124$ git checkout pull/124