8269865: Async UL needs to handle ERANGE on exceeding SEM_VALUE_MAX #216
Conversation
|
Webrevs
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Xin,
I wasn't expecting to see an attempt to balance the semaphore count on each write, but this seems to be okay.
One simplification suggested below.
Thanks,
David
write(); | ||
|
||
int n = write(); | ||
for (int i = n - 1; i > 0; --i) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
while (--n > 0) { // pre-decrement as we already performed the first wait
@navyxliu This change now passes all automated pre-integration checks. After integration, the commit message for the final commit will be:
You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed. At the time when this comment was updated there had been 45 new commits pushed to the
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I have to withdraw my initial review approval.
This certainly helps allow the writer thread to catch up sooner, but there is still no guarantee that it will keep up with the loggers sufficient to avoid the overflow.
What we have to ensure in addition is that the number of outstanding log messages permitted by the buffer size is always less than the max value allow by the semaphore. Unfortunately we don't know what that value is.
David
On Linux, SEM_VALUE_MAX is defined in <semaphore.h>. we can add an assertion that
Yes, you just unveil another issue. I found _buffer_max_size is 91k if I use
|
I don't like the idea of trying to limit the log size based on a platform specific private implementation detail - and you'd have to check that against any user supplied buffer size and reject a too large buffer. The other option is revert these changes and expose a way to ignore the ERANGE error in Semaphore. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been thinking about how to handle the ERANGE/EOVERFLOW in the Semaphore API. A default arg is one way. I was also considering a specific signal_xx method that allows overflow; or report to the caller that overflow happened. Not sure what might be best. What you have might be simplest for 17 but there are still some issues that need further investigation. See comments below. I think we need other opinions here too.
I'm surprised macOS has no overflow checks, it looks like its counter will just wrap when it overflows - which would break things, but of course is very unlikely in real use.
Thanks,
David
@@ -39,7 +39,7 @@ WindowsSemaphore::~WindowsSemaphore() { | |||
::CloseHandle(_semaphore); | |||
} | |||
|
|||
void WindowsSemaphore::signal(uint count) { | |||
void WindowsSemaphore::signal(uint count, bool ignore_overflow) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the Windows ReleaseSemaphore can also fail with the equivalent of EOVERFLOW if the count would exceed the maximum allowed. We either need to dig deeper into what value of GetlastError we would see, or else ignore all errors if ignore_overflow is set. I think I can tweak a gtest to see what happens.
// The only difference between this and Semaphore is that | ||
// it ignores errno ERANGE and EOVERFLOW. | ||
class AsyncLogSemaphore : public CHeapObj<mtSynchronizer> { | ||
SemaphoreImpl _impl; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is necessary, just call wait(1, true) where needed.
I ran a test on Linux, Windows and OSX. As expected Linux gives EOVERFLOW. OSX the test just times out without error. On Windows we get error 298: ERROR_TOO_MANY_POSTS so I think we should check for that and ignore it if ignoring overflow. Thanks, |
got it. I will cover Windows. |
Unlike Posix and Win32, it's not easy for me to find any document about Darwin semaphore API. I guess this is the equivalence of glibc on MacOS. check out if count has overflown, it should return kr from |
This patch handles overflow scenerios for Posix and Windows. MacOS platform doesn't have any error so we ignore it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Figuring out the best way to handle this is proving to be quite tricky - sorry. I need to think some more about it. Unfortunately I'm away for a few days after today.
David
@@ -50,7 +50,12 @@ class Semaphore : public CHeapObj<mtSynchronizer> { | |||
Semaphore(uint value = 0) : _impl(value) {} | |||
~Semaphore() {} | |||
|
|||
void signal(uint count = 1) { _impl.signal(count); } | |||
void signal(uint count = 1) { _impl.signal(count, false /* ignore_overflow */); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please change comment to:
/* don't ignore overflow */
as that describes what the false
argument actually means, rather than just showing what parameter it is providing the value for.
// Ignore error of overflow | ||
void signal_overflow(uint count = 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit unsure about how to name and describe this given that the overflow check only relates to debug builds - in a product build both signal
and signal_overflow
are identical.
I had missed previously that the ignore_overflow
parameter was only in the impl API not the main public API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks you for reviewing this change.
The reason we only had ignore_overflow in impl API because I defined my own platform-independent Semaphore class, aka. AsyncLogSemaphore in previous revision. I changed type of _sem from Sempahore to AsyncLogSemaphore. I "overloaded" signal(uint count) in that way so I don't pollute "public API".
I don't think this is necessary, just call wait(1, true) where needed.
I think your opinion is that it's not necessary. how about I just add a second parameter ignore_overflow to Semaphore::signal with the default value = false?
Hi, @tstuefe ,
If it works out, the one open issue is what we should do on MacOS. My option is that we just leave it alone since |
Sorry for dropping out, I'm snowed in. I put the patch into our AIX CI. Due to the slowness of our AIX hardware, this may take a day or two. |
Thanks! Let's see the result. Meanwhile, allow me refactor this patch and spend more time semaphore of Darwin-XNU |
Mailing list message from David Holmes on hotspot-runtime-dev: On 7/07/2021 3:26 pm, Xin Liu wrote:
You mean if count++ wraps around to become negative then the next call In any case due diligence has been done here. There is no documented David |
Mailing list message from Liu, Xin on hotspot-runtime-dev: I had a gtest. I think it can trigger overflow issue on different TEST(Semaphore, signal_overflow) { On 7/6/21 11:58 PM, David Holmes wrote:
|
Mailing list message from David Holmes on hotspot-runtime-dev: On 7/07/2021 3:26 pm, Xin Liu wrote:
You mean if count++ wraps around to become negative then the next call In any case due diligence has been done here. There is no documented David |
Mailing list message from Liu, Xin on hotspot-runtime-dev: I had a gtest. I think it can trigger overflow issue on different TEST(Semaphore, signal_overflow) { On 7/6/21 11:58 PM, David Holmes wrote:
|
I am not sure about that. The current code derived from there. HotSpot Monitor needs Thread::current(), but the logging event spans from the very beginning to the very end of JVM. We have to circumvent those cases. On the other hand, I carefully read the implementation of semaphore_signal on Darwin. I am pretty sure it returns KERN_SUCCESS when overflow happens. I added a line of comment in the last revision. Could you take a look at that? |
Mailing list message from David Holmes on hotspot-runtime-dev: On 9/07/2021 11:33 am, Xin Liu wrote:
But we already circumvent that for async logging. We can't use async
The issue is not what it returns but how it actually behaves. David |
1 similar comment
Mailing list message from David Holmes on hotspot-runtime-dev: On 9/07/2021 11:33 am, Xin Liu wrote:
But we already circumvent that for async logging. We can't use async
The issue is not what it returns but how it actually behaves. David |
That is not true. I deleted finalize() because it was complex and not necessary after we switch from Monitor to Semaphore. I reviewed this line. https://github.com/apple/darwin-xnu/blob/main/osfmk/kern/sync_sema.c#L400 thanks, |
Mailing list message from David Holmes on hotspot-runtime-dev: On 13/07/2021 11:41 am, Xin Liu wrote:
Ah - sorry. I was looking at the VM initialization issue not the thread
Sorry but there is no way I'm going to trust any ad-hoc guesses about I will continue to think this about this problem, but I'm not seeing any Thanks, |
Mailing list message from David Holmes on hotspot-runtime-dev: On 13/07/2021 1:03 pm, David Holmes wrote:
As @pchilano has pointed out to me we can use the lower-level Instead of having a binary semaphore to control access to the _buffer, void AsyncLogWriter::write() { { // critical region _buffer.pop_all(&logs); void AsyncLogWriter::run() { void AsyncLogWriter::enqueue_locked(const AsyncLogMessage& msg) { David
|
Aha, os::PlatformMonitor is just the cross-platform condition variable, no strings attached! We still have 2 week before phase-2 rampdown. Let me also take a look how JFR handle this case. It is said JFR also has an asynchronous buffer too. |
Mailing list message from David Holmes on hotspot-runtime-dev: On 13/07/2021 4:10 pm, Xin Liu wrote:
JDK 17 will enter Rampdown Phase Two this Thursday, 15 July, Thanks, |
1 similar comment
Mailing list message from David Holmes on hotspot-runtime-dev: On 13/07/2021 4:10 pm, Xin Liu wrote:
JDK 17 will enter Rampdown Phase Two this Thursday, 15 July, Thanks, |
okay. copy that. I will send out a new revision soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry this isn't right at all.
David
// _lock(1) denotes a critional region. | ||
Semaphore _lock; | ||
// _sem is a semaphore whose value denotes how many messages have been enqueued. | ||
// It decreases in AsyncLogWriter::run() | ||
Semaphore _sem; | ||
Semaphore _flush_sem; | ||
|
||
os::PlatformMonitor _cv; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No this isn't what you need. The _lock is the PlatformMonitor. You lock/unlock in AsyncLogLocker for the critical regions, and then use the wait/notify operations for the coordination.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you mean I should use os::PlatformMonitor _lock as a combination of mutux and cv, right?
|
||
while (!_data_available) { | ||
_lock.signal(); | ||
_cv.wait(0/* no timeout */); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can't call wait() on an unlocked monitor - that's not how monitors are used.
Second attempt. This patch also removed semaphore-based lock because PlatformMonitor can also be used as a Mutex lock.
hi, @dholmes-ora @pchilano, Thank you for letting me know os::PlatformMonitor. I used to think Monitor is a higher-level synchronization construct than Semaphore. it turns out that os::PlatformMonitor is as low-level as semaphore. Lesson learned. Could you take a look at this change and see if I use it correctly? thanks, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Xin,
This looks good. I'll run it through our testing.
I'll also try to get @pchilano to review it.
Thanks,
David
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Xin,
Looks good to me too.
The only small suggestion is that you could do the pop_all() and _stats.iterate when coming out of wait() since you are already holding the monitor, and then pass the logs to write(). Either way is fine though.
Thanks,
Patricio
Thanks for reviewing. I see what you mean. I take a note on that. yes, we may gain something if we merge two critical areas. |
Hi, Reviewers, Thanks a lot. I append a comment to explain why we choose PlatformMonitor over Semaphore. Meanwhile, I defer performance-wise change. Please let me know the results of test. I hope we can make it to jdk17 safely before the window is close. |
Mailing list message from David Holmes on hotspot-runtime-dev: On 14/07/2021 11:32 am, Xin Liu wrote:
I think it is cleaner as-is. Testing tiers 1-3 passed with no problems, so I think this can be Thanks, |
1 similar comment
Mailing list message from David Holmes on hotspot-runtime-dev: On 14/07/2021 11:32 am, Xin Liu wrote:
I think it is cleaner as-is. Testing tiers 1-3 passed with no problems, so I think this can be Thanks, |
/integrate |
Going to push as commit 67273ae.
Your commit was automatically rebased without conflicts. |
This patch solved the sempahore overflow issue with errno ERANGE or EOVERFLOW.
Previously, we have asymmetric p/v operations for semaphore _sem. Each iteration
only decrements _sem 1 but dequeues N messages. If logging threads keep preempting
async logging thread, it may cause the value of _sem accumulates until overflow!
We decide to ditch counting semaphore. os::PlatformMonitor can be used as low-level
combination of Mutex/CondVar. It may work if we simply ignore Semaphore::signal failure
due to value overflow, but we have to pollute Semaphore interface. Also it seems that MacOS
semaphore which is non-posix doesn't fail in this case.
Progress
Issue
Reviewers
Reviewing
Using
git
Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk17 pull/216/head:pull/216
$ git checkout pull/216
Update a local copy of the PR:
$ git checkout pull/216
$ git pull https://git.openjdk.java.net/jdk17 pull/216/head
Using Skara CLI tools
Checkout this PR locally:
$ git pr checkout 216
View PR using the GUI difftool:
$ git pr show -t 216
Using diff file
Download this PR as a diff file:
https://git.openjdk.java.net/jdk17/pull/216.diff