Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#4717 Solve high CPU spikes happening on Mac with JDK 1.8 #7950

Merged
merged 2 commits into from
May 11, 2022

Conversation

lorban
Copy link
Contributor

@lorban lorban commented May 3, 2022

Explicitly cancel the selection key before closing the channel to work around what seems to be a bug in MacOS / JDK 1.8.

@lorban lorban self-assigned this May 3, 2022
@lorban lorban added the Bug For general bugs on Jetty side label May 3, 2022
@lorban lorban linked an issue May 3, 2022 that may be closed by this pull request
@lorban lorban requested a review from sbordet May 3, 2022 14:57
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
@lorban lorban force-pushed the jetty-9.4.x-4717-macos-high-cpu branch from 7d63332 to f8ce8ea Compare May 3, 2022 15:00
@joakime
Copy link
Contributor

joakime commented May 3, 2022

Good stuff.

Do we need this same logic anywhere else in our code?

@@ -207,6 +207,8 @@ public void doClose()
LOG.debug("doClose {}", this);
try
{
if (_key != null)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_key is never null, so the check seems redundant, and is now also racy with replaceKey().
I would remove the null check and perhaps catch (Throwable) below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SocketChannelEndPointOpenCloseTest fails without this check, but it passes a null SelectionKey to the ChannelEndPoint constructor.

I thought adding that null check was the safest thing to do, but you're right, I overlooked the replaceKey() mechanism which makes this null check odd.

I've changed the test so that it passes a non-null SelectionKey but I wonder if we shouldn't be stricter in the constructor about accepting null arguments when that should not be allowed.

…t pass a null key to the ctor

Signed-off-by: Ludovic Orban <lorban@bitronix.be>
@lorban
Copy link
Contributor Author

lorban commented May 4, 2022

@joakime I checked that together with @sbordet and no, there aren't any other places where that should be done as ChannelEndPoint is the central place where that is abstracted away.

I also do not think we should merge this change to other branches as the problem only occurs on MacOS with JDK 1.8.

}
};
c.client = new SocketChannelEndPoint(SocketChannel.open(connector.socket().getLocalSocketAddress()), null, k, null);
c.server = new SocketChannelEndPoint(connector.accept(), null, k, null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, no! 😞
Aren't unit/mock/whatever tests great? Not!

Would it be possible to rewrite this test to something less hacky that actually tests real code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure of what you mean. This test is checking the open/shutdown state of SocketChannelEndPoint, which happened to not need the selection key but now does. Why would we want to change the scope of this test because we changed an implementation detail?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that creating "fake" SocketChannelEndPoints that lack the otherwise required ManagedSelector, the SelectionKey, etc. is basically just testing test code, not the real code path that will happen with real code.

It would be better to actually test with a normally connected pair of EndPoints, using ServerConnector and ClientConnector, so we would not need to "fake" the SelectionKey and half-create SocketChannelEndPoints without required components.

@joakime
Copy link
Contributor

joakime commented May 4, 2022

I also do not think we should merge this change to other branches as the problem only occurs on MacOS with JDK 1.8.

I wonder if the Windows users at #1446 and #2205 could benefit from this as well?

@lorban
Copy link
Contributor Author

lorban commented May 5, 2022

#2205 looks strangely similar, so it might benefit from this change too.

I'm still a bit on the fence about merging this fix or not. In my understanding this is clearly a JVM bug, but the fact that it's been observed on two different OS is making me question this conclusion a bit. But the problem has only ever been reported on JDK 1.8 and one person explicitly stated that it cannot be reproduced on newer JDKs so I'm still slightly in favor of not merging the fix forward.

@joakime WDYT?

Copy link
Contributor

@joakime joakime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy that this fix even exists.

And my reading of it is that even this extra key.cancel() is ultimately harmless on newer JVMs.
So why not keep it in for all JVMs in Jetty 9.4.x?

I also see the argument for not bothering merging this forward to Jetty 10+ as none of those have the ability to even run on a buggy JVM.

@lorban lorban requested a review from sbordet May 9, 2022 08:34
}
};
c.client = new SocketChannelEndPoint(SocketChannel.open(connector.socket().getLocalSocketAddress()), null, k, null);
c.server = new SocketChannelEndPoint(connector.accept(), null, k, null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that creating "fake" SocketChannelEndPoints that lack the otherwise required ManagedSelector, the SelectionKey, etc. is basically just testing test code, not the real code path that will happen with real code.

It would be better to actually test with a normally connected pair of EndPoints, using ServerConnector and ClientConnector, so we would not need to "fake" the SelectionKey and half-create SocketChannelEndPoints without required components.

@lorban lorban merged commit 0fc317b into jetty-9.4.x May 11, 2022
@lorban lorban deleted the jetty-9.4.x-4717-macos-high-cpu branch May 11, 2022 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For general bugs on Jetty side
Projects
None yet
Development

Successfully merging this pull request may close these issues.

High CPU spikes with jetty winstone threads
3 participants