Feature: disable blocked thread warnings until application is started #1379
Comments
|
I have the same issue when starting multiple microservices on the same node at the same time. |
|
I have the same issue and as you can see in this post from 2015 https://groups.google.com/forum/#!searchin/vertx/Karger/vertx/rbvgHUDWcsE/iQspUifkAAAJ
|
|
in some cases it might be a good thing that Vert.x performs some operation using a worker. I remember I checked for Ruby and I think something was preventing that though. |
|
OK. Can I filter those Messages somehow. Its very hard to find my own errors in that bulk of output. |
|
I have found it yeah !! Here my startup script for development : `export VERTX_HOME="$HOME/.local/bin/vertx3" export VERTX_OPTS='-Dvertx.options.maxEventLoopExecuteTime=100000000000' vertx version |
|
@erhard The thing is that proceeding this way, you will never be warned when an operation takes too much time on the event loop. |
|
@AshPlop time - verticlename id duration The problem is here blocking is not a real error it just slows down the processing and should be avoided. |
|
@erhard I don't know which logging implementation you are using but you can configure it to log everything comming from the BlockedThreadChecker in a separate file and limit the stacktrace to the number of lines you need to see if you code is at stake. Here is an example using logback. |
|
@AshPlop |
|
@AshPlop I did what you said an have now the blocking messages in a separate rotating log file. I used logback from the startup script: #!/usr/bin/env bash export VERTX_HOME="$HOME/.local/bin/vertx3" export VERTX_OPTS='-Dvertx.logger-delegate-factory-class- vertx run server_app.rb -conf dev.json Thanks for your hints |
|
Ultimately developers don't need this kind of nonsense being done for them and being in their face. You are wasting CPU cycles by doing this continuously and it so interferes with debugging as to be useless. Developers who don't understand performance and threading/locking/blocking issues are doomed to failure no matter what you try to do to hold their hand. Please just disable this stupid attempt at hand holding and try something else if you really feel you are more of an expert at software that the rest of the developers own the world are. For those looking for the simple solution, just put this in your code: or if you have a logging properties file just do |
|
I was curious what was causing it so I did Based on the output below, I sort of like this idea because I'm getting the warning message simply starting up an HTTPS server. |
|
For the most part I would think just setting Should be more than sufficient and adjust accordingly to your environment. However, you have to make sure that it is an environment timing itself. If it is something that does take a while I think I'd move it out of the event loop and push it into a worker when possible. (Enabling SSL in the worker isn't really something that I can be easily done from what I can tell). |
|
we could for this part (SSL context creation) which can be expensive disable the the check I think.
… On Sep 25, 2017, at 6:18 AM, Archimedes Trajano ***@***.***> wrote:
I was curious what was causing it so I did
final VertxOptions vertOptions = new VertxOptions();
vertOptions.setWarningExceptionTime(1);
Based on the output below, I sort of like this idea because I'm getting the warning message simply starting up an HTTPS server.
00:11:06.746 [vertx-blocked-thread-checker] WARN i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2414 ms, time limit is 2000
io.vertx.core.VertxException: Thread blocked
at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:123)
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:349)
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334)
at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:414)
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:402)
at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:295)
at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:446)
at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:466)
at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:255)
at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:222)
at net.trajano.ms.engine.JaxRsVerticle.start(JaxRsVerticle.java:42)
at io.vertx.core.AbstractVerticle.start(AbstractVerticle.java:111)
at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)
at io.vertx.core.impl.DeploymentManager$$Lambda$8/2141179775.handle(Unknown Source)
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
at io.vertx.core.impl.ContextImpl$$Lambda$9/48914743.run(Unknown Source)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:748)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#1379 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANxirSQh9sRdC8prBConBljFHLmTSH4ks5slymHgaJpZM4IHtX4>.
|
|
I think a more "proper" but less performant fix would be to have it start up in a blocking context so it is not initialized as part of the event loop. |
|
Why do you want to force the developers into discovery mode to get performant software?
If the underlying implementation is so indeterminate that you have to protect it like this, I think it's questionable in design.
Timeouts and varied latency, on a non-realtime OS, are going to happen. Demanding something (response timing) that the user has no control over and no desire to make a requirement, is a pretty large barrier for entry...
… On Sep 25, 2017, at 12:03 PM, Archimedes Trajano ***@***.***> wrote:
I think a more "proper" but less performant fix would be to have it start up in a blocking context so it is not initialized as part of the event loop.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
The issue I filed concerns the ability to change these settings at runtime, so that the extremely verbose blocked thread logging can be disabled while the application is starting, and then enabled once all verticles are started. I suggested three possible ways to accomplish this, and so far nobody has commented on pros/cons of any of those approaches (which is what I was hoping for before submitting a PR). Would any of those approaches adequately solve the other issues that are being discussed? |
|
Well for my case I put some of the larger initialization on a blocking queue and enable the route once it is fully initialized. In my case creating a Jersey application handler can take upwards of 5 seconds so I didn't like the thread block warning message since I have it set to 1.5 second (lower than the default) with stack trace. Since I don't active the route until the requisite data is ready the code won't get a null pointer exception. |
|
In Vertx.3 try this: But actually I don't like this idea. |
|
Worth remembering this issue when thinking about what BlockedThreadChecker does in the context of #1539 |
|
I just started to use AWS Fargate to deploy a vertx application and I suddenly see those warnings and exceptions as well. I have never seen those warnings at startup, when launching on an ubuntu server directly or during local development. I can also start the same docker container (that is deployed on Fargate) locally and the warnings are gone. Having a way to disable the blocked thread checker on startup would be so helpful. |
tl;dr: Following the "golden rule" during application startup and initialization may be unimportant. Therefore please provide a way to disable warnings until application is fully deployed and started.
The reason I am requesting this is because my company has a large vert.x 2 ruby application (ported from rails/sinatra) that is being upgraded to vert.x 3. We do a lot of pre-loading during application startup, and it would be difficult to make all of that asynchronous or wrapped with execute_blocking. We're getting the multiple warnings and large stacktrace during application startup.
But during application startup, I don't mind blocking the event loop, because there's no other work the application could be doing that's being blocked. Once verticles are started and accepting requests, I do want to see the warnings.
Therefore, I'd like to have some mechanism to disable event loop warnings/stacktraces until the application has been fully deployed.
A few ideas for how this could be done:
disableBlockedThreadWarningsandenableBlockedThreadWarningsmethods toVertx. Those would call new setter onBlockedThreadCheckerto set value on afinal AtomicBoolean. Within the timer taskrun()method, check value before logging. I like this solution the best.ready()(for example) method toVertx, along with a new option to disable warnings untilready()is called. Less flexible, but same effect during application startup.blockedThreadCheckInterval,maxEventLoopExecuteTime,maxWorkerExecuteTime,warningExceptionTime) into their own class, and provide a static instance with defaults. Provide a way to replace the instance at runtime.The text was updated successfully, but these errors were encountered: