Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs #1055

codependent · 2016-05-27T09:36:09Z

My microservices' logs show periodically, can't tell when, just at a certain moment the following exception:

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a98c42 rejected from java.util.concurrent.ThreadPoolExecutor@96e72c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) ~[na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[na:1.8.0_74]
    at com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:62) ~[eureka-client-1.4.6.jar:1.4.6]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_74]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_74]

My infraestructure has two Eureka peer aware servers, with the following configuration:


---
spring:
  profiles: archimedes1

server:
  port: ${PORT:8761}

eureka:
  instance:
    hostname: archimedes1
  client:
    serviceUrl:
      defaultZone: http://archimedes2:8762/eureka/

---
spring:
  profiles: archimedes2

server:
  port: ${PORT:8762}

eureka:
  instance:
    hostname: archimedes2
  client:
    serviceUrl:
      defaultZone: http://archimedes1:8761/eureka/

All my services, at some moment throw that exception. This is and example of one of them:


---
spring:
  application:
    name: user-api
  profiles: user-api1
  cloud:
    config:
      discovery:
        enabled: true
      failFast: true
      retry:
        maxAttempts: 100
        initialInterval: 2000
        maxInterval: 3000
        multiplier: 1.5

server:
  port: 9977

eureka:
  client:
    serviceUrl:
      defaultZone: http://archimedes1:8761/eureka/,http://archimedes2:8762/eureka/
    healthcheck:
      enabled: true

Despite the exception everything keeps running OK, but it's kind of annoying to get it. If you think this should be a Netflix's Eureka issue let me know and I'll close this one.

The text was updated successfully, but these errors were encountered:

daniellavoie · 2016-05-28T12:06:58Z

I built a setup with 2 peer aware eureka servers and one eureka client with the configuration you provided. I am not reproducing your problem. Can you upload to GH a code snippet reproducing this problem ?

daniellavoie · 2016-05-28T12:31:43Z

I am also concerned by your stacktrace.

ThreadPoolExecutor@96e72c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

It feels to me that you have overwritten eureka thread pool configuration. With this setup, any task will fail since your pool size are defined to 0 according to your stack trace.

codependent · 2016-05-30T06:55:46Z

@daniellavoie I'll prepare a sample project along the day.

codependent · 2016-05-31T07:21:12Z

I have finally prepared the sample projects: https://github.com/codependent/scn-brixton-release.git

Actually it has nothing to do with using peer Eureka servers, the miminum scenario needed is one Eureka Server, one config server and one microservice (Eureka client that uses the config server).

When I remove the config server the error doesn't show up.

To reproduce just start the Eureka server (archimedes), when it's up, start the Config server, and when everything is up just launch the user-api server. You'll see the usual:

2016-05-31 09:14:36.680   INFO 12192 --- [           main] c.codependent.sample.UserApiApplication  : Started UserApiApplication in 8.118 seconds (JVM running for 8.732)

Then after 5 minutes the error shows up in the log:

2016-05-31 09:19:31.085 ERROR 12192 --- [ver-bootstrap-0] c.netflix.discovery.TimedSupervisorTask  : task supervisor rejected the task

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@19e8a47a rejected from java.util.concurrent.ThreadPoolExecutor@172f5937[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) ~[na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[na:1.8.0_74]
    at com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:62) ~[eureka-client-1.4.6.jar:1.4.6]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_74]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_74]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_74]

2016-05-31 09:19:36.365  INFO 12192 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration

royremi · 2016-05-31T15:15:26Z

Maybe i'm wrong, but i've seen this error when you have the devtools on the classpath.

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
        </dependency>

when I recompile in intellij a lot of my apps show this error and I have to restart them manually. Idk if it's can be you're case!

codependent · 2016-05-31T16:01:57Z

@royremi I just checked the dependencies. I don't have it, not even as a transitive dependency.

daniellavoie · 2016-06-01T02:04:28Z

Great, got to reproduce it. I'll debug to see what is wrong with that executor.

daniellavoie · 2016-06-01T19:12:37Z

I won't have too much time for the rest of the week but here is what I found for now.

There is two discovery client running. Spring Cloud AutoConfiguration sends a termination signals to the EurekaClient located in the parent spring context. This signal terminates the executor service who receives a new task from TimedSupervisorTask. Since the EurekaClient was previously shutdown, the tasks fails and generate the stack trace. I haven't figured out why the TimedSupervisorTask is generated for that terminated client. I'm not sure if it is normal to have 2 discovery clients running. One comes from the bootstrap mechanism.

asarkar · 2016-06-02T02:07:03Z

I was on Brixton.M3 and there were exceptions reported as in Netflix/eureka#638. Now I upgraded but have a different exception reported in this issue. Given the simple reproduction of the latest exception, I'm wondering how did any test app manage to not throw this exception during the GA release of Brixton. What does the Spring Cloud team do differently (for integration testing) that we may be missing?

cs94njw · 2016-07-25T15:42:21Z

I accidentally had this in my Microservice's .yml file:

eureka:
  instance:
     prefer-ip-address: true
     leaseRenewalIntervalInSeconds: 30   # DO NOT DO THIS IN PRODUCTION

After reading this bug report, I wondered if perhaps "eureka->instance" variables shouldn't exist in a Non-Eureka project. So I moved that configuration to my Eureka project, and it seems to be happy.

EDIT: I take that back. The error still occurs for me.

trendsoa · 2016-08-01T02:58:10Z

same error when I change my service to use spring cloud config discovery in bootstrap.yml

spring:
  application:
    name: store-service
  cloud:
    config:
      discovery:
        enabled: true
        service-id: CONFIG-SERVER

in my service configuration, I also use EnableDiscoveryClient annotation

looks like some of the logs are duplicate

2016-08-01 10:50:25.578  INFO 10392 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8086 (http)
2016-08-01 10:50:25.578  INFO 10392 --- [           main] c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8086
2016-08-01 10:50:25.594  INFO 10392 --- [           main] c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8086

tjuchniewicz · 2016-08-29T12:24:22Z

My "solution" for this error:

logging:
  level:
    com.netflix.discovery.TimedSupervisorTask: "OFF"

asarkar · 2016-08-29T15:24:48Z

You call that a solution? That's like doctor telling a patient not to worry because the MRI machine isn't working.

spencergibb · 2016-08-29T15:36:11Z

@abhijitsarkar please be aware of your tone. That last comment was not necessary. So far, besides the error showing in the log, no one has demonstrated that it is negatively affecting their applications.

asarkar · 2016-08-29T19:05:23Z

My tone was humorous with a hint of sarcasm. Besides, this is an open forum where people are discussing issues about a software. If you expect everyone to always speak in a manner that's pleasing to you, consider moving to a closed forum like Spring JIRA with only developer access.

spencergibb · 2016-08-29T19:08:39Z

Tone and sarcasm don't always come across really well in short text. I wasn't the only one who reacted like I did. We have a Code of Conduct we expect folks to abide by.

spencergibb · 2016-08-29T19:11:14Z

Pleasing to me vs. respectful to all is very different.

asarkar · 2016-08-29T19:28:42Z

My analogy to the doctor in response to a suggestion of suppressing error logs was disrespectful? And violates the code of conduct? Sorry, I just don't get it.

The last relevant comment from a committer was 3 months ago saying that they won't have time to look into this issue. And from your comment above, it seems like you're waiting for the users to prove that they are negatively impacted before you do anything about it. Of course, this is a community driven open source but don't you think ludicrous log suppression isn't the answer?

lferna · 2017-06-19T11:47:26Z

How to reproduce the error:

With this Eureka configuration:

eureka.instance.hostname=localhost
eureka.instance.port=8761

spring.application.name=eureka
spring.cloud.config.enabled=false

server.port=${eureka.instance.port}

eureka.server.enableSelfPreservation=false

eureka.client.registerWithEureka=false
eureka.client.fetchRegistry=false
eureka.client.serviceUrl.defaultZone=http://${eureka.instance.hostname}:${eureka.instance.port}/eureka/

This POM configuration;

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-parent</artifactId>
        **<version>Camden.SR4</version>**
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
 
   <dependencies>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-eureka-server</artifactId>
    </dependency>
  </dependencies>

This configuration client:

spring.cloud.config.enabled=true
spring.application.name=<name>
#Uncomment service discovery or URI
spring.cloud.config.discovery.enabled=true
spring.cloud.config.uri=<URI>

The error:

2017-06-19 13:40:57.841 ERROR [mscustomer,,,] 15516 --- [ver-bootstrap-0] c.netflix.discovery.TimedSupervisorTask  : task supervisor rejected the task

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@652f1860 rejected from java.util.concurrent.ThreadPoolExecutor@6be8eab5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[na:1.8.0_101]

Versions:

Spring Boot version: 1.4.0
Spring Cloud Netflix: Camdem.SR4
Spring Cloud Config: 1.2.3.RELEASE (**the error only reproduces with Spring Cloud Config**. If I use local configuration, the error doesn't show)

Upgrade the Netflix version, shows the log in warn level, is that the solution for this problem?

Thanks.

dsyer · 2017-06-19T11:50:29Z

Thanks. Please create a sample project and link to it. And please format your code snippets. How did you upgrade the netflix version? What was the outcome?

dsyer · 2017-06-19T11:58:33Z

FWIW I updated the app in the @codependent sample above to use Dalston.SR1 and I don't see any exceptions logged at all (after running for about 10min). 5min seems to be the pattern with the original sample (Brixton.RELEASE). I propose we close this as fixed.

lferna · 2017-06-19T17:41:03Z

Hi, thanks for your reply. I've tested the @codependent and with the original dependencies (Brixton) the error shows after 5 minutes. With Camdem.SR4 the error shows again. And with Dalston.SR1 the error shows again.

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6be81451 rejected from java.util.concurrent.ThreadPoolExecutor@652fe75c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

I've updated this dependencies:

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		  <version>1.4.0.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>Dalston.SR1</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

spencergibb · 2017-06-19T17:58:23Z

Dalston is not compatible with boot 1.4.x, btw.

lferna · 2017-06-19T18:08:47Z

You're right, but with the 1.5.x the error continues :(

dsyer · 2017-06-19T19:09:13Z

Not for me. What log level is it coming out at? What did you do to see that log? You didn't even paste the whole stack trace BTW.

codependent · 2017-06-19T22:46:12Z

I just updated the sample proyect to work with Spring Boot 1.5.4.RELEASE and Spring Cloud Dalston.SR1. With this setup the exception no longer shows up.

Thank you for having a look at it.

lferna · 2017-06-20T06:26:15Z

Hi @codependent , with Spring Boot 1.5.4.RELEASE the exception dissapear, thanks.

mradulpandey · 2017-08-25T04:52:31Z

I observed if i set eureka.client.enabled=true in my config server and
hit http://localhost:8888/monitor?path=application
i get the following response

{
    "timestamp": 1503601813361,
    "status": 500,
    "error": "Internal Server Error",
    "exception": "java.util.concurrent.RejectedExecutionException",
    "message": "Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7f865c88 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@20ac99dd[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]",
    "path": "/monitor"
}

Versions:

   spring.boot.version --> 1.5.6.RELEASE
   spring.cloud.version --> Dalston.SR1, Dalston.SR2 and Dalston.SR3

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-config-server</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-config-monitor</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-eureka</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-actuator-docs</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-rsa</artifactId>
</dependency>

<dependency>
    <groupId>org.jolokia</groupId>
    <artifactId>jolokia-core</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

dsyer · 2017-08-25T05:50:00Z

@PANDEYmradul this issue is closed and yours is unrelated. Please open another ticket.

mradulpandey · 2017-08-25T09:38:43Z

@dsyer i created issue #2228

daniellavoie mentioned this issue May 29, 2016

RejectedExecutionException logged on console #1060

Closed

codependent changed the title ~~Microservices registered on peer aware Eureka instances launching java.util.concurrent.RejectedExecutionException periodically~~ Microservices launching java.util.concurrent.RejectedExecutionException May 31, 2016

spring-cloud locked and limited conversation to collaborators Aug 29, 2016

spring-cloud unlocked this conversation Jun 19, 2017

dsyer changed the title ~~Microservices launching java.util.concurrent.RejectedExecutionException~~ Boot strap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs Jun 19, 2017

dsyer changed the title ~~Boot strap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs~~ Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs Jun 19, 2017

codependent closed this as completed Jun 19, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs #1055

Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs #1055

codependent commented May 27, 2016

daniellavoie commented May 28, 2016

daniellavoie commented May 28, 2016

codependent commented May 30, 2016

codependent commented May 31, 2016

royremi commented May 31, 2016

codependent commented May 31, 2016

daniellavoie commented Jun 1, 2016

daniellavoie commented Jun 1, 2016

asarkar commented Jun 2, 2016 •

edited

Loading

cs94njw commented Jul 25, 2016 •

edited by spencergibb

Loading

trendsoa commented Aug 1, 2016 •

edited by spencergibb

Loading

tjuchniewicz commented Aug 29, 2016

asarkar commented Aug 29, 2016

spencergibb commented Aug 29, 2016

asarkar commented Aug 29, 2016

spencergibb commented Aug 29, 2016 •

edited

Loading

spencergibb commented Aug 29, 2016

asarkar commented Aug 29, 2016

lferna commented Jun 19, 2017 •

edited

Loading

dsyer commented Jun 19, 2017 •

edited

Loading

dsyer commented Jun 19, 2017

lferna commented Jun 19, 2017

spencergibb commented Jun 19, 2017

lferna commented Jun 19, 2017

dsyer commented Jun 19, 2017

codependent commented Jun 19, 2017 •

edited

Loading

lferna commented Jun 20, 2017

mradulpandey commented Aug 25, 2017 •

edited

Loading

dsyer commented Aug 25, 2017

mradulpandey commented Aug 25, 2017

Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs #1055

Bootstrap discovery (locating config server via discovery) leads to harmless RejectedExecutionException in logs #1055

Comments

codependent commented May 27, 2016

daniellavoie commented May 28, 2016

daniellavoie commented May 28, 2016

codependent commented May 30, 2016

codependent commented May 31, 2016

royremi commented May 31, 2016

codependent commented May 31, 2016

daniellavoie commented Jun 1, 2016

daniellavoie commented Jun 1, 2016

asarkar commented Jun 2, 2016 • edited Loading

cs94njw commented Jul 25, 2016 • edited by spencergibb Loading

trendsoa commented Aug 1, 2016 • edited by spencergibb Loading

tjuchniewicz commented Aug 29, 2016

asarkar commented Aug 29, 2016

spencergibb commented Aug 29, 2016

asarkar commented Aug 29, 2016

spencergibb commented Aug 29, 2016 • edited Loading

spencergibb commented Aug 29, 2016

asarkar commented Aug 29, 2016

lferna commented Jun 19, 2017 • edited Loading

dsyer commented Jun 19, 2017 • edited Loading

dsyer commented Jun 19, 2017

lferna commented Jun 19, 2017

spencergibb commented Jun 19, 2017

lferna commented Jun 19, 2017

dsyer commented Jun 19, 2017

codependent commented Jun 19, 2017 • edited Loading

lferna commented Jun 20, 2017

mradulpandey commented Aug 25, 2017 • edited Loading

dsyer commented Aug 25, 2017

mradulpandey commented Aug 25, 2017

asarkar commented Jun 2, 2016 •

edited

Loading

cs94njw commented Jul 25, 2016 •

edited by spencergibb

Loading

trendsoa commented Aug 1, 2016 •

edited by spencergibb

Loading

spencergibb commented Aug 29, 2016 •

edited

Loading

lferna commented Jun 19, 2017 •

edited

Loading

dsyer commented Jun 19, 2017 •

edited

Loading

codependent commented Jun 19, 2017 •

edited

Loading

mradulpandey commented Aug 25, 2017 •

edited

Loading