Intermittent Registry failure to perform Health check due to missing temporary directory (Cannot load environment)

[ikomangmahendra]

Overview of the issue

Intermittent Registry failure to perform Health check due to missing temporary directory (Cannot load environment).

I try to investigate this issue with no luck because this issue is intermittent and difficult to predict when it will occur.
Sometimes it happens in three days, one week and sometimes two week.

I have implemented a temporary solution to prevent this issue from happening during business hours, I restart the registry server every midnight with cron job scheduler.

2022-11-07 00:55:28.380 INFO 8589 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration 2022-11-07 00:59:21.640 WARN 8589 --- [DiscoveryClient-InstanceInfoReplicator-0] .c.s.e.MultipleJGitEnvironmentRepository : Could not fetch remote for main remote: https://bitbucket.org/xtramiledev/thread-scc 2022-11-07 01:00:28.381 INFO 8589 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration 2022-11-07 03:05:28.399 INFO 8589 --- [AsyncResolver-bootstrap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver : Resolving eureka endpoints via configuration 2022-11-07 03:05:37.701 WARN 8589 --- [DiscoveryClient-InstanceInfoReplicator-0] o.s.c.c.s.c.ConfigServerHealthIndicator : Health check failed java.lang.IllegalStateException: Cannot load environment at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:323) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:262) at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.getLocations(MultipleJGitEnvironmentRepository.java:139) at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55) at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOneFromCandidate(MultipleJGitEnvironmentRepository.java:188) at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:178) at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:64) at org.springframework.cloud.config.server.config.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:72) at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82) at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.processContributor(EurekaHealthCheckHandler.java:163) at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:148) at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:141) at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:130) at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1440) at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /tmp/config-repo-12194334409392335656 at org.eclipse.jgit.lib.BaseRepositoryBuilder.build(BaseRepositoryBuilder.java:627) at org.eclipse.jgit.api.Git.open(Git.java:93) at org.eclipse.jgit.api.Git.open(Git.java:73) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository$JGitFactory.getGitByOpen(JGitEnvironmentRepository.java:782) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.openGitRepository(JGitEnvironmentRepository.java:638) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.createGitClient(JGitEnvironmentRepository.java:613) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:296) ... 20 common frames omitted 2022-11-07 03:05:37.707 INFO 8589 --- [DiscoveryClient-InstanceInfoReplicator-0] com.netflix.discovery.DiscoveryClient : Saw local status change event StatusChangeEvent [timestamp=1667750737707, current=DOWN, previous=UP] 2022-11-07 03:05:37.707 INFO 8589 --- [DiscoveryClient-InstanceInfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_JHIPSTER-REGISTRY/jhipsterRegistry:2885c7c3dc2ba92e131b64ce78745905: registering service... 2022-11-07 03:05:37.807 INFO 8589 --- [DiscoveryClient-InstanceInfoReplicator-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_JHIPSTER-REGISTRY/jhipsterRegistry:2885c7c3dc2ba92e131b64ce78745905 - registration status: 204

Motivation for or Use Case

Other microservices functionality won't work such as:

• microservice can't talk to each others
• integration to AWS S3 is broken

Reproduce the error

This is the difficult part to figure it out due to intermittent.

JHipster Registry Version(s)

All the jar files are downloaded directly from this git repository
7.4.0 and 7.3.0

This issue does not occur on version 6, I currently have a production application that has been running for over a year without restarting the registry

Operating System and JDK

AWS Linux AMI

openjdk 11.0.13 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)

Here is the script to run the registry:
java -jar jhipster-registry-7.4.0.jar \ --spring.profiles.active=prod \ --jhipster.security.authentication.jwt.secret={this is should be base64} \ --spring.cloud.config.server.composite.0.type=git \ --spring.cloud.config.server.composite.0.uri=https://bitbucket.org/{project-spring-cloud} \ --spring.cloud.config.server.composite.0.username={username} \ --spring.cloud.config.server.composite.0.password={password} \ --spring.cloud.config.profile=prod \ --spring.cloud.config.label=master

Note: statement inside curly brackets are replace for security consideration

• Checking this box is mandatory (this is just to show you read everything)