Docker-compose deploy on RHEL8 stops working after 24 minutes.

[mondeelr]

For a POC i first deployed your "local installation" on my laptor on Windows 10. This works and stays up.
For a workshop i deployed the same configuration on a Redhat Enterprise Linux server for internal use. The configuration at the end starts up and i can log in.

But after about 24 minutes the application becomes unresponsive. In the logging i get :

Startup

fdp_1 | 2020-10-08 20:18:39,374 4174 [main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Bootstrapping Spring Data MongoDB repositories in DEFAULT mode.
fdp_1 | 2020-10-08 20:18:39,506 4306 [main] INFO org.springframework.data.repository.config.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 122ms. Found 8 MongoDB repository interfaces.
fdp_1 | 2020-10-08 20:18:40,533 5333 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'spring.data.mongodb-org.springframework.boot.autoconfigure.mongo.MongoProperties' of type [org.springframework.boot.autoconfigure.mongo.MongoProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:40,540 5340 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.boot.autoconfigure.data.mongo.MongoDbFactoryDependentConfiguration' of type [org.springframework.boot.autoconfigure.data.mongo.MongoDbFactoryDependentConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:40,543 5343 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.boot.autoconfigure.data.mongo.MongoDbFactoryConfiguration' of type [org.springframework.boot.autoconfigure.data.mongo.MongoDbFactoryConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:40,549 5349 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration' of type [org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:40,912 5712 [main] INFO org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
fdp_1 | 2020-10-08 20:18:40,971 5771 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mongo' of type [com.mongodb.MongoClient] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,002 5802 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mongoDbFactory' of type [org.springframework.data.mongodb.core.SimpleMongoDbFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,032 5832 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.boot.autoconfigure.data.mongo.MongoDataConfiguration' of type [org.springframework.boot.autoconfigure.data.mongo.MongoDataConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,049 5849 [cluster-ClusterId{value='5f7f74207286e55d90907b1f', description='null'}-mongo:27017] INFO org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:1}] to mongo:27017
fdp_1 | 2020-10-08 20:18:41,063 5863 [cluster-ClusterId{value='5f7f74207286e55d90907b1f', description='null'}-mongo:27017] INFO org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 12]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2648730}
fdp_1 | 2020-10-08 20:18:41,130 5930 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mongoCustomConversions' of type [org.springframework.data.mongodb.core.convert.MongoCustomConversions] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,263 6063 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mongoMappingContext' of type [org.springframework.data.mongodb.core.mapping.MongoMappingContext] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,293 6093 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mappingMongoConverter' of type [org.springframework.data.mongodb.core.convert.MappingMongoConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,368 6168 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#301d8120' of type [org.springframework.beans.factory.config.ObjectFactoryCreatingFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,369 6169 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#301d8120' of type [org.springframework.beans.factory.config.ObjectFactoryCreatingFactoryBean$TargetBeanObjectFactory] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,370 6170 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'mongoTemplate' of type [org.springframework.data.mongodb.core.MongoTemplate] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,403 6203 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#29182679' of type [org.springframework.beans.factory.config.PropertiesFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,404 6204 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#29182679' of type [java.util.Properties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,405 6205 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#5432050b' of type [org.springframework.data.repository.core.support.PropertiesBasedNamedQueries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,411 6211 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#2e52fb3e' of type [org.springframework.data.repository.core.support.RepositoryFragmentsFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,412 6212 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean '(inner bean)#2e52fb3e' of type [org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,540 6340 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclRepository' of type [org.springframework.data.mongodb.repository.support.MongoRepositoryFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,542 6342 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclRepository' of type [com.sun.proxy.$Proxy119] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,545 6345 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclConfig' of type [nl.dtls.fairdatapoint.config.AclConfig$$EnhancerBySpringCGLIB$$ec5531cd] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,547 6347 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'cacheConfig' of type [nl.dtls.fairdatapoint.config.CacheConfig$$EnhancerBySpringCGLIB$$643f1005] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,554 6354 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'cacheManager' of type [org.springframework.cache.concurrent.ConcurrentMapCacheManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,559 6359 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'permissionGrantingStrategy' of type [org.springframework.security.acls.domain.DefaultPermissionGrantingStrategy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,561 6361 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclAuthorizationStrategy' of type [org.springframework.security.acls.domain.AclAuthorizationStrategyImpl] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,562 6362 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclCache' of type [org.springframework.security.acls.domain.SpringCacheBasedAclCache] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,572 6372 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'lookupStrategy' of type [org.springframework.security.acls.mongodb.BasicLookupStrategy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,572 6372 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclService' of type [org.springframework.security.acls.mongodb.MongoDBMutableAclService] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,574 6374 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'defaultMethodSecurityExpressionHandler' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,578 6378 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler@1bcf67e8' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,580 6380 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'aclMethodSecurityConfiguration' of type [nl.dtls.fairdatapoint.config.AclMethodSecurityConfiguration$$EnhancerBySpringCGLIB$$c8971de] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
fdp_1 | 2020-10-08 20:18:41,585 6385 [main] INFO org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'methodSecurityMetadataSource' of type [org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

Involuntary shutdown

(the tokens call were an attempt at a keep-alive solution)

fdp_1 | 2020-10-08 20:41:41,546 1386346 [http-nio-80-exec-5] INFO nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://fdp/tokens
fdp-client_1 | 10.162.28.92 - - [08/Oct/2020:20:41:41 +0000] "POST /tokens HTTP/1.1" 200 195 "-" "Jakarta Commons-HttpClient/3.1" "-"
fdp_1 | 2020-10-08 20:41:51,676 1396476 [http-nio-80-exec-7] INFO nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://fdp/tokens
fdp-client_1 | 10.162.28.92 - - [08/Oct/2020:20:41:51 +0000] "POST /tokens HTTP/1.1" 200 195 "-" "Jakarta Commons-HttpClient/3.1" "-"
fdp_1 | 2020-10-08 20:42:41,246 1446046 [cluster-ClusterId{value='5f7f74207286e55d90907b1f', description='null'}-mongo:27017] INFO org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
fdp_1 | com.mongodb.MongoSocketOpenException: Exception opening socket
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:131) [mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at java.lang.Thread.run(Thread.java:832) [?:?]
fdp_1 | Caused by: java.net.SocketTimeoutException: Connect timed out
fdp_1 | at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
fdp_1 | at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
fdp_1 | at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
fdp_1 | at java.net.Socket.connect(Socket.java:648) ~[?:?]
fdp_1 | at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | ... 3 more
fdp_1 | 2020-10-08 20:42:42,524 1447324 [cluster-ClusterId{value='5f7f74227286e55d90907b20', description='null'}-mongo:27017] INFO org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
fdp_1 | com.mongodb.MongoSocketOpenException: Exception opening socket
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:131) [mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at java.lang.Thread.run(Thread.java:832) [?:?]
fdp_1 | Caused by: java.net.SocketTimeoutException: Connect timed out
fdp_1 | at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
fdp_1 | at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
fdp_1 | at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
fdp_1 | at java.net.Socket.connect(Socket.java:648) ~[?:?]
fdp_1 | at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | ... 3 more
fdp_1 | 2020-10-08 20:42:42,713 1447513 [cluster-ClusterId{value='5f7f74227286e55d90907b22', description='null'}-mongo:27017] INFO org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
fdp_1 | com.mongodb.MongoSocketOpenException: Exception opening socket
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:131) [mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at java.lang.Thread.run(Thread.java:832) [?:?]
fdp_1 | Caused by: java.net.SocketTimeoutException: Connect timed out
fdp_1 | at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
fdp_1 | at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
fdp_1 | at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]
fdp_1 | at java.net.Socket.connect(Socket.java:648) ~[?:?]
fdp_1 | at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:?]
fdp_1 | ... 3 more
mongo_1 | 2020-10-08T20:56:02.826+0000 I NETWORK [conn5] end connection 172.31.0.5:54302 (7 connections now open)
mongo_1 | 2020-10-08T20:56:02.826+0000 I NETWORK [conn3] end connection 172.31.0.5:54298 (6 connections now open)
mongo_1 | 2020-10-08T20:56:43.786+0000 I NETWORK [conn7] end connection 172.31.0.5:54348 (5 connections now open)
mongo_1 | 2020-10-08T20:57:37.034+0000 I NETWORK [conn6] end connection 172.31.0.5:54304 (4 connections now open)
mongo_1 | 2020-10-08T20:58:30.282+0000 I NETWORK [conn8] end connection 172.31.0.5:54382 (3 connections now open)
mongo_1 | 2020-10-08T20:58:34.378+0000 I NETWORK [conn2] end connection 172.31.0.5:54296 (2 connections now open)
mongo_1 | 2020-10-08T20:58:34.378+0000 I NETWORK [conn4] end connection 172.31.0.5:54300 (0 connections now open)
mongo_1 | 2020-10-08T20:58:34.378+0000 I NETWORK [conn1] end connection 172.31.0.5:54294 (1 connection now open)

Configuration

# docker-compose.yml

version: '3'
services:

mongo:
    image: mongo:4.0.12
    ports:
        - 27017:27017
    volumes:
        - ./mongo/data:/data/db
blazegraph:
    image: metaphacts/blazegraph-basic:2.2.0-20160908.003514-6
    ports:
        - 8080:8080
    volumes:
        - ./blazegraph:/blazegraph-data

fdp:
    image: fairdata/fairdatapoint:1.6.0
    volumes:
        - ./application.yml:/fdp/application.yml:ro

fdp-client:
    image: fairdata/fairdatapoint-client:1.6.0
    ports:
        - 80:80
    environment:
        - FDP_HOST=fdp

"docker-compose.yml" [dos] 37L, 806C

# application.yml

# ... other configuration

repository:
type: 5
blazegraph:
url: http://blazegraph:8080/blazegraph
~
"application.yml" [readonly][noeol][dos] 8L, 139C 1,1

[kburger]

Hi @mondeelr, thank you for providing the logs. A couple of questions:

• Is the 24 minute threshold consistent?
• How much memory does the server have (or has allocated to docker)?
• And what is the status of the docker services after the 24 minute mark? The status in for example docker-compose ps?

[mondeelr]

This issue was caused by an agent (puppet-agent) removing rules created by Docker from the local firewall causing the application to stop communicating.