Timeout connecting to Elasticsearch

[dejmail]

Hi there

I've taken your docker-compose.yml file and deployed it on a VPS with 16gb RAM, 4 cores and plenty disk space. I keep getting timeout errors on Snowstorm, and not sure where else to go from here. I've adjusted the vm.max_map_count. The containers are in the same network, so should be able to communicate with each other.

version: '2.1'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    container_name: elasticsearch
    environment:
      - node.name=snowstorm
      - cluster.name=snowstorm-cluster
      - cluster.initial_master_nodes=snowstorm
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
      # - discovery.type=single-node
    volumes:
      - elastic:/usr/share/elasticsearch/data
    networks:
      elastic:
        aliases:
         - es
    healthcheck:
      test: ["CMD", "curl", "-f", "http://es:9200"]
      interval: 1s
      timeout: 1s
      retries: 60
    ports:
      - 127.0.0.1:9200:9200
    mem_reservation: 4g


  snowstorm:
    image: snomedinternational/snowstorm:latest
    container_name: snowstorm
    restart: on-failure
    depends_on:
      elasticsearch:
        condition: service_healthy
    entrypoint: java -Xms2g -Xmx4g -jar snowstorm.jar --elasticsearch.urls=http://es:9200
    networks:
      elastic:
        aliases:
         - snowstorm
    ports:
      - 8080:8080

  browser:
    image: snomedinternational/snomedct-browser:latest
    container_name: browser
    depends_on:
      - snowstorm
    links:
      - snowstorm:snowstorm
    networks:
      - elastic
    environment:
      - API_HOST=http://snowstorm:8080/
    ports:
      - 80:80
    restart: always


networks:
  elastic:

volumes:
  elastic:

and the errors are

:: Snowstorm, SNOMED CT Terminology Server :: (v8.1.0) 2023-08-08 10:58:45.751 INFO 1 --- [ main] o.snomed.snowstorm.SnowstormApplication : Starting SnowstormApplication v8.1.0 using Java 11.0.19 on f8cd1bc03097 with PID 1 (/app/snowstorm.jar started by snowstorm in /app) 2023-08-08 10:58:45.771 INFO 1 --- [ main] o.snomed.snowstorm.SnowstormApplication : No active profile set, falling back to 1 default profile: "default" 2023-08-08 10:58:52.123 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode! 2023-08-08 10:58:52.125 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Elasticsearch repositories in DEFAULT mode. 2023-08-08 10:58:52.693 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 515 ms. Found 25 Elasticsearch repository interfaces. 2023-08-08 10:58:53.927 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode! 2023-08-08 10:58:53.928 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Reactive Elasticsearch repositories in DEFAULT mode. 2023-08-08 10:58:53.961 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 32 ms. Found 0 Reactive Elasticsearch repository interfaces. 2023-08-08 10:58:55.335 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=e9f6b9d0-8d6a-37a1-a60b-7fc787b540e6 2023-08-08 10:58:57.754 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'permissionEvaluator' of type [org.snomed.snowstorm.config.MethodSecurityConfig$1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2023-08-08 10:58:57.754 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'methodSecurityConfig' of type [org.snomed.snowstorm.config.MethodSecurityConfig$$EnhancerBySpringCGLIB$$6a99a476] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2023-08-08 10:58:58.471 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http) 2023-08-08 10:58:58.490 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat] 2023-08-08 10:58:58.490 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.63] 2023-08-08 10:58:58.599 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext 2023-08-08 10:58:58.600 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 12501 ms 2023-08-08 10:58:58.749 INFO 1 --- [ main] ca.uhn.fhir.util.VersionUtil : HAPI FHIR version 5.6.3-si - Rev 996095adf0 2023-08-08 10:58:59.328 INFO 1 --- [ main] ication$$EnhancerBySpringCGLIB$$f5740159 : Elasticsearch host: http://es:9200 2023-08-08 10:58:59.328 INFO 1 --- [ main] ication$$EnhancerBySpringCGLIB$$f5740159 : Elasticsearch index prefix: 2023-08-08 10:59:00.032 INFO 1 --- [ main] o.s.s.config.elasticsearch.IndexConfig : Index name prefix: '' 2023-08-08 10:59:01.362 INFO 1 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Spring Data Elasticsearch: 4.1.16 2023-08-08 10:59:01.362 INFO 1 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client in build: 7.10.0 2023-08-08 10:59:01.362 INFO 1 --- [ main] o.s.d.elasticsearch.support.VersionInfo : Version Elasticsearch Client used: 7.10.0 2023-08-08 10:59:02.899 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:04.036 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:05.114 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:06.174 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:07.238 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:08.277 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:09.393 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:10.299 INFO 1 --- [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Identifier cache manager polling commencing with 10 second period. 2023-08-08 10:59:10.303 INFO 1 --- [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Topping up IdCache(0 - 00) by 1000 2023-08-08 10:59:11.471 WARN 1 --- [ main] .s.d.e.r.s.SimpleElasticsearchRepository : Cannot create index: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] 2023-08-08 10:59:11.479 ERROR 1 --- [heManagerDaemon] o.s.s.c.d.s.i.IdentifierCacheManager : Failed to top-up IdCache(0 - 00) with 1000 identifiers org.springframework.dao.DataAccessResourceFailureException: Timeout connecting to [es/172.19.0.2:9200]; nested exception is java.lang.RuntimeException: Timeout connecting to [es/172.19.0.2:9200] at...

Anything else I have perhaps forgotten to do ?

Thanks

[dejmail]

Annoyingly a system reboot fixed the issue, or perhaps running the whole instance as an unprivilidged user. I was setting this all up in Ansible, which is possibly why that happened, wouldn't normally run a docker container as root.