[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense #2224

clam0r · 2023-04-09T11:51:30Z

The bug

Can't connect to immich after update to version 1.53.0.

I have tried this using the default docker-compose.yml and .env using an empty media folder and still recieve the same result.

I am running my docker compose using Portainer BE 2.17.1 if that is for any importance.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.53.0

Version of Immich Mobile App

not used

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    volumes:
      - tsdata:/data
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080

    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/home/clamor/proj_immich/media
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
TYPESENSE_API_KEY=ijöolghvgvwrtijökrgewöhukicuhöoikerudgksfhöv

Reproduction steps

1. start docker yml
2. go to website of immich (http://10.180.36.111:2283/)
3. see error:  500 connect ECONNREFUSED 172.18.0.8:3001

Additional information

Error log of immich server:

2023-04-09T11:46:14.023730080Z [Nest] 1  - 04/09/2023, 11:46:14 AM     LOG [RouterExplorer] Mapped {/album/:albumId/download, GET} route +2ms

2023-04-09T11:46:14.025105527Z [Nest] 1  - 04/09/2023, 11:46:14 AM     LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +2ms

2023-04-09T11:46:14.070500307Z [Nest] 1  - 04/09/2023, 11:46:14 AM     LOG [SearchService] Running bootstrap

2023-04-09T11:46:19.093769064Z Request #1681040774073: Request to Node 0 failed due to "EAI_AGAIN getaddrinfo EAI_AGAIN typesense"

2023-04-09T11:46:19.093849171Z Request #1681040774073: Sleeping for 4s and then retrying request...

2023-04-09T11:46:23.100113851Z Request #1681040774073: Request to Node 0 failed due to "ECONNREFUSED connect ECONNREFUSED 172.19.0.6:8108"

2023-04-09T11:46:23.100171749Z Request #1681040774073: Sleeping for 4s and then retrying request...

2023-04-09T11:46:32.106677476Z Request #1681040774073: Request to Node 0 failed due to "EAI_AGAIN getaddrinfo EAI_AGAIN typesense"

2023-04-09T11:46:32.106944408Z Request #1681040774073: Sleeping for 4s and then retrying request...

2023-04-09T11:46:46.111684631Z Request #1681040774073: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"

2023-04-09T11:46:46.111744345Z Request #1681040774073: Sleeping for 4s and then retrying request...

2023-04-09T11:46:55.121473721Z Request #1681040774073: Request to Node 0 failed due to "EAI_AGAIN getaddrinfo EAI_AGAIN typesense"

2023-04-09T11:46:55.121674164Z Request #1681040774073: Sleeping for 4s and then retrying request...

2023-04-09T11:47:04.130508975Z Request #1681040774073: Request to Node 0 failed due to "EAI_AGAIN getaddrinfo EAI_AGAIN typesense"

2023-04-09T11:47:04.130591038Z Request #1681040774073: Sleeping for 4s and then retrying request...

Error log of Typesense:

2023-04-09T11:46:22.929646979Z W20230409 11:46:22.929589   118 raft_server.cpp:570] Single-node with no leader. Resetting peers.

2023-04-09T11:46:22.929746083Z W20230409 11:46:22.929688   118 node.cpp:894] node default_group:172.19.0.6:8107:8108 set_peer from 172.18.0.6:8107:8108 to 172.19.0.6:8107:8108

2023-04-09T11:46:22.931380989Z I20230409 11:46:22.931210   118 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 21 votedfor 0.0.0.0:0:0 time: 1415

2023-04-09T11:46:22.983584211Z E20230409 11:46:22.983394     7 backward.hpp:4199] Stack trace (most recent call last) in thread 7:

2023-04-09T11:46:22.983741702Z E20230409 11:46:22.983669     7 backward.hpp:4199] #7    Object "", at 0xffffffffffffffff, in 

2023-04-09T11:46:22.983864482Z E20230409 11:46:22.983800     7 backward.hpp:4199] #6    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fe0eec69bb3, in __clone

2023-04-09T11:46:22.983971688Z E20230409 11:46:22.983893     7 backward.hpp:4199] #5    Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fe0eebd8b42, in 

2023-04-09T11:46:22.984097821Z E20230409 11:46:22.984035     7 backward.hpp:4199] #4    Object "/opt/typesense-server", at 0x84d418, in bvar::detail::SamplerCollector::sampling_thread(void*)

2023-04-09T11:46:22.984202233Z E20230409 11:46:22.984125     7 backward.hpp:4199] #3    Object "/opt/typesense-server", at 0x84b908, in bvar::detail::SamplerCollector::run()

2023-04-09T11:46:22.984318309Z E20230409 11:46:22.984256     7 backward.hpp:4199] #2  | Source "/usr/local/include/bvar/passive_status.h", line 62, in get_value

2023-04-09T11:46:22.984422232Z E20230409 11:46:22.984346     7 backward.hpp:4199]       Source "/usr/local/include/bvar/passive_status.h", line 140, in take_sample [0x7a4413]

2023-04-09T11:46:22.984535444Z E20230409 11:46:22.984474     7 backward.hpp:4199] #1    Source "/opt/braft-80d97b2475b3c0afca79c19b64d46bb665d704f4/src/braft/util.cpp", line 215, in counter_percentile [0x7e18cf]

2023-04-09T11:46:22.984664021Z E20230409 11:46:22.984596     7 backward.hpp:4199] #0    Source "/usr/local/include/bvar/detail/percentile.h", line 294, in snprintf [0x7e52d5]

2023-04-09T11:46:22.984806986Z Illegal instruction (Illegal operand [0x7e52d5])

2023-04-09T11:46:23.078912942Z E20230409 11:46:23.078696     7 typesense_server.cpp:102] Typesense 0.24.0 is terminating abruptly.

This is my docker ps output:

CONTAINER ID   IMAGE                                                COMMAND                  CREATED              STATUS                            PORTS                                                                                  NAMES
7e909eb69cee   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   About a minute ago   Up About a minute                 0.0.0.0:2283->8080/tcp, :::2283->8080/tcp                                              immich_proxy
00711bfde396   ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-mic…"   About a minute ago   Up About a minute                 3001/tcp                                                                               immich_microservices
4c38ccb053f0   ghcr.io/immich-app/immich-server:release             "/bin/sh ./start-ser…"   About a minute ago   Up About a minute                 3001/tcp                                                                               immich_server
10d36500c734   typesense/typesense:0.24.1                           "/opt/typesense-serv…"   About a minute ago   Restarting (132) 26 seconds ago                                                                                          immich_typesense
c221980268dd   ghcr.io/immich-app/immich-web:release                "/bin/sh ./entrypoin…"   19 minutes ago       Up 19 minutes                     3000/tcp                                                                               immich_web
8c055c918d9b   ghcr.io/immich-app/immich-machine-learning:release   "gunicorn src.main:s…"   19 minutes ago       Up 19 minutes                                                                                                            immich_machine_learning
16387cf6af03   redis:6.2                                            "docker-entrypoint.s…"   27 minutes ago       Up 27 minutes                     6379/tcp                                                                               immich_redis
89d083ba6431   postgres:14                                          "docker-entrypoint.s…"   27 minutes ago       Up 27 minutes                     5432/tcp                                                                               immich_postgres

alextran1502 · 2023-04-09T11:53:51Z

What is the CPU used in this machine? What version did you upgrade from?

clam0r · 2023-04-09T11:58:33Z

I upgraded from v1.50.1 but get the same results on a fresh install

CPU Model Name: Pentium(R) Dual-Core CPU E5500 @ 2.80GHz

bo0tzz · 2023-04-09T12:03:39Z

That's a very old CPU that typesense probably won't work on (although you'd have to ask them to be sure). I recommend removing typesense from your docker-compose.yml and setting TYPESENSE_ENABLED=false in your .env.

pacjo · 2023-04-10T19:05:54Z

Just had the same issue. Disabling typesense fixed it, but wouldn't there be a way to have search with older cpus? Mine is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz

badcrc · 2023-05-21T09:37:36Z

I just got this error while updating my stack.

My CPU is:

vendor_id       : GenuineIntel
cpu family      : 6
model           : 42
model name      : Intel(R) Xeon(R) CPU E31260L @ 2.40GHz
stepping        : 7
microcode       : 0x2f
cpu MHz         : 2394.556
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips        : 4789.11
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

I didn't find anything on the typesense requirements site, so I must assume that it uses AVX2 or something new like that?

bo0tzz · 2023-05-21T09:56:38Z

Typesense uses AVX, yes. I believe some people have had success with compiling it themselves to get around that.

Ainyaku · 2023-08-16T02:49:06Z

Is there any workaround for this yet without disabling search? I have the same issue on a Intel Pentium D CPU 2.80GHz

clam0r added bug Something isn't working needs triage Bug that needs triage from maintainer labels Apr 9, 2023

bo0tzz closed this as completed Apr 9, 2023

pacjo mentioned this issue Apr 10, 2023

Add an option for disabling typesense imagegenius/docker-immich#59

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense #2224

[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense #2224

clam0r commented Apr 9, 2023 •

edited

alextran1502 commented Apr 9, 2023

clam0r commented Apr 9, 2023

bo0tzz commented Apr 9, 2023

pacjo commented Apr 10, 2023

badcrc commented May 21, 2023

bo0tzz commented May 21, 2023

Ainyaku commented Aug 16, 2023

[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense #2224

[BUG] connect ECONNREFUSED 172.18.0.8:3001 due to typesense #2224

Comments

clam0r commented Apr 9, 2023 • edited

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Additional information

alextran1502 commented Apr 9, 2023

clam0r commented Apr 9, 2023

bo0tzz commented Apr 9, 2023

pacjo commented Apr 10, 2023

badcrc commented May 21, 2023

bo0tzz commented May 21, 2023

Ainyaku commented Aug 16, 2023

clam0r commented Apr 9, 2023 •

edited