feat(server): Automatic watching of library folders #6192

etnoy · 2024-01-05T12:32:08Z

Adds chokidar to monitor external library folders
Adds library watch configuration via config file and web UI
Filters watched files by file extension
Checks for added, changed, and deleted files
Allows the configuration of chokidar parameters via config file (polling true/false, polling interval, stability threshold etc.)
Plenty of unit and e2e tests

Performance is still an unknown. I suggest we merge this and let daring users enable the watcher config flag and report back.

…/library-watching

cloudflare-pages · 2024-01-05T12:35:18Z

Deploying with Cloudflare Pages

Latest commit:	`aff0810`
Status:	✅ Deploy successful!
Preview URL:	https://71bdc3eb.immich.pages.dev
Branch Preview URL:	https://feat-library-watching.immich.pages.dev

View logs

bo0tzz

Does this implementation have a way to remove the watcher for a library without restarting the container?

How will the file watching behave when it's running in multiple containers simultaneously?

server/src/domain/library/library.service.ts

server/src/domain/system-config/system-config.core.ts

etnoy · 2024-01-05T12:51:19Z

Does this implementation have a way to remove the watcher for a library without restarting the container?

How will the file watching behave when it's running in multiple containers simultaneously?

It's just a draft still, much work left to be done. I wanted the GitHub actions tests to run

…/library-watching

…/immich into feat/library-watching

…/library-watching

alextran1502 · 2024-01-29T19:14:39Z

What happens if the external library gets unmounted?

…/library-watching

etnoy · 2024-01-29T22:33:20Z

What happens if the external library gets unmounted?

I tried, and it sent file deletion events for all files that are now missing. The files then become offline

…/library-watching

nodis · 2024-02-01T03:53:23Z

Hello, I upgraded to 1.94.1 and enabled “Automatic library watching (experimental)”, but the following error occurred:

[Nest] 7  - 02/01/2024, 3:51:28 AM   ERROR [LibraryService] Library watcher for library c69faf55-f96d-4aa0-b83b-2d80cbc27d98 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/media/volume1/homes/nodis/Photos/MobileBackup/Mi 10 Pro/DCIM/Screenshots/@eaDir/Screenshot_2015-12-25-23-41-06_com.excelliance.dualaid.png'
2024/02/01
11:51:28
[Nest] 7  - 02/01/2024, 3:51:28 AM   ERROR [LibraryService] Library watcher for library c69faf55-f96d-4aa0-b83b-2d80cbc27d98 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/media/volume1/homes/nodis/Photos/MobileBackup/Mi 10 Pro/DCIM/Screenshots/@eaDir/Screenshot_2015-12-25-23-41-08_com.excelliance.dualaid.jpg'
2024/02/01
11:51:28
[Nest] 7  - 02/01/2024, 3:51:28 AM   ERROR [LibraryService] Library watcher for library c69faf55-f96d-4aa0-b83b-2d80cbc27d98 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/media/volume1/homes/nodis/Photos/MobileBackup/Mi 10 Pro/DCIM/Screenshots/@eaDir/Screenshot_2015-12-25-23-41-06_com_1457834466869.excelliance.dualaid.png'
2024/02/01
11:51:28
[Nest] 7  - 02/01/2024, 3:51:28 AM   ERROR [LibraryService] Library watcher for library c69faf55-f96d-4aa0-b83b-2d80cbc27d98 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/media/volume1/homes/nodis/Photos/MobileBackup/Mi 10 Pro/DCIM/Screenshots/@eaDir/Screenshot_2015-12-25-23-41-08_com_1457834467052.excelliance.dualaid.png'
2024/02/01
11:51:28
[Nest] 7  - 02/01/2024, 3:51:28 AM   ERROR [LibraryService] Library watcher for library c69faf55-f96d-4aa0-b83b-2d80cbc27d98 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/media/volume1/homes/nodis/Photos/MobileBackup/Mi 10 Pro/DCIM/Screenshots/@eaDir/Screenshot_2015-12-25-23-42-24_com.excelliance.dualaid.png'

alextran1502 · 2024-02-01T04:50:12Z

@nodis you can fix your system with this https://stackoverflow.com/questions/55763428/react-native-error-enospc-system-limit-for-number-of-file-watchers-reached

nodis · 2024-02-01T06:04:26Z

@nodis you can fix your system with this https://stackoverflow.com/questions/55763428/react-native-error-enospc-system-limit-for-number-of-file-watchers-reached

@alextran1502 thank you very much

nodis · 2024-02-01T09:20:12Z

@nodis you can fix your system with this https://stackoverflow.com/questions/55763428/react-native-error-enospc-system-limit-for-number-of-file-watchers-reached

@alextran1502 hello，I followed the instructions and added a line to the sysctl. conf file: "fs.inotify.max_user_watches=524288"
The "immich" web homepage cannot be opened, displaying "502 Bad Gateway",
The error log for container "immich_server" is as follows:

[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +39ms
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)

[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (2)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (4)...
Error: connect ECONNREFUSED 172.21.0.5:5432
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 02/01/2024, 8:25:57 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (6)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (7)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (8)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [TypeOrmModule] Unable to connect to the database. Retrying (9)...
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:25:57 AM   ERROR [ExceptionHandler] connect ECONNREFUSED 172.21.0.5:5432
Error: connect ECONNREFUSED 172.21.0.5:5432
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
[Nest] 7  - 02/01/2024, 8:30:41 AM     LOG [SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 02/01/2024, 8:30:49 AM     LOG [LibraryService] Starting to watch library 8b05f464-3703-4834-a886-89d97f1bbdda with import path(s) /media/volume2/photo
[Nest] 7  - 02/01/2024, 8:49:33 AM     LOG [SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 02/01/2024, 8:49:35 AM     LOG [LibraryService] Starting to watch library 8b05f464-3703-4834-a886-89d97f1bbdda with import path(s) /media/volume2/photo
/usr/src/app/node_modules/pg-pool/index.js:205
        response.callback(new Error('timeout exceeded when trying to connect'))

                          ^
Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)

Node.js v20.11.0

[Nest] 7  - 02/01/2024, 8:56:06 AM     LOG [SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 02/01/2024, 8:56:06 AM     LOG [LibraryService] Starting to watch library 8b05f464-3703-4834-a886-89d97f1bbdda with import path(s) /media/volume2/photo

danieldietzler · 2024-02-01T11:07:57Z

@nodis Your database isn't running. Can you double check that?

etnoy added 3 commits January 5, 2024 13:18

feat: initial watch support

b107a50

Merge branch 'main' of https://github.com/immich-app/immich into feat…

a0e28b4

…/library-watching

allow offline files

a8f7c95

etnoy added 🗄️server external-library Issues related to external libraries labels Jan 5, 2024

bo0tzz reviewed Jan 5, 2024

View reviewed changes

server/src/domain/library/library.service.ts Outdated Show resolved Hide resolved

server/src/domain/library/library.service.ts Outdated Show resolved Hide resolved

server/src/domain/system-config/system-config.core.ts Outdated Show resolved Hide resolved

etnoy added 22 commits January 6, 2024 00:20

chore: ignore query errors when resetting e2e db

c64ba21

revert db query

791585b

add savepoint

e72f47f

guard the user query

96be1dd

chore: openapi and db migration

eeadbfb

wip

ae5df0c

Merge branch 'main' of https://github.com/immich-app/immich into feat…

de00a17

…/library-watching

support multiple libraries

e187ad3

Merge branch 'main' of https://github.com/immich-app/immich into feat…

334a38d

…/library-watching

fix tests

c831c53

wip

62efb7e

can now cleanup chokidar watchers

24af782

fix unit tests

f5005b3

add library watch queue

5b3192e

Merge branch 'main' of https://github.com/immich-app/immich into feat…

e55365a

…/library-watching

add missing init from merge

c67a0dc

wip

a3576cf

Merge branch 'main' of https://github.com/immich-app/immich into feat…

2c93818

…/library-watching

can now filter file extensions

868ceb5

Merge branch 'main' of https://github.com/immich-app/immich into feat…

ec29808

…/library-watching

remove watch api from non job client

57450e1

Fix e2e test

c75e10e

etnoy added 9 commits January 29, 2024 11:50

fix: import right app service

3e711e8

Merge branch 'feat/library-watching' of https://github.com/immich-app…

abd35e7

…/immich into feat/library-watching

Merge branch 'main' of https://github.com/immich-app/immich into feat…

f354947

…/library-watching

don't be truthy

4d0fd0d

fix test speling

b3a3424

stricter library update tests

5911a43

move fs watcher mock to external file

cd78deb

subscribe to config changes

20e4dd7

docker does not need polling

245d2e0

etnoy added 2 commits January 29, 2024 23:06

make library watch() private

6a0bf61

Merge branch 'main' of https://github.com/immich-app/immich into feat…

ca278ed

…/library-watching

etnoy and others added 3 commits January 30, 2024 00:04

feat: add configuration ui

41c4617

Merge branch 'main' of https://github.com/immich-app/immich into feat…

e044bdc

…/library-watching

merge main

c84d216

alextran1502 approved these changes Jan 30, 2024

View reviewed changes

Merge branch 'main' of https://github.com/immich-app/immich into feat…

2773c18

…/library-watching

etnoy enabled auto-merge (squash) January 30, 2024 23:47

etnoy added 4 commits January 31, 2024 00:54

fix compilation error

e09301c

Merge branch 'main' of https://github.com/immich-app/immich into feat…

ea613d1

…/library-watching

add milestone

af51b64

fix e2e test

aff0810

etnoy merged commit 068e703 into main Jan 31, 2024
24 checks passed

etnoy deleted the feat/library-watching branch January 31, 2024 08:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): Automatic watching of library folders #6192

feat(server): Automatic watching of library folders #6192

etnoy commented Jan 5, 2024 •

edited

Loading

cloudflare-pages bot commented Jan 5, 2024 •

edited

Loading

bo0tzz left a comment

etnoy commented Jan 5, 2024

alextran1502 commented Jan 29, 2024

etnoy commented Jan 29, 2024

nodis commented Feb 1, 2024

alextran1502 commented Feb 1, 2024

nodis commented Feb 1, 2024

nodis commented Feb 1, 2024

danieldietzler commented Feb 1, 2024

feat(server): Automatic watching of library folders #6192

feat(server): Automatic watching of library folders #6192

Conversation

etnoy commented Jan 5, 2024 • edited Loading

cloudflare-pages bot commented Jan 5, 2024 • edited Loading

Deploying with Cloudflare Pages

bo0tzz left a comment

Choose a reason for hiding this comment

etnoy commented Jan 5, 2024

alextran1502 commented Jan 29, 2024

etnoy commented Jan 29, 2024

nodis commented Feb 1, 2024

alextran1502 commented Feb 1, 2024

nodis commented Feb 1, 2024

nodis commented Feb 1, 2024

danieldietzler commented Feb 1, 2024

etnoy commented Jan 5, 2024 •

edited

Loading

cloudflare-pages bot commented Jan 5, 2024 •

edited

Loading