Uploading large video file via immich cli fails #7052

Froghut · 2024-02-12T17:05:58Z

The bug

When I try to upload a ~11GB video file via immich cli on windows I get the following error:

file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:9719
            throw new FetchError(e, "The request failed and the interceptors did not return an alternative response");
                  ^

FetchError: The request failed and the interceptors did not return an alternative response
    at AssetApi.fetchApi (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:9719:19)
    at async AssetApi.request (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:9771:22)
    ... 3 lines matching cause stack trace ...
    at async Command.<anonymous> (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:22454:3) {
  cause: TypeError: fetch failed
      at node:internal/deps/undici/undici:12442:11
      at async AssetApi.fetchApi (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:9704:20)
      at async AssetApi.request (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:9771:22)
      at async AssetApi.uploadFileRaw (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:13469:22)
      at async AssetApi.uploadFile (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:13481:22)
      at async UploadCommand.run (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:22389:30)
      at async Command.<anonymous> (file:///C:/Users/user/scoop/persist/nodejs/bin/node_modules/@immich/cli/dist/index.js:22454:3) {
    cause: RequestContentLengthMismatchError: Request body length does not match content-length header
        at AsyncWriter.write (node:internal/deps/undici/undici:9780:19)
        at writeIterable (node:internal/deps/undici/undici:9740:23) {
      code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
    }
  }
}

I connect directly to the immich docker ocntainer. There is no error message in the immich docker logs.

The OS that Immich Server is running on

Docker for Windows on Windows 11

Version of Immich Server

v1.94.1

Version of Immich Mobile App

v1.94.1

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:release
    command: [ "start.sh", "immich" ]
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.88
    volumes:
      - d:/bilder/immich:/usr/src/app/upload
    ports:
      - "13001:3001"
    env_file:
      - ../immich/immich.env
    environment:
      - NODE_ENV=production
    depends_on:
      - immich-redis
      - immich-postgres
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.immich-rtr.entrypoints=https"
      - "traefik.http.routers.immich-rtr.rule=Host(`example.com`)" 
      - "traefik.http.routers.immich-rtr.service=immich-svc"
      - "traefik.http.services.immich-svc.loadbalancer.server.port=3001"

  immich-microservices:
    container_name: immich-microservices
    image: ghcr.io/immich-app/immich-server:release
    command:  [ "start.sh", "microservices" ]
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.89
    volumes:
      - d:/bilder/immich:/usr/src/app/upload
    env_file:
      - ../immich/immich.env
    environment:
      - NODE_ENV=production
    depends_on:
      - immich-redis
      - immich-postgres
    restart: unless-stopped

  immich-machine-learning:
    container_name: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.90
    volumes:
      - c:/docker/immich/cache:/cache
    env_file:
      - ../immich/immich.env
    environment:
      - NODE_ENV=production
    restart: unless-stopped

  immich-redis:
    container_name: immich-redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.92
    restart: unless-stopped
    
  immich-postgres:
    container_name: immich-postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.93
    env_file:
      - ../immich/immich.env
    environment:
      POSTGRES_PASSWORD: xxx
      POSTGRES_USER: postgres
      POSTGRES_DB: immich
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - immich_db_data:/var/lib/postgresql/data
    restart: unless-stopped

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich-postgres
DB_USERNAME=postgres
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich-redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=d:/bilder/immich/

###################################################################################
# JWT SECRET
#
# This JWT_SECRET is used to sign the authentication keys for user login
# You should set it to a long randomly generated value
# You can use this command to generate one: openssl rand -base64 128
###################################################################################

JWT_SECRET=xxx

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

Upload a big video file via immich cli

Additional information

No response

The text was updated successfully, but these errors were encountered:

lakestonelabs · 2024-02-13T16:01:20Z

I just ran into the same problem yesterday when trying to upload video files that are greater than 4GB. I'm running v1.94.1 on the server. I used the docker method to run the cli import tool and am using version latest on that container. Finding these large files find . -type f -size +4G and moving them outside of the import location seems to help but would be best if we could import these large files. The files that are causing this error on are .MOV files from my iPhone.

refringe · 2024-02-20T00:38:37Z

I also just received this error when attempting to upload a large collection of images and videos through the CLI:

docker run -it -v "$(pwd)":/import:ro -e IMMICH_INSTANCE_URL=http://192.168.0.3:8082/api -e IMMICH_API_KEY=KEY_WAS_HERE -v '/mnt/user/family/Photos':'/photo-import':'rw' ghcr.io/immich-app/immich-cli:latest upload --include-hidden --recursive /photo-import

file:///usr/src/app/dist/index.js:9719
            throw new FetchError(e, "The request failed and the interceptors did not return an alternative response");
                  ^

FetchError: The request failed and the interceptors did not return an alternative response
    at AssetApi.fetchApi (file:///usr/src/app/dist/index.js:9719:19)
    at async AssetApi.request (file:///usr/src/app/dist/index.js:9771:22)
    ... 3 lines matching cause stack trace ...
    at async Command.<anonymous> (file:///usr/src/app/dist/index.js:22125:3) {
  cause: TypeError: fetch failed
      at node:internal/deps/undici/undici:12344:11
      at async AssetApi.fetchApi (file:///usr/src/app/dist/index.js:9704:20)
      at async AssetApi.request (file:///usr/src/app/dist/index.js:9771:22)
      at async AssetApi.uploadFileRaw (file:///usr/src/app/dist/index.js:13168:22)
      at async AssetApi.uploadFile (file:///usr/src/app/dist/index.js:13180:22)
      at async UploadCommand.run (file:///usr/src/app/dist/index.js:22060:30)
      at async Command.<anonymous> (file:///usr/src/app/dist/index.js:22125:3) {
    cause: RequestContentLengthMismatchError: Request body length does not match content-length header
        at AsyncWriter.write (node:internal/deps/undici/undici:9682:19)
        at writeIterable (node:internal/deps/undici/undici:9642:23) {
      code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
    }
  }
}

The file that it tripped on is a 11.7 GB large MP4. When I reran the import the script halted on the same file a second time. This is on an Unraid system with 128 GB of RAM.

For a work-around I'll attempt to remove anything larger than 4GB from the import directory like @lakestonelabs suggested.

If importing files this large isn't a possibility, we should at least be catching the error, logging it, and continuing with the import.

justin13888 · 2024-02-27T04:39:51Z

Same problem when using deployment via Docker Compose on Ubuntu 22.04. Cli is installed on Node v21.6.2 on same machine

justin13888 · 2024-02-27T07:06:39Z

Also, same files managed to upload over website so it's more likely an issue with CLI

Zelnes · 2024-03-07T10:13:52Z

Same behaviour, size bigger than 4Go failed to import. And the --ignore parameters does not seem to work, I had to move the files.
Also, when I upload a lot of files, I have to disable all jobs, otherwise the uploads fails.

Version : v1.97.0

maximberezin97 · 2024-03-11T02:11:22Z

I am experiencing this same issue running Immich 1.98.1 on a Compose setup with Node.js v20.11.1 and Immich CLI 2.0.7.

$ docker exec -ti immich_server bash
node@207a6cbdc17a:/usr/src/app$ id -u
1000
node@207a6cbdc17a:/usr/src/app$ cd upload/User
node@207a6cbdc17a:/usr/src/app/upload/User$ ls -lhAF
-rw-rw-rw- 1 99 users 5.6G Mar 10  2024  big_file.mp4
node@207a6cbdc17a:/usr/src/app/upload/User$ immich login-key http://localhost:3001/api MY_API_KEY
Logging in...
Logged in as user@example.com
Wrote auth info to /home/node/.config/immich/auth.yml
node@207a6cbdc17a:/usr/src/app/upload/User$ time immich upload /usr/src/app/upload/User
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% | ETA: 0s | 0/5.6 GB: /usr/src/app/upload/User/big_file.mp4
file:///usr/src/app/node_modules/@immich/cli/dist/index.js:9719
            throw new FetchError(e, "The request failed and the interceptors did not return an alternative response");
                  ^

FetchError: The request failed and the interceptors did not return an alternative response
    at AssetApi.fetchApi (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:9719:19)
    at async AssetApi.request (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:9771:22)
    ... 3 lines matching cause stack trace ...
    at async Command.<anonymous> (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:22454:3) {
  cause: TypeError: fetch failed
      at node:internal/deps/undici/undici:12345:11
      at async AssetApi.fetchApi (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:9704:20)
      at async AssetApi.request (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:9771:22)
      at async AssetApi.uploadFileRaw (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:13469:22)
      at async AssetApi.uploadFile (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:13481:22)
      at async UploadCommand.run (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:22389:30)
      at async Command.<anonymous> (file:///usr/src/app/node_modules/@immich/cli/dist/index.js:22454:3) {
    cause: RequestContentLengthMismatchError: Request body length does not match content-length header
        at AsyncWriter.write (node:internal/deps/undici/undici:9682:19)
        at writeIterable (node:internal/deps/undici/undici:9642:23) {
      code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
    }
  }
}

datatags · 2024-03-12T17:53:11Z

Affects CLI v2.1.0 as well, but gives a slightly different error message

$ immich upload IMG_5820.MOV
Crawling for assets...
Checking assets | ████████████████████████████████████████ | 100% | ETA: 0s | 2/2 assets
Uploading assets | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% | ETA: 0s | 0/21.0 GB
node:internal/deps/undici/undici:13737
      Error.captureStackTrace(err, this);
            ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:13737:13
    at async UploadCommand.uploadAsset (file:///home/datatags/.nvm/versions/node/v21.7.1/lib/node_modules/@immich/cli/dist/index.js:18473:22)
    at async Promise.all (index 0)
    at async UploadCommand.upload (file:///home/datatags/.nvm/versions/node/v21.7.1/lib/node_modules/@immich/cli/dist/index.js:18297:21)
    at async UploadCommand.run (file:///home/datatags/.nvm/versions/node/v21.7.1/lib/node_modules/@immich/cli/dist/index.js:18220:31)
    at async Command.<anonymous> (file:///home/datatags/.nvm/versions/node/v21.7.1/lib/node_modules/@immich/cli/dist/index.js:18501:3) {
  [cause]: RequestContentLengthMismatchError: Request body length does not match content-length header
      at AsyncWriter.write (node:internal/deps/undici/undici:10497:19)
      at writeIterable (node:internal/deps/undici/undici:10457:23) {
    code: 'UND_ERR_REQ_CONTENT_LENGTH_MISMATCH'
  }
}

Node.js v21.7.1

alternativesurfer · 2024-03-12T18:42:10Z

I wonder if its a greater issue than just CLI.
I am unable to upload files larger than 150MB through the browser right now, but if I upload through the mobile app it works just fine. Mobile app I tested to upload 6GB files.

mccarverone · 2024-03-16T08:47:49Z

This is definitely still an issue

mccarverone · 2024-03-20T20:48:25Z

And it affects 1.99.0 as well.

aviv926 added the cli Tasks related to the Immich CLI label Feb 19, 2024

aviv926 mentioned this issue Feb 19, 2024

feat(cli): concurrent upload #7192

Merged

jrasm91 mentioned this issue Mar 22, 2024

refactor: cli #8199

Merged

jrasm91 closed this as completed in #8199 Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uploading large video file via immich cli fails #7052

Uploading large video file via immich cli fails #7052

Froghut commented Feb 12, 2024

lakestonelabs commented Feb 13, 2024 •

edited

refringe commented Feb 20, 2024 •

edited

justin13888 commented Feb 27, 2024

justin13888 commented Feb 27, 2024 •

edited

Zelnes commented Mar 7, 2024 •

edited

maximberezin97 commented Mar 11, 2024 •

edited

datatags commented Mar 12, 2024

alternativesurfer commented Mar 12, 2024

mccarverone commented Mar 16, 2024

mccarverone commented Mar 20, 2024

Uploading large video file via immich cli fails #7052

Uploading large video file via immich cli fails #7052

Comments

Froghut commented Feb 12, 2024

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Additional information

lakestonelabs commented Feb 13, 2024 • edited

refringe commented Feb 20, 2024 • edited

justin13888 commented Feb 27, 2024

justin13888 commented Feb 27, 2024 • edited

Zelnes commented Mar 7, 2024 • edited

maximberezin97 commented Mar 11, 2024 • edited

datatags commented Mar 12, 2024

alternativesurfer commented Mar 12, 2024

mccarverone commented Mar 16, 2024

mccarverone commented Mar 20, 2024

lakestonelabs commented Feb 13, 2024 •

edited

refringe commented Feb 20, 2024 •

edited

justin13888 commented Feb 27, 2024 •

edited

Zelnes commented Mar 7, 2024 •

edited

maximberezin97 commented Mar 11, 2024 •

edited