No space left on device ffmpeg? #15

yetdog · 2023-05-23T13:57:19Z

A couple of minutes after starting the app:

birdcage-birdcage_backend-1   | [2023-05-23 06:53:53,189: INFO/MainProcess] Task app.stream_processing.record_stream[e96a1804-b4f7-49f0-9533-498501bbf974] received
birdcage-birdcage_backend-1   | [2023-05-23 06:53:53,196: INFO/MainProcess] Task app.stream_processing.record_stream[72ccb0d9-d551-4aef-8282-28c09023effb] received
birdcage-birdcage_backend-1   | [2023-05-23 06:53:53,209: INFO/MainProcess] Task app.stream_processing.record_stream[fdfc459f-9c70-4c8a-928b-1bc81076f3ea] received
birdcage-birdcage_backend-1   | [2023-05-23 06:53:53,225: INFO/MainProcess] Task app.stream_processing.analyze_recordings[7d9c8ed6-d884-4f5c-b916-58d566c40b8c] received
birdcage-birdcage_backend-1   | [2023-05-23 06:54:08,405: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/01dc7e2d0d2c4d22be7cd6d61a599a60.wav
birdcage-birdcage_backend-1   | [2023-05-23 06:54:24,629: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/c40c82589b4a4cc4b1da690c8b22d389.wav
birdcage-birdcage_backend-1   | [2023-05-23 06:54:40,981: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/1611058f51eb4307837829a51cb5410b.wav
birdcage-birdcage_backend-1   | [2023-05-23 06:54:57,151: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/6c806fdd70914f2faa912fe54e1455ce.wav
birdcage-birdcage_backend-1   | [2023-05-23 06:55:13,367: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/f99913c16d204fd38bf4af21874e7bde.wav
birdcage-birdcage_backend-1   | av_interleaved_write_frame(): No space left on device
birdcage-birdcage_backend-1   | Error writing trailer of /app/../tmp/29c32bb919c94d548f418ce37d311eb0.wav: No space left on device
birdcage-birdcage_backend-1   | [2023-05-23 06:55:26,827: WARNING/ForkPoolWorker-1] Error: ffmpeg error (see stderr output for detail)
birdcage-birdcage_backend-1   | [2023-05-23 06:55:26,827: WARNING/ForkPoolWorker-1] Recording failed. Error: ffmpeg error (see stderr output for detail)

Clearly I have plenty of space. Odd!

[root@homedocker1 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         2.0G     0  2.0G   0% /dev
tmpfs                            2.0G     0  2.0G   0% /dev/shm
tmpfs                            2.0G   18M  2.0G   1% /run
tmpfs                            2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/centos-root          150G   60G   91G  40% /
/dev/mapper/centos-home           42G  1.3G   40G   3% /home
/dev/sda1                       1014M  156M  859M  16% /boot
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/b811da5f34bbbcba628e46a87b25df3a16e808fcea8ab3887b948b98f5e4f269/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/f6488296b205ad41f22d44dddda125dac660c90a3df79589ba00e617849f0682/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/a9ba92bc3f9911070240b4b0d7457f9e1cfe1c4add8ab7ca621b2914f42f82fb/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/900782b1e17c2cbdbc733c4130171e2796cec3b91bfb73daba099192b356339d/merged
tmpfs                            396M     0  396M   0% /run/user/0
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/6c44a375c3f726a495f9af742aa70777ea7eab7806381b6179f8cabd5a574feb/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/6b10720f7bc30a4058181c7fe6fa808cf2597431ecc67119d20426de83e0eee7/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/e25837eb93c2babadc836add6be8d6479c39897e4e8c2f54c73e5a0b7f62ce73/merged
overlay                          150G   60G   91G  40% /var/lib/docker/overlay2/604448a1ad41468051a055f5610fb843e14e23f8c9aa78b2a2721dcd73c5a852/merged

The text was updated successfully, but these errors were encountered:

Phaeton · 2023-05-23T14:04:37Z

If you docker exec -it birdcage_backend bash into the server and run du -sh tmp, how big is that folder? Is birdnetserver running? Your log shows multiple writes to /tmp, but does not show the analysis requests. I would guess birdnetserver is down or you need to increase the tmpfs: size. 16M may be too small, see the line comment in the docker-compose.yml.

yetdog · 2023-05-23T14:23:58Z

If you docker exec -it birdcage_backend bash into the server and run du -sh tmp, how big is that folder? Is birdnetserver running? Your log shows multiple writes to /tmp, but does not show the analysis requests. I would guess birdnetserver is down or you need to increase the tmpfs: size. 16M may be too small, see the line comment in the docker-compose.yml.

Appreciate all your help on this!

I did a docker system prune and then also added in a volume mapping for ./tmp -> /tmp (like your fork), and it seemed to stop erroring on space.

However, you're correct that there are no log entries for analysis, and subsequently, nothing showing up in the UI for detections. birdnetserver is running (I think?):

[root@homedocker1 birdcage]# docker logs birdnet-analyzer_birdnetserver_1
UP AND RUNNING! LISTENING ON 0.0.0.0:8080
UP AND RUNNING! LISTENING ON 0.0.0.0:8080
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

The logs I've provided have all been from the attached output of docker compose up, for what it's worth. Would the analysis lines show up there?

yetdog · 2023-05-23T14:38:09Z

And I ran out of space again, this time after far more recordings. tmpfs raised to 64MB this time. I'm guessing what's happening is, the recordings are being made, but not analyzed, so the raw recordings are never being rotated out and whatever the size of tmpfs is, will eventually be exhausted.

I see the initial task created to loop through the recordings:

birdcage-birdcage_backend-1   | [2023-05-23 06:53:53,225: INFO/MainProcess] Task app.stream_processing.analyze_recordings[7d9c8ed6-d884-4f5c-b916-58d566c40b8c] received

But can't seem to find where it's falling over.

I wonder... is config.py, at the root of the birdcage_backend container in use? Because It's still leveraging that 192.168.1.75 address...hmmm

root@1ae0ba7a08ed:/app# cat /config.py 
import os

DATABASE_FILE = os.environ.get('DATABASE_FILE', 'birdcage.db')
API_SERVER_PORT = int(os.environ.get('API_SERVER_PORT', 7006))
TEMP_DIR_NAME = os.environ.get('TEMP_DIR_NAME', 'tmp')
ANALYZE_SERVER = os.environ.get('ANALYZE_SERVER', '192.168.1.75')
ANALYZE_PORT = int(os.environ.get('ANALYZE_PORT', 7667))
DETECTION_DIR_NAME = os.environ.get('DETECTION_DIR_NAME', 'detections')
CORS_ORIGINS = os.environ.get('CORS_ORIGINS', 'http://192.168.1.75:7009')
REDIS_SERVER = os.environ.get('REDIS_SERVER', '192.168.1.75')
REDIS_PORT = os.environ.get('REDIS_PORT', 6380)

Phaeton · 2023-05-23T14:47:02Z

I did a docker system prune and then also added in a volume mapping for ./tmp -> /tmp (like your fork), and it seemed to stop erroring on space.

You will eventually want to unwind this. You must have been looking at my unpatched main branch. Long term you do not want to be writing the temporary data to disk. You are running out of space as the files are not being analyzed and removed. And it does not take long to fill up a 16mb buffer...

However, you're correct that there are no log entries for analysis, and subsequently, nothing showing up in the UI for detections. birdnetserver is running (I think?):
[root@homedocker1 birdcage]# docker logs birdnet-analyzer_birdnetserver_1
UP AND RUNNING! LISTENING ON 0.0.0.0:8080
UP AND RUNNING! LISTENING ON 0.0.0.0:8080
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

I agree, looks like the analysis server is up. But the data is not making it to the analysis server. You sure you got things mapped in your docker-compose.yml correctly? Correct paths and ports? You should be seeing activity in both birdnetserver and birdcage_backend.

yetdog · 2023-05-23T14:51:42Z

Yeah, I walked back the /tmp thing. We're good there.

Only thing I can think - while the birdnetserver container is running port 8080 internally, it's advertising to the host on 7667. Birdcage is pointing at the host IP address on that port 7667 for the analysis server. Which, seems correct, but I'm admittedly not expert on docker networking. My compose file looks like this (this server IP that runs docker is 192.168.111.170)

    environment:
      DATABASE_FILE: /db/birdcage.db
      API_SERVER_PORT: 7007
      TEMP_DIR_NAME: tmp
      ANALYZE_SERVER: 192.168.111.170
      ANALYZE_PORT: 7667
      DETECTION_DIR_NAME: detections
      CORS_ORIGINS: http://192.168.111.170:7008
      REDIS_SERVER: redis
      REDIS_PORT: 6379

And docker ps brings the following:

[root@homedocker1 BirdNET-Analyzer]# docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS                 PORTS                                                                              NAMES
5e508dc6fc29   mmcc73/birdcage_frontend:latest   "./start_app.sh"         9 seconds ago    Up 7 seconds           0.0.0.0:7008->7008/tcp, :::7008->7008/tcp                                          birdcage-birdcage_frontend-1
9bca399be736   mmcc73/birdcage_backend:latest    "./start_app.sh"         9 seconds ago    Up 8 seconds           0.0.0.0:7007->7007/tcp, :::7007->7007/tcp                                          birdcage-birdcage_backend-1
7c3adefdaf7a   redis:latest                      "docker-entrypoint.s…"   50 minutes ago   Up 8 seconds           0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                          birdcage-redis-1
9294a6e6c974   birdnetserver                     "python3 server.py"      2 hours ago      Up 2 hours             0.0.0.0:7667->8080/tcp, :::7667->8080/tcp                                          birdnet-analyzer_birdnetserver_1

Phaeton · 2023-05-23T15:02:44Z

Everything looks OK to me in that screenshot. You able to publish a snip of your docker-compose.yml that includes all 4 containers? The thing I would guess without seeing it all would be that your birdnetserver container is in a different network than your other 3. Make sure all 4 containers are in the same one.

mmcc-xx · 2023-05-23T15:03:26Z

FYI: The values in config.py are overridden by environment variables set in the docker-compose file. I have the defaults set to values that work in my test environment.

Start with getting the birdnet-analyzer service running. If it is working correctly, you should be able to do a wget from the command line of your server to http://(analyzer ip address):(analyzer port)/healthcheck and get back a json string like {"msg": "Server is healthy."}

Once you have that working, put that IP address and port into the docker compose file for ANALYZE_SERVER and ANALYZE_PORT.

Feel free to share your whole docker-compose file. We'll get it going I have no doubt.

Later today I'll push up an image of the analyzer so you can avoid that step. If and when the BirdNET folks start providing images themselves I'll switch.

yetdog · 2023-05-23T15:10:54Z

I just added the birdnetserver container to the docker compose. I suspect the networking is acting as an isolating factor between your app, and the birdnetserver:

[root@homedocker1 birdcage]# cat docker-compose.yml 
version: '3.8'

services:
  birdnetserver:
    restart: unless-stopped
    image: birdnetserver
    ports:
      - 7667:8080
    networks:
     - birdcage_net
  redis:
    image: "redis:latest"
    ports:
      - "6379:6379"
    networks:
      - birdcage_net

  birdcage_backend:
    image: "mmcc73/birdcage_backend:latest"
    ports:
      - "7007:7007"
    environment:
      DATABASE_FILE: /db/birdcage.db
      API_SERVER_PORT: 7007
      TEMP_DIR_NAME: tmp
      ANALYZE_SERVER: 192.168.111.170
      ANALYZE_PORT: 7667
      DETECTION_DIR_NAME: detections
      CORS_ORIGINS: http://192.168.111.170:7008
      REDIS_SERVER: redis
      REDIS_PORT: 6379
    tmpfs:
      - /tmp:size=64M #you might want to increase this size if you are recording a bunch of streams, if you streams are particularly hi-res, or if your analyzer might be periodically unavailable
    volumes:
      - "./detections:/detections"
      - "./db:/db"
      - "/etc/localtime:/etc/localtime:ro"
      - "/home/matt/birdcage/timezone:/etc/timezone:ro"
    depends_on:
      - redis
    networks:
      - birdcage_net

  birdcage_frontend:
    image: "mmcc73/birdcage_frontend:latest"
    ports:
      - "7008:7008"
    environment:
      API_SERVER_URL: http://192.168.111.170:7007
      WEBUI_PORT: 7008
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/home/matt/birdcage/timezone:/etc/timezone:ro"
    depends_on:
      - birdcage_backend
    networks:
      - birdcage_net

networks:
  birdcage_net:
    driver: host

Note I changed the driver from bridge to host. This doesn't seem to have made a difference either way. Of note, if I play with the port number for ANALYZE_PORT (setting it to the birdnetserver container's internal 8080 port), I get a hard failure during docker compose up:

birdcage-birdcage_backend-1   | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.111.170', port=8080): Max retries exceeded with url: /predictedspecies?latitude=35.86264&longitude=-86.66016&week_number=21&sf_thresh=0.03 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1efdfa1d20>: Failed to establish a new connection: [Errno 111] Connection refused'))

So I would assume that the absence of this error means it is connecting successfully / can see the server. So...yeah. I'm confused. :)

Also, yes, I get the server healthy message when browsing to the appropriate URL.

mmcc-xx · 2023-05-23T15:18:51Z

The backend shouldn't be trying to hit port 8080... it should be trying to hit port 7667. Let me look to see if I screwed something up.

yetdog · 2023-05-23T15:22:57Z

The backend shouldn't be trying to hit port 8080... it should be trying to hit port 7667. Let me look to see if I screwed something up.

Sorry, I was just playing around trying things. I set it to 8080 to see what happens and it indeed errored.

mmcc-xx · 2023-05-23T15:24:22Z

If you hit http://192.168.111.170:7667/predictedspecies?latitude=35.86264&longitude=-86.66016&week_number=21&sf_thresh=0.03 in your browser do you get a bunch of data back?

mmcc-xx · 2023-05-23T15:28:48Z

In your docker compose you have "driver: host" ... not sure that's what you want. Probably bridge. It should still work, but...

yetdog · 2023-05-23T15:30:29Z

If you hit http://192.168.111.170:7667/predictedspecies?latitude=35.86264&longitude=-86.66016&week_number=21&sf_thresh=0.03 in your browser do you get a bunch of data back?

[["Cardinalis cardinalis_Northern Cardinal", 0.9906850457191467], ["Zenaida macroura_Mourning Dove", 0.9454413652420044], ["Thryothorus ludovicianus_Carolina Wren", 0.8862005472183228], ["Corvus brachyrhynchos_American Crow", 0.874495267868042], ["Baeolophus bicolor_Tufted Titmouse", 0.850361704826355], ["Cyanocitta cristata_Blue Jay", 0.8412767648696899], ["Passerina cyanea_Indigo Bunting", 0.8409078121185303], ["Poecile carolinensis_Carolina Chickadee", 0.7993695735931396], ["Turdus migratorius_American Robin", 0.7658781409263611], ["Mimus polyglottos_Northern Mockingbird", 0.712925136089325], ["Melanerpes carolinus_Red-bellied Woodpecker", 0.6868361830711365], ["Sialia sialis_Eastern Bluebird", 0.6477306485176086], ["Pipilo erythrophthalmus_Eastern Towhee", 0.6381068825721741], ["Spinus tristis_American Goldfinch", 0.5951390266418457], ["Molothrus ater_Brown-headed Cowbird", 0.5703653693199158], ["Agelaius phoeniceus_Red-winged Blackbird", 0.5568423867225647], ["Spizella passerina_Chipping Sparrow", 0.5523957014083862], ["Vireo olivaceus_Red-eyed Vireo", 0.5459585189819336], ["Hirundo rustica_Barn Swallow", 0.5355029702186584], ["Dryobates pubescens_Downy Woodpecker", 0.500104546546936], ["Contopus virens_Eastern Wood-Pewee", 0.4711223244667053], ["Archilochus colubris_Ruby-throated Hummingbird", 0.46849337220191956], ["Sturnus vulgaris_European Starling", 0.46818703413009644], ["Sitta carolinensis_White-breasted Nuthatch", 0.45386624336242676], ["Quiscalus quiscula_Common Grackle", 0.446478933095932], ["Vireo griseus_White-eyed Vireo", 0.4420704245567322], ["Sayornis phoebe_Eastern Phoebe", 0.43810853362083435], ["Coccyzus americanus_Yellow-billed Cuckoo", 0.4327424168586731], ["Polioptila caerulea_Blue-gray Gnatcatcher", 0.42373040318489075], ["Toxostoma rufum_Brown Thrasher", 0.4217589199542999], ["Haemorhous mexicanus_House Finch", 0.4191194176673889], ["Piranga rubra_Summer Tanager", 0.413064569234848], ["Icteria virens_Yellow-breasted Chat", 0.40824925899505615], ["Myiarchus crinitus_Great Crested Flycatcher", 0.40818101167678833], ["Hylocichla mustelina_Wood Thrush", 0.39580532908439636], ["Chaetura pelagica_Chimney Swift", 0.39167892932891846], ["Geothlypis trichas_Common Yellowthroat", 0.37341997027397156], ["Tyrannus tyrannus_Eastern Kingbird", 0.3419959545135498], ["Ardea herodias_Great Blue Heron", 0.3340374231338501], ["Spizella pusilla_Field Sparrow", 0.3266265392303467], ["Setophaga americana_Northern Parula", 0.304625540971756], ["Passer domesticus_House Sparrow", 0.30313053727149963], ["Empidonax virescens_Acadian Flycatcher", 0.28157538175582886], ["Passerina caerulea_Blue Grosbeak", 0.2567811906337738], ["Dryocopus pileatus_Pileated Woodpecker", 0.2534903883934021], ["Charadrius vociferus_Killdeer", 0.2501795291900635], ["Melospiza melodia_Song Sparrow", 0.24751131236553192], ["Sturnella magna_Eastern Meadowlark", 0.24723376333713531], ["Icterus spurius_Orchard Oriole", 0.22201243042945862], ["Branta canadensis_Canada Goose", 0.21589051187038422], ["Progne subis_Purple Martin", 0.21257755160331726], ["Setophaga citrina_Hooded Warbler", 0.21015393733978271], ["Stelgidopteryx serripennis_Northern Rough-winged Swallow", 0.1935686320066452], ["Bombycilla cedrorum_Cedar Waxwing", 0.18788810074329376], ["Ardea alba_Great Egret", 0.1847226768732071], ["Setophaga pinus_Pine Warbler", 0.16816453635692596], ["Dumetella carolinensis_Gray Catbird", 0.1652422547340393], ["Vireo flavifrons_Yellow-throated Vireo", 0.1633426398038864], ["Melanerpes erythrocephalus_Red-headed Woodpecker", 0.1610034704208374], ["Buteo lineatus_Red-shouldered Hawk", 0.158102348446846], ["Piranga olivacea_Scarlet Tanager", 0.1465430110692978], ["Colinus virginianus_Northern Bobwhite", 0.1340598464012146], ["Protonotaria citrea_Prothonotary Warbler", 0.1337331086397171], ["Buteo jamaicensis_Red-tailed Hawk", 0.12973639369010925], ["Butorides virescens_Green Heron", 0.12375551462173462], ["Coragyps atratus_Black Vulture", 0.1230330765247345], ["Corvus ossifragus_Fish Crow", 0.11474326997995377], ["Geothlypis formosa_Kentucky Warbler", 0.11308159679174423], ["Petrochelidon pyrrhonota_Cliff Swallow", 0.10281942039728165], ["Troglodytes aedon_House Wren", 0.10149995982646942], ["Tachycineta bicolor_Tree Swallow", 0.09985006600618362], ["Aix sponsa_Wood Duck", 0.09982909262180328], ["Spiza americana_Dickcissel", 0.09857700020074844], ["Setophaga dominica_Yellow-throated Warbler", 0.09813401103019714], ["Dryobates villosus_Hairy Woodpecker", 0.0937262773513794], ["Anas platyrhynchos_Mallard", 0.08513417094945908], ["Colaptes auratus_Northern Flicker", 0.08422033488750458], ["Meleagris gallopavo_Wild Turkey", 0.07459843903779984], ["Setophaga discolor_Prairie Warbler", 0.0718996524810791], ["Columba livia_Rock Pigeon", 0.06416220963001251], ["Bubulcus ibis_Cattle Egret", 0.063527412712574], ["Megaceryle alcyon_Belted Kingfisher", 0.06253904104232788], ["Parkesia motacilla_Louisiana Waterthrush", 0.06203804537653923], ["Ictinia mississippiensis_Mississippi Kite", 0.055010803043842316], ["Pheucticus ludovicianus_Rose-breasted Grosbeak", 0.053437814116477966], ["Sitta pusilla_Brown-headed Nuthatch", 0.05150870978832245], ["Nannopterum auritum_Double-crested Cormorant", 0.049560029059648514], ["Strix varia_Barred Owl", 0.0492498055100441], ["Icterus galbula_Baltimore Oriole", 0.04573008790612221], ["Seiurus aurocapilla_Ovenbird", 0.04404589533805847], ["Streptopelia decaocto_Eurasian Collared-Dove", 0.04313020780682564], ["Mniotilta varia_Black-and-white Warbler", 0.04062333703041077], ["Pandion haliaetus_Osprey", 0.04015040025115013], ["Vireo gilvus_Warbling Vireo", 0.04002200439572334], ["Actitis macularius_Spotted Sandpiper", 0.03787084296345711], ["Setophaga ruticilla_American Redstart", 0.03749420493841171], ["Setophaga petechia_Yellow Warbler", 0.036387648433446884], ["Egretta thula_Snowy Egret", 0.03591585531830788], ["Haliaeetus leucocephalus_Bald Eagle", 0.030686460435390472]]

In your docker compose you have "driver: host" ... not sure that's what you want. Probably bridge. You don't need the redis server on the host network. It should still work, but...

Yeah, I switched it back to bridge. Doesn't seem to make a difference either way.

mmcc-xx · 2023-05-23T15:46:35Z

Ok. Now try http://192.168.111.170:7007 - does a page with some API documentation come up?

mmcc-xx · 2023-05-23T15:53:29Z

The back end server is supposed to spin up one task per stream, and one task to handle analysis. The analysis task monitors for files in the tmp directory and, pulls in some metedata from the database, calls the analysis service, stores results as necessary, and deletes the input file from the tmp directory. I'm not understanding why that isn't working.

Can you verify that there is a db directory in the directory where your docker compose file is? And that in this db directory is a file called birdcage.db and it has a non 0 size?

yetdog · 2023-05-23T16:05:00Z

Ok. Now try http://192.168.111.170:7007 - does a page with some API documentation come up?

Yes sir - swagger comes up!

The back end server is supposed to spin up one task per stream, and one task to handle analysis. The analysis task monitors for files in the tmp directory and, pulls in some metedata from the database, calls the analysis service, stores results as necessary, and deletes the input file from the tmp directory. I'm not understanding why that isn't working.

Can you verify that there is a db directory in the directory where your docker compose file is? And that in this db directory is a file called birdcage.db and it has a non 0 size?

[root@homedocker1 birdcage]# ls -la db/
total 84
drwxr-xr-x. 2 root root    25 May 23 08:57 .
drwxr-xr-x. 5 root root    87 May 23 08:12 ..
-rw-r--r--. 1 root root 86016 May 23 08:57 birdcage.db

Latest logs:

birdcage-birdcage_backend-1   | [2023-05-23 08:59:20,842: INFO/MainProcess] Task app.stream_processing.analyze_recordings[42aa06fc-d009-4e6f-96c9-9085398d71f2] received
birdcage-birdcage_backend-1   | [2023-05-23 08:59:20,847: INFO/MainProcess] Task app.stream_processing.record_stream[ecd26479-4de7-4c1a-abe7-804dc2b51556] received
birdcage-birdcage_backend-1   | [2023-05-23 08:59:20,849: INFO/MainProcess] Task app.stream_processing.record_stream[1e0d2274-e85e-48da-88e4-c15f5c032945] received
birdcage-birdcage_backend-1   | [2023-05-23 08:59:20,850: INFO/MainProcess] Task app.stream_processing.analyze_recordings[51eff7d7-c503-47d9-9351-e3389749d16b] received
birdcage-birdcage_backend-1   | [2023-05-23 08:59:20,852: INFO/MainProcess] Task app.stream_processing.record_stream[464e6164-ca83-4ed4-bf75-eacbf6a27c7d] received
birdcage-birdcage_backend-1   | [2023-05-23 09:04:32,680: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/a32a5565b56e4b87bebe57418f90f838.wav
birdcage-birdcage_backend-1   | [2023-05-23 09:04:48,813: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/57a7c9ee6d864ff6b6825ef3ca47ea01.wav

Running from docker-compose.yml file:

[root@homedocker1 birdcage]# cat docker-compose.yml 
version: '3.8'

services:
  birdnetserver:
    restart: unless-stopped
    image: birdnetserver
    ports:
      - 7667:8080
    networks:
     - birdcage_net
  redis:
    image: "redis:latest"
    ports:
      - "6379:6379"
    networks:
      - birdcage_net

  birdcage_backend:
    image: "mmcc73/birdcage_backend:latest"
    ports:
      - "7007:7007"
    environment:
      DATABASE_FILE: /db/birdcage.db
      API_SERVER_PORT: 7007
      TEMP_DIR_NAME: tmp
      ANALYZE_SERVER: 192.168.111.170
      ANALYZE_PORT: 7667
      DETECTION_DIR_NAME: detections
      CORS_ORIGINS: http://192.168.111.170:7008
      REDIS_SERVER: redis
      REDIS_PORT: 6379
    tmpfs:
      - /tmp:size=64M #you might want to increase this size if you are recording a bunch of streams, if you streams are particularly hi-res, or if your analyzer might be periodically unavailable
    volumes:
      - "./detections:/detections"
      - "./db:/db"
      - "/etc/localtime:/etc/localtime:ro"
      - "/home/matt/birdcage/timezone:/etc/timezone:ro"
    depends_on:
      - redis
    networks:
      - birdcage_net

  birdcage_frontend:
    image: "mmcc73/birdcage_frontend:latest"
    ports:
      - "7008:7008"
    environment:
      API_SERVER_URL: http://192.168.111.170:7007
      WEBUI_PORT: 7008
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "/home/matt/birdcage/timezone:/etc/timezone:ro"
    depends_on:
      - birdcage_backend
    networks:
      - birdcage_net

networks:
  birdcage_net:
    driver: bridge

mmcc-xx · 2023-05-23T16:12:12Z

After "recording successful" you should see some output from the analysis task indicating that it is working on the wav files, which you are not....

Do you have multiple streams defined? If so, could you try with just one?

Assuming that doesn't tell us anything I'll add some additional debugging messages later today to see if we can figure out what's happening.

Also, I assume you aren't getting any mp3s in the dectections folder correct?

yetdog · 2023-05-23T16:21:16Z

I just stopped and destroyed all containers, and redeployed fresh. Added back in a single RTSP stream, CTRL-C'd out, then started back up.

Correct, no .mp3s which makes sense given that it seems that analysis isn't happening on the .wav's created.

birdcage-birdcage_backend-1   | [2023-05-23 09:20:16,604: INFO/MainProcess] Task app.stream_processing.record_stream[b375bd01-accd-4394-b4f0-b4f4debe3150] received
birdcage-birdcage_backend-1   | [2023-05-23 09:20:16,614: INFO/MainProcess] Task app.stream_processing.analyze_recordings[0d187b13-d13d-4312-96f3-8bd65821e7ac] received
birdcage-birdcage_backend-1   | [2023-05-23 09:20:31,782: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/17957d6f2ec346ddba2169d6a0aaf916.wav
birdcage-birdcage_backend-1   | [2023-05-23 09:20:47,968: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/314dadeca1314f74b518f6018f58525e.wav
birdcage-birdcage_backend-1   | [2023-05-23 09:21:04,165: WARNING/ForkPoolWorker-1] Recording successful. File saved to: /app/../tmp/eb6d6d0b4daf4be3a42050ec598b92d9.wav

I can bash into the birdnetserver container and run the client.py script successfully...so it looks like something just isn't being called birdcage->birdnetserver.

Again - really appreciate the work on this!

mmcc-xx · 2023-05-23T16:24:28Z

Sure - if you are having this problem then other people will have this problem. I'll add some debugging messages later on.

Works fine on my machine :)

yetdog · 2023-05-23T16:30:46Z

I can certain spin up another Linux host and try there. I’m an old RHEL stodger and I know Centos/Alma aren’t preferred for docker. (Despite docker supposedly being independent from the underlying OS. Hi Java! Don’t systemd me, bro. ;) Will try something fresh and report back. One Q; do you put birdnet server in the same compose file/network ?

…

Sent from my iPhone On May 23, 2023, at 11:26 AM, mmcc-xx ***@***.***> wrote: Sure - if you are having this problem then other people will have this problem. I'll add some debugging messages later on. Works fine on my machine :) — Reply to this email directly, view it on GitHub<#15 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABRHNAXI5B2SBCYKSF5IPTLXHTQENANCNFSM6AAAAAAYL6DAXU>. You are receiving this because you authored the thread.Message ID: ***@***.***> [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#15 (comment)", "url": "#15 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

mmcc-xx · 2023-05-23T16:55:38Z

do you put birdnet server in the same compose file/network ?

I don't but I should and I will when I finally push an image.

I’m an old RHEL stodger

I once sold Bob Young, founder of Red Hat, a typewriter. I bring this up whenever someone mentions red hat.

yetdog · 2023-05-23T17:56:05Z

Just tried a fresh Ubuntu 22 VM. Birdnetserver doesnt start due to this:

birdcage-birdnetserver-1 exited with code 1
birdcage-birdnetserver-1      | Traceback (most recent call last):
birdcage-birdnetserver-1      |   File "server.py", line 11, in <module>
birdcage-birdnetserver-1      |     import config as cfg
birdcage-birdnetserver-1      |   File "/config.py", line 115, in <module>
birdcage-birdnetserver-1      |     LABELS: list[str] = []
birdcage-birdnetserver-1      | TypeError: 'type' object is not subscriptable

and so everything cascades from there. :)

yetdog · 2023-05-23T18:05:09Z

Ah ha. So it appears that BirdNET had a commit kahst/BirdNET-Analyzer@81052b8 that changed all that typing, so I grabbed the previous version of config.py from there and then it all works great.

SO. With all of that, on my new ubuntu22 VM.....drumroll? IT WORKS!

birdcage-birdnetserver-1      | Analyzing /tmp/tmpfwc_xu12.wav
birdcage-birdnetserver-1      | Finished /tmp/tmpfwc_xu12.wav in 0.57 seconds
birdcage-birdnetserver-1      | Results:
birdcage-birdnetserver-1      | {'6.0;9.0': [('Quiscalus quiscula_Common Grackle', 0.0977), ('Quiscalus mexicanus_Great-tailed Grackle', 0.0122), ('Melanerpes carolinus_Red-bellied Woodpecker', 0.0075)], '3.0;6.0': [('Bombycilla cedrorum_Cedar Waxwing', 0.0655)], '0.0;3.0': [('Spizella passerina_Chipping Sparrow', 0.022)]}

mmcc-xx · 2023-05-23T18:06:42Z

Phew. Ok that emphasizes the need for a known good image of the analyzer. I'll push one later today and update the docer-compose. Thanks for the troubleshooting.

yetdog · 2023-05-23T18:08:24Z

Phew. Ok that emphasizes the need for a known good image of the analyzer. I'll push one later today and update the docer-compose. Thanks for the troubleshooting.

My pleasure!

Is there a lag where it takes a bit for detections to show up on the main web page?

yetdog · 2023-05-23T18:13:18Z

I think I've spoken too soon. Back to original behavior, even with the new setup! I added cameras 2 and 3 and it just gets in its recording loop.

I removed cams 2 & 3, and it's back to analyzing. :)

mmcc-xx · 2023-05-23T18:33:49Z

Ok, good to know! I never actually tested it with multiple cameras :) I'll get on that tonight.

…

On Tue, May 23, 2023 at 2:13 PM yetdog ***@***.***> wrote: I think I've spoken too soon. Back to original behavior, even with the new setup! I added cameras 2 and 3 and it just gets in its recording loop. I removed cams 2 & 3, and it's back to analyzing. :) — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKBEDJGZ3Q35VUATFNNKJTXHT44VANCNFSM6AAAAAAYL6DAXU> . You are receiving this because you commented.Message ID: ***@***.***>

mmcc-xx · 2023-05-23T23:33:52Z

OK - I pushed new images and posted a new docker-compose file that I think will address both problems. I think the issue with the multiple cameras was that I hadn't configured celery to have enough worker processes. Now by default there are 10, which I think should handle 8 streams. Maybe 7. But more than 1.

I also push an image for the analyzer service

mmcc-xx · 2023-05-25T17:10:39Z

I think this is fixed, so I'm closing the issue. If I'm wrong feel free to open another one.

mmcc-xx closed this as completed May 25, 2023

rpatel3001 mentioned this issue Jun 7, 2023

no space left on device #30

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No space left on device ffmpeg? #15

No space left on device ffmpeg? #15

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

yetdog commented May 23, 2023

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023 •

edited

Loading

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

mmcc-xx commented May 23, 2023 •

edited

Loading

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023 •

edited

Loading

yetdog commented May 23, 2023 •

edited

Loading

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023 via email

mmcc-xx commented May 23, 2023 •

edited

Loading

yetdog commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023 via email

mmcc-xx commented May 23, 2023

mmcc-xx commented May 25, 2023

No space left on device ffmpeg? #15

No space left on device ffmpeg? #15

Comments

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

yetdog commented May 23, 2023

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

yetdog commented May 23, 2023

Phaeton commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023 • edited Loading

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

mmcc-xx commented May 23, 2023 • edited Loading

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023 • edited Loading

yetdog commented May 23, 2023 • edited Loading

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023 via email

mmcc-xx commented May 23, 2023 • edited Loading

yetdog commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023

yetdog commented May 23, 2023

yetdog commented May 23, 2023

mmcc-xx commented May 23, 2023 via email

mmcc-xx commented May 23, 2023

mmcc-xx commented May 25, 2023

yetdog commented May 23, 2023 •

edited

Loading

mmcc-xx commented May 23, 2023 •

edited

Loading

mmcc-xx commented May 23, 2023 •

edited

Loading

yetdog commented May 23, 2023 •

edited

Loading

mmcc-xx commented May 23, 2023 •

edited

Loading