memory

[RobMeerwijk]

mp3fs in docker keeps a lot of memory occupied: 4.3 GB

I use mp3fs for daily converting new music to mp3 by means of rsync. After adding a few CD's the job is finished. The memory is not yet freed 12 hours later.

[khenriks]

Hi, can you provide more info about how you're running it, like docker command? Do you know which version it is?

[khenriks]

The first 30 lines or so of docker logs output would actually be really helpful.

[RobMeerwijk]

Hi,

I use docker-compose.yml:

# Basic Docker Compose file for mp3fs
version: '3'
services:
    mp3fs:
        image: khenriks/mp3fs:latest
        devices:
            - /dev/fuse
        cap_add:
            - SYS_ADMIN
        security_opt:
            - apparmor:unconfined
        volumes:
            - /volume1/media/Music:/music:ro
            - /volume1/mp3fs/mount:/mnt:shared
        command: "-b 128"

The command I use is docker-compose up -d

docker version:

Client:
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.15.13
 Git commit:        b455053
 Built:             Thu Aug 19 07:13:24 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.13
  Git commit:       a3bc36f
  Built:            Thu Aug 19 07:11:25 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        ea3508454ff2268c32720eb4d2fc9816d6f75f88
 runc:
  Version:          v1.0.0-rc93
  GitCommit:        31cc25f16f5eba4d0f53e35374532873744f4b31
 docker-init:
  Version:          0.19.0
  GitCommit:        ed96d00

docker logs:

2021-12-09T08:33:13.320499054Z [2021-12-09 08:33:13] tid=140186708821824 DEBUG: LAME ready to initialize.
2021-12-09T08:33:13.322319578Z [2021-12-09 08:33:13] tid=140186708821824 DEBUG: mp3fs version: 1.1.1
2021-12-09T08:33:13.322435047Z LAME library version: 3.100
2021-12-09T08:33:13.322491357Z FLAC library version: 1.3.2
2021-12-09T08:33:13.322540077Z Xiph.Org libVorbis 1.3.6
2021-12-09T08:33:13.322588907Z FUSE library version: 2.9
2021-12-09T08:33:13.322635967Z 
2021-12-09T08:33:13.322674527Z [2021-12-09 08:33:13] tid=140186708821824 DEBUG: MP3FS options:
2021-12-09T08:33:13.322723476Z basepath:       /music
2021-12-09T08:33:13.322766906Z bitrate:        128
2021-12-09T08:33:13.322810286Z desttype:       mp3
2021-12-09T08:33:13.322856546Z gainmode:       1
2021-12-09T08:33:13.322897426Z gainref:        89
2021-12-09T08:33:13.322938716Z log_format:     [%T] tid=%I %L: %M
2021-12-09T08:33:13.322985205Z log_maxlevel:   DEBUG
2021-12-09T08:33:13.323027275Z log_stderr:     1
2021-12-09T08:33:13.323069105Z log_syslog:     0
2021-12-09T08:33:13.323110735Z logfile:        
2021-12-09T08:33:13.323151565Z quality:        5
2021-12-09T08:33:13.323193855Z statcachesize:  0
2021-12-09T08:33:13.323249545Z vbr:            0
2021-12-09T08:33:13.323294794Z FUSE library version: 2.9.9
2021-12-09T08:33:13.323340144Z nullpath_ok: 0
2021-12-09T08:33:13.323382634Z nopath: 0
2021-12-09T08:33:13.323448424Z utime_omit_ok: 0
2021-12-09T08:33:13.323492164Z unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
2021-12-09T08:33:13.323548654Z INIT: 7.23
2021-12-09T08:33:13.323590664Z flags=0x0003fffb
2021-12-09T08:33:13.323634913Z max_readahead=0x00030000
2021-12-09T08:33:13.323673763Z    INIT: 7.19
2021-12-09T08:33:13.323710353Z    flags=0x00000011
2021-12-09T08:33:13.323748503Z    max_readahead=0x00030000
2021-12-09T08:33:13.323799433Z    max_write=0x00020000
2021-12-09T08:33:13.323839763Z    max_background=0
2021-12-09T08:33:13.323879953Z    congestion_threshold=0
2021-12-09T08:33:13.323920312Z    unique: 1, success, outsize: 40

Actually I found a solution in docker-compose.yml that works on my docker version. But as I understand it, the options on limits are depending on the docker-compose version and not backwards/forwards compatible:

# Basic Docker Compose file for mp3fs
version: '3'
services:
    mp3fs:
        image: khenriks/mp3fs:latest
        devices:
            - /dev/fuse
        cap_add:
            - SYS_ADMIN
        security_opt:
            - apparmor:unconfined
        volumes:
            - /volume1/media/Music:/music:ro
            - /volume1/mp3fs/mount:/mnt:shared
        command: "-b 128"
        deploy:
           resources:
             limits:
                memory: 512M

[khenriks]

Do you run rsync locally or access mp3fs over the network? The main thing I can think of that would keep the memory occupied is if the file handles never get closed.

[RobMeerwijk]

I use mp3fs locally. It runs in docker on a synology NAS. It connects to a folder on the same NAS. The fs-mount-point is also local. Op 10-12-2021 om 04:02 schreef K Henriksson:

…

Do you run rsync locally or access mp3fs over the network? The main thing I can think of that would keep the memory occupied is if the file handles never get closed. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHCK2J7K3UUTJO46FA72WI3UQFUU3ANCNFSM5JT3VAEQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[RobMeerwijk]

For me the issue is solved, because I can limit available memory by means of docker-compose.

[khenriks]

I'll close this since it's addressed for you, but I am curious about the memory usage, so I might add some options to help with tracking memory usage.

[RobMeerwijk]

That is ok, I have limited the memory to the container, so no problem anymore. Op 15-1-2022 om 23:30 schreef K Henriksson:

…

I'll close this since it's addressed for you, but I am curious about the memory usage, so I might add some options to help with tracking memory usage. — Reply to this email directly, view it on GitHub <#70 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHCK2J6CXGWCEE3EZUKLVCTUWHYQDANCNFSM5JT3VAEQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>