Add logs if we see ENOSPC

[dmanjunath]

Trello Card Link

https://trello.com/c/dyViGxGG/1654-track-upload-errors

Description

We see this error occasionally, but both disk space and inode usage seems fine. Since it is causing uploads to fail, we should try to log it out as soon as an operation fails in case this is a transient error.

Edit: My gut take as to the root cause is that large uploads/many concurrent uploads cause some file system inode limit or something because of https://github.com/AudiusProject/audius-protocol/blob/master/creator-node/src/routes/tracks.js#L73-L77. We do promise.all to move all the segments around. I feel like chunking may alleviate this problem.

Services

• Discovery Provider
• Creator Node
• Identity Service
• Libs
• Contracts
• Service Commands
• Mad Dog

Does it touch a critical flow like Discovery indexing, Creator Node track upload, Creator Node gateway, or Creator Node file system?

Delete an option.

• 🚨 Yes, this touches track upload

How Has This Been Tested?

Uploaded tracks locally to verify it worked with this change.

Then threw a ENOSPC error manually and observed if it printed the output

[2020-11-06T17:18:29.544Z]  INFO: audius_creator_node/389 on 630e01723352: (requestID=YhacSgYbUX, requestMethod=POST, requestHostname=cn2_creator-node_1, requestUrl=/track_content)
    Successfully executed command df -h with output: 
    Filesystem                Size      Used Available Use% Mounted on
    overlay                  58.4G     44.4G     11.0G  80% /
    tmpfs                    64.0M         0     64.0M   0% /dev
    tmpfs                   995.4M         0    995.4M   0% /sys/fs/cgroup
    shm                      64.0M         0     64.0M   0% /dev/shm
    /dev/vda1                58.4G     44.4G     11.0G  80% /file_storage
    /dev/vda1                58.4G     44.4G     11.0G  80% /etc/resolv.conf
    /dev/vda1                58.4G     44.4G     11.0G  80% /etc/hostname
    /dev/vda1                58.4G     44.4G     11.0G  80% /etc/hosts
    grpcfuse                465.6G    377.0G     67.6G  85% /usr/src/audius-libs
    grpcfuse                465.6G    377.0G     67.6G  85% /usr/src/app
    /dev/vda1                58.4G     44.4G     11.0G  80% /usr/src/app/node_modules
    tmpfs                   995.4M         0    995.4M   0% /proc/acpi
    tmpfs                    64.0M         0     64.0M   0% /proc/kcore
    tmpfs                    64.0M         0     64.0M   0% /proc/keys
    tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
    tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
    tmpfs                   995.4M         0    995.4M   0% /sys/firmware
    
HandleResponse Error: ENOSPC no space left on device track_content
    at saveFileToIPFSFromFS (/usr/src/app/src/fileManager.js:66:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-11-06T17:18:29.610Z]  INFO: audius_creator_node/389 on 630e01723352: (requestID=YhacSgYbUX, requestMethod=POST, requestHostname=cn2_creator-node_1, requestUrl=/track_content)
    Successfully executed command df -ih with output: 
    Filesystem              Inodes      Used Available Use% Mounted on
    overlay                   3.7M      1.6M      2.1M  43% /
    tmpfs                   248.8K        16    248.8K   0% /dev
    tmpfs                   248.8K        15    248.8K   0% /sys/fs/cgroup
    shm                     248.8K         1    248.8K   0% /dev/shm
    /dev/vda1                 3.7M      1.6M      2.1M  43% /file_storage
    /dev/vda1                 3.7M      1.6M      2.1M  43% /etc/resolv.conf
    /dev/vda1                 3.7M      1.6M      2.1M  43% /etc/hostname
    /dev/vda1                 3.7M      1.6M      2.1M  43% /etc/hosts
    grpcfuse                  4.5G      3.5M      4.5G   0% /usr/src/audius-libs
    grpcfuse                  4.5G      3.5M      4.5G   0% /usr/src/app
    /dev/vda1                 3.7M      1.6M      2.1M  43% /usr/src/app/node_modules
    tmpfs                   248.8K         1    248.8K   0% /proc/acpi
    tmpfs                   248.8K        16    248.8K   0% /proc/kcore
    tmpfs                   248.8K        16    248.8K   0% /proc/keys
    tmpfs                   248.8K        16    248.8K   0% /proc/timer_list
    tmpfs                   248.8K        16    248.8K   0% /proc/sched_debug
    tmpfs                   248.8K         1    248.8K   0% /sys/firmware