bug storage: zot crashes if fd-limit reached

[adodon2go]

Describe the bug
Zot crashes if file descriptor limit is reached

To Reproduce
Steps to reproduce the behavior:

1. Check the `Soft Limit" of zot process regarding file descriptors:

midgard@ubuntu:$ ps -ax | grep zot
49395 pts/1 Sl+ 0:00 bin/zot serve examples/config-test.json
49484 pts/2 S+ 0:00 grep --color=auto zot
midgard@ubuntu:$ cat /proc/49395/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 95802 95802 processes
Max open files 1024 1048576 files
Max locked memory 67108864 67108864 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 95802 95802 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
midgard@ubuntu:~$

2. In above case the value is 1024.
3. Do some load test on zot (During 5 minutes interval a number of 1000 concurrent clients that uploads a small image like alpine:3.15)
4. Zot crashes
   `
   {"level":"error","error":"open /mnt/ramdisk/zot-extended/client62/.uploads/c0a605ed-ff0a-408e-a5d6-8672e2385458: too many open files","blob":"/mnt/ramdisk/zot-extended/client62/.uploads/c0a605ed-ff0a-408e-a5d6-8672e2385458","goroutine":132746,"caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:936","time":"2022-01-10T22:19:51.773990827+02:00","caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:936","message":"failed to open blob"}
   2022/01/10 22:19:51 http: Accept error: accept tcp 127.0.0.1:5000: accept4: too many open files; retrying in 5ms
   {"level":"error","error":"open /mnt/ramdisk/zot-extended/client49/.uploads/8cf9ef17-fb8e-4f8c-a65d-7c3c13466ff5: too many open files","blob":"/mnt/ramdisk/zot-extended/client49/.uploads/8cf9ef17-fb8e-4f8c-a65d-7c3c13466ff5","goroutine":132752,"caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:936","time":"2022-01-10T22:19:51.820136769+02:00","caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:936","message":"failed to open blob"}
   2022/01/10 22:19:53 http: Accept error: accept tcp 127.0.0.1:5000: accept4: too many open files; retrying in 5ms
   {"level":"fatal","error":"open /mnt/ramdisk/zot-extended/client173/.uploads/f39028fc-9d2e-4639-9652-88250dcf7f97: too many open files","goroutine":132007,"caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:846","time":"2022-01-10T22:19:53.049291244+02:00","caller":"zotregistry.io/zot/pkg/storage/storage_fs.go:846","message":"failed to open file"}

`

Expected behavior
Zot server should return some error code to the client like 503 in case FD limit is reached (should not crash)

[adodon2go]

See the gist for the concurrent clients upload script